Repair/Uninstall Mesos after cleanup - docker

The mesos server ran out of disk space and so we were doing a cleanup by removing some of the old docker containers. But now the marathon won't start and digging deeper shows nor does zookeeper. The docker log says that it cannot load some containers.
But what we noticed was that zookeeper get started then stops. So we had at look at the zookeeper folder and the the conf was missing. This was also removed on the other master server as well which we had not touched. I presume this is to do with the link between the masters. Now the slave has this conf folder but it has the default folder and files and I noticed that this is a symlink that points to the /etc/alternatives/zookeeper-conf folder.
Running the dockerfile to recreate the missing cointainer says:
Error response from daemon: Cannot start container d13b8aa28d383a3ca54b39ce74f5a81d80030a2ad0dde52966293ced9ef26663: [8] System error: exec: "mesos-master": executable file not found in $PATH
It doesn't recognise the Restart command either.
Is there a quick way to repair this to get it working as it used to? I am using Mesos 0.23 on Ubuntu 14.04
How do I uninstall Mesos?
Any help is appreciated as I am fairly new to this and so only have a basic understanding of how all this works.

re "how to uninstall Mesos", my way is
configure --prefix="your_install_path"
make
make uninstall

Related

Docker install of AZCore results in authserver+worldserver doesn't exist error

I'm trying to spin up a fresh server using the azerothcore docker installation guide. I have completed all of the early installation steps, up until running the containers. Upon running the containers (for worldserver and authserver) i see the following output from the containers. It appears the destination of the world and auth servers in dist/bin is missing, how may i resolve this issue?
Check your docker settings. Make sure you have enough memory. If containers have low memory they will not finish the compile. Check if you have build issues.

Minikube restarts forever

I have an issue. I typed the minikube start command and it stuck. What should I do? Is deleting minikube the only solution?
Restarting existing docker container for "minikube"
You have provided too little information to conclusively solve your problem. But one way is to actually delete the minikube and restart. You can see this similar question. Make sure that you have proper privileges to run docker containers.
Generally, this problem occurs quite often on Ubuntu. You can find very extensive thread on github.
In addition to the Restarting existing docker container for "minikube", you should also get some other information (like specific error). If they are insufficient, you can always open an issue on github.
In the thread above you can find a couple of potential solutions. Here is one of them:
When I run minikube --start --driver=docker --alsologtostderr, I get the same error message with "no such file or directory".
Edit: I was able to fix this by changing to .deb docker instead of snap docker.
Per https://kubernetes.io/docs/tasks/tools/install-minikube/:
"If you're using the none driver in Debian or a derivative, use the .deb packages for Docker rather than the snap package, which does not work with Minikube. You can download .deb packages from Docker."
I did $ snap remove docker, then followed these instructions:
https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository
Maybe the error messages could be amended to tell this to the user?
You can try sudo minikube delete to delete the container first,
then minikube start and see if the issue is fixed or not.

Zombie folders coming back from the dead with Docker running on WSL2

This issue is really strange, but I've no idea how to resolve it. I'm running Win10 and I've got WSL2 on it. When I start Docker with it using the "WSL 2 Based engine", it retrieves some deleted folders out of nowhere. I delete them every single time and after I've restarted my PC and started up Docker again, the folders are back.
There are absolutely no docker containers running ("docker ps" returns nothing) so it couldn't be that some rogue volume definition is being ran along with some container. The folders also only appear once I start up Docker.
The directory inside of which the zombie folders appear is also the source for a Mutagen volume when the containers are running, but as I said - no containers are running.
I ran into the same problem and resolved it by running Troubleshoot from Docker for Windows and purge data on WSL2.
I've not found a solution to selectively clear just that one folder that keeps getting raised from the dead.

Docker Compose stuck downloading or pulling fs layer

I have the latest Docker for Mac installed, and I'm running into a problem where it appears that docker-compose up is stuck in a Downloading state for one of the containers:
± |master ✗| → docker-compose up --build
Pulling container (repo.io/company/container:prod)...
prod: Pulling from company/container
somehash: Already exists
somehash: Already exists
somehash: Already exists
somehash: Already exists
somehash: Pulling fs layer
somehash: Already exists
somehash: Already exists
somehash: Downloading [=================================================> ] 234.6 MB/239.3 MB
somehash: Download complete
somehash: Download complete
^^ this is literally what it looks like on my command line. Stopping and starting hasn't helped, it immediately outputs this same output.
I've tried to rm the container but I guess it doesn't yet exist, it returns the output No stopped containers. --force-recreate also gets stuck in the same place. And perhaps I'm not googling for the right terminology but I haven't found anything useful to try - any pointers?
I just needed to restart Docker.
Linux users can use sudo service docker restart.
Docker for Mac has a handy button for this in the Docker widget in the macOS toolbar:
If you happen to be using Docker Toolkit try docker-machine restart.
I faced the same problem! Restarting the service didn't help, downloading again didn't help. It used to get stuck at random instances leaving me with no option but to kill the pull request.
One thing which worked for me was to download 1 file at a time. For Ubuntu users, you can use the following steps:
Stop the service:
sudo service docker stop
Start docker with max concurrent download set as 1:
sudo dockerd --max-concurrent-downloads 1
Download the required image:
sudo docker pull <image_name>
Download images, after that stop the terminal and start the daemon again as it was earlier.
sudo service docker start
I had the similar situation this morning where my network suddenly went down and I was forced to power cycle the modern, while docker-compose was still in the middle of downloading stuff from docker hub.
Yes, bouncing the docker daemon process seems to resolve this.
For Linux users - do sudo service docker restart to fix it.
Go to the Docker Preferences from its menu bar icon. Within there is a "bug" icon. Click on that and then "clean / Purge data"
I'm running OSX and restarting Docker for Mac didn't help. Neither did a full restart or upgrading VirtualBox. What did work was turning my wifi interface on and off every time it got stuck. I had to do this repeatedly, but it eventually downloaded the entire image.
Directly download the necessary images using docker, e.g.
docker pull company/container
and then run
docker-compose up
again. Worked for me on MacOS.
I found a possible workaround.
I have my docker engine installed in a Ubuntu 18.04 Snap Environment.
I discovered searching in some forums that users relate this behaviors to limitation in the download bandwith.
So in the picture below you are going to watch that the components was stucked
Part of the Downloads stucked and finally I cancelled the process CTRL + C
I added two parameters or flags in the configuration file that controls the docker daemon behavior: max-concurrent-downloads 1 and max-concurrent-uploads 1
In my case remember, i am working in a snap environment. This file is located in this directory: /var/lib/docker/current/config/daemon.json
REMEMBER TO STOP ALL DOCKER PROCESS BEFORE THE FILE MODIFICATION, AND CREATE BACKUP OF THE FILE
Add the two lines in the picture. This is going to help you to limit the downloads to only one by one
This is the process that helped me to resolve this problem.
Download Succesfull
I had this issue in my VirtualBox when doing a docker pull on the image but it got stuck at a specific position and never moved from there. So, the issue was due to the network adapters in my VM. I was using NAT by default. When I switched it to "Bridged adapter", the issue went away.
I had a similar problem on docker for windows for a couple of days and when I tried to connect to the virtual machine (via Hyper-V Manager) the downloads started speeding along. I have no idea why but it worked for me...
Completely remove docker
Install docker again
It should work now
I tried to restar docker, update docker, but didnt help

Unable to run rabbitmq using marathon mesos

I am unable to run rabbitmq using marathon/mesos framework. I have tried it with rabbitmq images available in docker hub as well as custom build rabbitmmq docker image. In the mesos slave log I see the following error:
E0222 12:38:37.225500 15984 slave.cpp:2344] Failed to update resources for container c02b0067-89c1-4fc1-80b0-0f653b909777 of executor rabbitmq.9ebfc76f-ba61-11e4-85c9-56847afe9799 running task rabbitmq.9ebfc76f-ba61-11e4-85c9-56847afe9799 on status update for terminal task, destroying container: Failed to determine cgroup for the 'cpu' subsystem: Failed to read /proc/13197/cgroup: Failed to open file '/proc/13197/cgroup': No such file or directory
On googling I could find one hit as follows
https://github.com/mesosphere/marathon/issues/632
Not sure if this is the issue even I am facing. Anyone tried running rabbitmq using marathon/mesos/docker?
Looks like the process went away (likely crashed) before the container was set up. You should check stdout and stderr to see what happened, and fix the root issue.
"cmd": "", is the like'y culprit. I'd look at couchbase docker containers for a few clues on how to get it working.

Resources