How can I restart docker daemon running in rootless mode on Linux?
Stopping it works fine with:
docker --user stop docker.service
but starting it back again fails when using:
docker --user start docker.service
The command doesn't return anything but when checking the docker info it says:
ERROR: Cannot connect to the Docker daemon at unix:///run/user/1000/docker.sock. Is the docker daemon running?
It doesn't give any further information...
I had this error a couple of times before, when I accidentally run docker with sudo and therefore got mixed up permissions in my data-root (defined in daemon.json). But this time chowning it back to $USER didn't help with the restart. Also restarting the host machine didn't help (as it did a couple of times previously).
Ok, it seems that "userns-remap" is not compatible with rootless mode:
Rootless mode executes the Docker daemon and containers inside a user namespace. This is very similar to userns-remap mode, except that with userns-remap mode, the daemon itself is running with root privileges, whereas in rootless mode, both the daemon and the container are running without root privileges. Rootless mode does not use binaries with SETUID bits or file capabilities, except newuidmap and newgidmap, which are needed to allow multiple UIDs/GIDs to be used in the user namespace.
I was trying to fix permission issues on shared volumes by experimenting with setting UIDs/GIDs and added "userns-remap" to the ~/.config/docker/daemon.json:
{
"data-root": "/home/me/docker/image-storage",
"userns-remap": "me"
}
So deleting userns-remap from the config file fixed the restarting issue... Man, docker, at least a hint to the config file would be great... Because the userns-remap option was mentioned on some official docker doc pages I didn't even consider it as the source of the trouble in the first place.
Related
I read the Enable Live Restore, but when I tried it.
ubuntu#ip-10-0-0-230:~$ cat /etc/docker/daemon.json
{
"live-restore": true
}
I started an nginx container in detached mode.
sudo docker run -d nginx
c73a20d1bb620e2180bc1fad7d10acb402c89fed9846f06471d6ef5860f76fb5
$sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
c73a20d1bb62 nginx "nginx -g 'daemon of…" 5 seconds ago Up 4
seconds
Then I stopped the dockerd
sudo systemctl stop snap.docker.dockerd.service
and I checked that there was no container running
ps aux | grep nginx
After that, I restarted the docker service and still, there wasn't any container.
Any Idea? How this "enable live restore" works?
From the documentation, after modifying the daemon.json (adding "live-restore": true) you need to :
Restart the Docker daemon. On Linux, you can avoid a restart (and avoid any downtime for your containers) by reloading the Docker daemon. If you use systemd, then use the command systemctl reload docker. Otherwise, send a SIGHUP signal to the dockerd process.
You can also do this but it's not recommended :
If you prefer, you can start the dockerd process manually with the --live-restore flag. This approach is not recommended because it does not set up the environment that systemd or another process manager would use when starting the Docker process. This can cause unexpected behavior.
It seems that you had not done this step. You said that you've made the modification to the daemon.json and directly started a container and then stopped the dockerd.
In order to make the Live Restore functionality work follow all steps in the right order :
Modify the daemon.json by adding "live-restore": true
Reload the Docker daemon with the command :
sudo systemctl reload docker
Then try the functionality with your example (firing up a container and making the daemon unavailable).
I've tested and it works if you follow the steps in order :
Tested with Docker version 19.03.2, build 6a30dfc and Ubuntu 19.10 (Eoan Ermine)
You've installed Docker via snap : snap.docker.dockerd.service
Unfortunately, it's not recommended since snap model is not fully compatible with Docker. Furthermore, docker-snap is no longer maintained by Docker, Inc. Users encounters some issues when they installed Docker via snap see 1 2
You should delete the snap Docker installation to avoid any potential overlapping installation issues via this command :
sudo snap remove docker --purge
Then install Docker with the official way and after that try the Live Restore functionality by following the above steps.
Also be careful when restarting the daemon the documentation says that :
Live restore upon restart
The live restore option only works to restore containers if the daemon options, such as bridge IP addresses and graph driver, did not change. If any of these daemon-level configuration options have changed, the live restore may not work and you may need to manually stop the containers.
Also about downtime :
Impact of live restore on running containers
If the daemon is down for a long time, running containers may fill up the FIFO log the daemon normally reads. A full log blocks containers from logging more data. The default buffer size is 64K. If the buffers fill, you must restart the Docker daemon to flush them.
On Linux, you can modify the kernel’s buffer size by changing /proc/sys/fs/pipe-max-size.
I've followed the directions here, and everything works well until I restart my computer. After restarting, it seems like the docker daemon loses track of the Google credentials.
$ docker run --log-driver=gcplogs ...
fails with:
docker: Error response from daemon: failed to initialize logging driver: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
ERRO[0000] error waiting for container: context canceled
This is strange to me, because running $ systemctl show --property=Environment docker returns the value in my systemd configuration:
Environment=GOOGLE_APPLICATION_CREDENTIALS=/etc/path/to/application_default_credentials.json
If I $ sudo systemctl restart docker, then docker runs sucessfully and logs are sent to stackdriver. But I want this docker image to run automatically on startup, and restarting docker with sudo gets in the way.
Is there a way to initialize the docker daemon with the necessary environment variables, so gcplogs is ready on boot without restarting docker?
I had two versions of docker installed -- one through adding docker's repo to apt, and one through snap. Running
sudo systemctl list-unit-files| grep docker | grep enabled
showed two installations of docker:
docker.service enabled
snap.docker.dockerd.service enabled
Having two docker installations was causing problems for startup. I removed the snap installation, rebooted, and everything now works.
I think you may try to edit the systemd: Unit dependencies and order, let docker.service start after google-accounts-daemon.service.
You can see all the service in google vm by
sudo systemctl list-unit-files| grep google | grep enabled
And you will see
google-accounts-daemon.service enabled
google-clock-skew-daemon.service enabled
google-instance-setup.service enabled
google-network-daemon.service enabled
google-shutdown-scripts.service enabled
google-startup-scripts.service enabled
I am aware there are a lot of questions about running Docker on windows, however this question is about running the brand new Docker for Windows, on Windows.
In my case I am using Windows 10 Pro 64 bit. According to the site this version should be supported.
I have been following a tutorial I found here:
https://prakhar.me/docker-curriculum/
I also tried following the official guide of course: https://docs.docker.com/docker-for-windows/
In both tutorials I get the same error message when trying to assign a port using either the -P parameter or when trying to specify a port -p 8080:5000:
In the official guide I run docker run -d -p 80:80 --name webserver nginx and get:
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: driver failed programming external connectivity on endpoint webserver (f9946544e4c6ad2dd9cb8cbccd251e4d48254e86562bd8e6da75c3bd42c7e45a): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:80:tcp:172.17.0.2:80: input/output error.
Following the unofficial guide i run docker run -p 8888:5000 prakhar1989/catnip and get basically the same error:
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: driver failed programming external connectivity on endpoint focused_swartz (48a0c005779c6e89bf525ead2ecff44a7f092495cd22ef7d19973002963cb232): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:8888:tcp:172.17.0.2:5000: input/output error.
If I don't try to assign a port the container will run, but then I don't know how to access it.
The docker version I am running:
Docker version 1.12.3, build 6b644ec`
docker-compose version 1.8.1, build 004ddae`
docker-machine.exe version 0.8.2, build e18a919`
Any help would be very appreciated. Thank you.
Here's a new twist.
The last Windows 10 update (Fall Creators Update, 2017) has a new "feature". It automatically starts any applications that were running when you last shutdown.
This reconstitutes Docker for Windows in a bad state. That made it appear those ports were in use by something else - it was the ghost of itself. This explained why those ports were still in use even though I stopped/started my containers and even reboot!
The solution in this case is to simply restart Docker daemon.
To prevent this after the next shutdown, don't use the shutdown button. Type this instead:
shutdown /s /t 0
This bypasses the new feature.
See the answer from Jason[MS] in this thread:
https://answers.microsoft.com/en-us/insider/forum/insider_wintp-insider_perf-insiderplat_pc/programs-autostart-after-boot-in-windows-10-fall/09dd8d3e-7b36-45d1-9181-6587dd5d53ab
Here's one guy's workaround (from the end of this thread - haven't tried it myself):
http://www.icttoolbox.nl/info/stop-windows-10-creator-fall-reopening-programs-reboot/
Restarting the Docker daemon fixes this problem temporarily, but to get rid of it ultimately I had to disable Windows 10 fast startup, which is the feature #biscuit314 described.
To disable Windows 10 fast startup, get to the Control Panel > Power Options > Choose what the power buttons do > Change settings that are currently unavailable > Uncheck Turn on fast startup (recommended) and hit Save changes
This is caused by a port numbering conflict: github issue here https://github.com/docker/compose/issues/3277
Essentially the port is in use! The reason resetting worked is because it wiped other mappings off.
1) Stop all the running containers docker stop $(docker ps -a -q) then
2) Stop the Docker on your machine & restart it.
Then run the required command. This will solve the issue.
If its in windows OS, Please do restart the Docker
This has fixed the issue for me
For Linux - Debian Users,
Use docker stop $(docker ps -a -q) only when you know whether you want to stop all the containers or not.... If yes then please run docker rm $(docker ps -a -q) to remove containers ....
Then stop the docker daemon - systemctl stop docker
Then start docker daemon - systemctl start docker
Also verify whether docker daemon is up or not - service docker status
After following all above mention steps you should be fine.....
I got the same issue before on window 10.
Restart docker, it works
Try stopping docker and initiating it again on administrator mode. After it starts open power shell on administrator mode as well.
Because the error says "mkdir" maybe this will solve your problem. Im not sure but it worked for me.
In the case of using -P a port conflict does not seen to be the reason for the error once -P will chose ports randomly. The error it self wasn't quite friendly to me but because I saw the mkdir word on it I imagined it might be a permission error, thats why I restarted docker on administrator mode and started power shell on administrator mode.
I tried all the suggestions on this issue: killing all the containers, restarting Docker Desktop, disabling "Fast Startup," restarting my computer, making sure "Experimental Features" were disabled. None of that stuff worked.
I did eventually get it running. Here are some things you may want to try (because I'm not sure what actually fixed it).
Find "Docker Desktop" and right-click to "Run as Administrator..."
Pay attention to the port that it's complaining about. Some people say this could just be Docker's unfriendly way of saying "that port is in use." In my case, the port was 80. I went into the Services on Windows Pro and disabled the "World Wide Web Publishing Service" just to be safe.
If you are here because you have this issue in Visual Studio 2019:
According to this post, the VS team is preparing a fix for this issue in 16.5 version, meanwhile, you can add the property "publishAllPorts": true in your launchSettings.json, for example:
"Docker": {
"commandName": "Docker",
"launchBrowser": true,
"launchUrl": "{Scheme}://{ServiceHost}:44374", #<== Set a fixed port
"environmentVariables": {
"ASPNETCORE_URLS": "https://+:44374;https://+:5000",
"ASPNETCORE_HTTPS_PORT": "44374"
},
"publishAllPorts": true, #<== This is equivalent to the -P flag in 'docker run'
"useSSL": true
}
Notice that the property "httpPort": XYZT is not defined. Having it defined will make the workaround not work.
It worked for me with this setup:
Windows 10 1709 Build 16299.1747 with Fast Start OFF
Docker Desktop 2.2.05 (43884)
Docker Engine 19.03.8
Visual Studio 2019 Enterprise 16.5.4.
Microsoft.VisualStudio.Azure.Containers.Tools.Targets 1.10.8
I realized that the command VS was creating, contained the -p parameter twice, one with the port I specified and another with port 80, like this: -p 3010:80 -p 3010:3010.
After adding publishAllPorts it now creates the container and I can remotely debug it.
This is what worked for me after trying every thing. Seems we need to kill the running process.
Exit docker, right-click the icon Quit Docker Desktop
Open the windows task manager, find and kill the process com.docker.backend.exe
restart docker, double-click the icon to open the Docker Desktop
I am trying to run Tomcat in a Docker container with limited success. After I tried various things, I wanted to "reset" without completely deleting everything. I did stop and remove the virtual machine from the Virtualbox console. I then tried docker-machine create and docker-machine restart. My question is, if things reach a state in which the application appears to be hanging, what is the best procedure for starting from scratch that does not involve, for example, actually rebuilding the Docker container?
EDIT: All I am now asking is, given that "docker version" returns Client information but when it reaches the Server information I get the "An error occurred trying to connect" message, is what now needs to be done? What is it not connecting to? I tried with apparent success "docker-machine restart" but got no further with "docker version" after that.
First, don't delete the boot2docker VM itself (created by docker-machine)
If you want to reset, you might have to delete the container and image (quickly rebuilt with a docker build). But you can stay in the same docker-based boot2docker VM. No need for deletion.
Retrying a docker container session simply involve killing/removing the current container, and doing a new docker run.
Then, don't forget check what is not working: does a docker ps -a shows your container running? Can you access Tomcat from the boot2docker Linux host? From your actual OS host?
Based on that diagnostic and the exact content of your Dockerfile, you will be able to debug the issue.
The main issue might come from the fact docker command are executed from outside the VM.
That works only if the commands from docker-machine env <machine-name> are set.
See docker-machine env:
For cmd.exe:
$ docker-machine.exe env --shell cmd dev
set DOCKER_TLS_VERIFY=1
set DOCKER_HOST=tcp://192.168.99.101:2376
set DOCKER_CERT_PATH=C:\Users\captain\.docker\machine\machines\dev
set DOCKER_MACHINE_NAME=dev
# Run this command to configure your shell: copy and paste the above values into your command prompt.
(replace "dev" by the name of your docker machine here, probably "default")
But it is also perfectly fine to make all docker command from within the VM. No "env" to set.
Everything is on the VM (images, Dockerfile which can be on the Windows host as well, as long as it is under C:\Users\<yourLogin>, since that folder is automatically mounted as /c/Users/<yourLogin>)
According to this documentation, I should be able to add -r=false - to the docker daemon - however adding this to the DOCKER_OPTS seems to have no effect.
Rebooting causes it to restart containers that I want to run under supervision.
On ubuntu, this is in /etc/init.d/docker:
DOCKER=/usr/bin/docker
DOCKER_PIDFILE=/var/run/docker.pid
DOCKER_OPTS="-r=false"
On Ubuntu - you will want to use the service command, not init.d script.
service docker restart
to restart.
The config for this is in:
/etc/init/docker.conf
You can add -r=false in there, after the -d parameter - and it works. Ignore the /etc/init.d/docker script on ubuntu.