To use --squash feature of Docker, you need to enable the experimental features of the Docker daemon. How this can be done on Arch Linux?
Thanks.
This is how I got it to work in ArchLinux with systemd
/etc/docker/daemon.json
added the json blob below
{
"experimental": true
}
then restart the service
systemctl restart docker.service
For confirmation, run docker version and you'll see the following line in the output
Client: Docker Engine - Community
....
Experimental: true
....
On Arch Linux, you can add --experimental=true to the end of the ExecStart=/usr/bin/dockerd -H fd:// line in the /usr/lib/systemd/system/docker.service file in order to enable the experimental feature.
Then, you need to reload and restart the Docker daemon:
systemctl daemon-reload
systemctl restart docker.service
With recent docker versions neither of the methods works any more. See https://github.com/docker/cli/issues/947
As a summary, there is docker and docker cli. Both are not the same. Experimental mode must be enabled for docker cli. Thus one has to edit the file $HOME/.docker/config.json.
Related
I am using Docker 18.09 and I am trying to build some images for my work. The problem is that the images are always inside the root directory, precisely the /var/lib/docker/overlay2 are in /var/docker/. As there isn't enough space in my root directory, so I want to change this default directory to my other disk but none of the solutions I have looked upon the internet have worked for me.
I have gone through these already but none of them are working:
https://forums.docker.com/t/how-do-i-change-the-docker-image-installation-directory/1169
https://medium.com/#ibrahimgunduz34/how-to-change-docker-data-folder-configuration-33d372669056
How to change the docker image installation directory?
The default directory to store docker related data (containers, images and so on) is /var/lib/docker.
To override this default location use -g option.
While starting docker deamon use -g option.
dockerd -g /mnt/path/to/docker/dir
In your case, the best option is to attach some external storage to machine at some mountpoint. And mention that mountpoint in -g option.
Hope this helps.
Update:
-g option is deprecated. Use --data-root option. Check this.
I also faced similar issue with containerd used with rke2, I am using RKE2 with containerd and its storing all the images on /var/lib/rancher/rke2, which was causing VMs'/' partition getting full all the time.
I wanted to move containerd root directory to custom directory
I changed rke agent start command in rke service file and it worked.
Created a new FS /containerdata/containerd and configured rke service to point to this directory for containerd data
change /usr/local/lib/systemd/system/rke2-agent.service
ExecStart=/usr/local/bin/rke2 agent --data-dir /containerdata/containerd/
Reload and retart rke2-agent.service
systemctl daemon-reload
systemctl restart rke2-agent.service
This may cause pods to be unstable, but system gets stable over time.
I used the following technique when i too ran into this problem and using a different filesystem which had a larger size was the only available option left.
Stop the docker service.
sudo systemctl stop docker.service
Edit the docker.service file to include the new directory in the ExecStart line as below.
ExecStart=/usr/bin/dockerd -g /u01/docker -H fd:// --containerd=/run/containerd/containerd.sock
Reload daemon
sudo systemctl daemon reload
Restart docker
sudo systemctl start docker.service
Note: the docker.service file path is shown when you ran the systemctl stop docker.service
A much simpler and straightforward way out for this problem is to simply create a symbolic link to the new path once you move the existing folder like below.
sudo systemctl stop docker.service
sudo mv /var/lib/docker/ /
sudo ln -s / /var/lib/docker
sudo systemctl start docker.service
I'm trying to configure docker (version 17.03.1-ce) in ubuntu 16.04 using configuration file /etc/docker/daemon.json to add an host:
{
"debug": true,
"hosts": ["tcp://0.0.0.0:1234", "unix:///var/run/docker.sock"],
"dns" : ["8.8.8.8","8.8.4.4"]
}
when I try to restart docker.. it fails
#service docker restart
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
watching on systemctl status docker.service:
Starting Docker Application Container Engine...
docker-slave-ubuntu-build dockerd[24806]: unable to configure the Docker daemon with file /etc/docker/daemon.json:
the following directives are specified both as a flag and in the configuration file:
hosts: (from flag: [fd://], from file: [tcp://0.0.0.0:4243 unix:///var/run/docker.sock])
Where I can remove the mentioned flag ? I have to modify maintainer's script ?
For systemd, my preferred method is to deploy a simple override file (you may need to first create the directory):
$ cat /etc/systemd/system/docker.service.d/override.conf
# Disable flags to dockerd, all settings are done in /etc/docker/daemon.json
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd
This removes the -H ... default flag from dockerd along with any other options and lets you manage docker from the daemon.json file. This also allows docker to make changes to their startup scripts as long as they don't modify the ExecStart and you'll continue to receive those changes without maintaining your own copy of the docker.service.
After creating this file, run systemctl daemon-reload; systemctl restart docker.
It looks like this is an issue merging configuration from both the command line and configuration file. The default systemd unit file is specifying -H fd:// and it conflicts with your tcp://0.0.0.0:1234 and unix:///var/run/docker.sock.
There are a number of GitHub issues on the subject:
https://github.com/moby/moby/issues/22339
https://github.com/moby/moby/issues/21559
https://github.com/moby/moby/issues/25471
https://github.com/moby/moby/pull/27473
They don't seem to consider this a bug. But it is definitely an annoyance. A workaround is to copy the default unit file and remove the -H fd:// from it:
$ sudo cp /lib/systemd/system/docker.service /etc/systemd/system/
$ sudo sed -i 's/\ -H\ fd:\/\///g' /etc/systemd/system/docker.service
$ sudo systemctl daemon-reload
$ sudo service docker restart
I found this on the Docker docs and it worked on Docker 18.09.1 and Centos 8:
To work around this problem, create a new file /etc/systemd/system/docker.service.d/docker.conf with the following contents, to remove the -H argument that is used when starting the daemon by default.
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd
Then reload
systemctl daemon-reload
The reason is:
Docker listens on a socket by default. On Debian and Ubuntu systems using systemd, this means that a host flag -H is always used when starting dockerd. If you specify a hosts entry in the daemon.json, this causes a configuration conflict (as in the above message) and Docker fails to start.
Here is the link: https://docs.docker.com/config/daemon/#troubleshoot-conflicts-between-the-daemonjson-and-startup-scripts
In my case I tried to add both the daemon.json under /etc/docker and a *.conf file under /etc/systemd/system/docker.service.d.
It turned out it was enough to have a .conf file only (in my case called override.conf):
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375
this way I could expose the docker socket.
I had copied the daemon.json from a website. After running
sudo systemctl stop docker
/usr/sbin/dockerd
it showed me a better error message that stated that I had a strange invisible character in the daemon.json file
I get following error when starting Docker daemon from command line:
Error starting daemon: error while opening volume store metadata database: timeout
OS is Linux.
Any pointer how can I resolve this?
(Google search didn't yield anything)
It depends on your exact Linux distro, and docker version.
See for instance issue 26022: it has the same error message on Fedora after a
yum -y install docker-engine-1.13.1-1.el7.centos
Try and follow again the full installation procedure for your exact distro.
After investigating a lot and trying many commands, this has worked for me:
*Do not use '&&' to make the most compact command or otherwise it will not work.
sudo rm /var/run/docker.pid
sudo systemctl stop docker.socket
sudo systemctl stop docker
systemctl start docker
systemctl enable docker
sudo systemctl start docker
If this error appears ( running $ docker [command] ):
Cannot connect to the Docker daemon at unix:///home/mg/.docker/desktop/docker.sock. Is the docker daemon running?
The first thing you should do is to have Docker Desktop installed on your pc, of which you can get here https://docs.docker.com/desktop/windows/wsl/
You should also enable wsl2, Just going through the documentation from the link above should be enough.
Also make sure Settings > General > Use the WSL 2 based engine... box is checked.
REFERENCE https://stackoverflow.com/a/72890783/21061651
Looking at docs there is no instruction on how to run it behind a proxy.
https://docs.docker.com/installation/ubuntulinux/
Reading on forums, the instruction is to update /etc/default/docker to export the proxy setup.
export http_proxy="http://127.0.0.1:3128/"
export https_proxy="http://127.0.0.1:3128/"
export HTTP_PROXY="http://127.0.0.1:3128/"
export HTTPS_PROXY="http://127.0.0.1:3128/"
Then we restart/start docker
sudo service docker start
Inside a container, if I run 'apt-get', npm install, bower install I cant get through the proxy.
Not sure what I m missing.
Ubuntu 14.04 LTS
For Ubuntu 14.04 LTS who uses SysVinit, you should modify /etc/default/docker file:
# cat /etc/default/docker
# Docker Upstart and SysVinit configuration file
#
# THIS FILE DOES NOT APPLY TO SYSTEMD
#
# Please see the documentation for "systemd drop-ins":
# https://docs.docker.com/engine/articles/systemd/
#
.......
# If you need Docker to use an HTTP proxy, it can also be specified here.
export http_proxy="http://web-proxy.corp.xxxxxx.com:8080/"
export https_proxy="https://web-proxy.corp.xxxxxx.com:8080/"
......
Then restart docker:
service docker restart
Ubuntu 16.04 LTS / Ubuntu 18.04 LTS
For Ubuntu 16.04 LTS who uses Systemd, you can follow this post:
(1) Create a systemd drop-in directory:
mkdir /etc/systemd/system/docker.service.d
(2) Add proxy in /etc/systemd/system/docker.service.d/http-proxy.conf file:
# cat /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=https://web-proxy.corp.xxxxxx.com:8080/"
Environment="HTTPS_PROXY=https://web-proxy.corp.xxxxxx.com:8080/"
Environment="NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com"
(3) Flush changes:
systemctl daemon-reload
(4) Restart Docker:
systemctl restart docker
Official Reference
For Ubuntu 14.04.2 LTS
Linux vagrant-ubuntu-trusty-64 3.13.0-54-generic #91-Ubuntu SMP Tue May 26 19:15:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Edit you /etc/default/docker file
sudo vim /etc/default/docker
Add this line at the bottom:
export http_proxy="http://PROXY_IP:PROXY_PORT"
Restart the docker service
sudo service docker restart
You can try to add lines in /etc/environment:
https_proxy="http://127.0.0.1:3128"
http_proxy="http://127.0.0.1:3128"
ftp_proxy="http://127.0.0.1:3128"
no_proxy="127.0.0.1/8, localhost, 192.168.0.0/16"
it will be useful for all services on your Linux system
Then edit /lib/systemd/system/docker.service.
In the end of the [Service] section, add line:
EnvironmentFile=/etc/environment
And then:
sudo systemctl daemon-reload
sudo systemctl restart docker.service
systemctl will have to installed, which can be problematic.
In case /etc/systemd/system/docker.service.d/http-proxy.conf or /etc/default/docker solution does not work for you, simply use the below command:
docker build [OPTIONS] PATH --build-arg http_proxy=http://your.proxy:port --build-arg https_proxy=http://your.proxy:port --build-arg no_proxy=.internal.domain,localhost,127.0.0.1
In Ubuntu 14.04 LTS:
An interesting issue about the HTTP_PROXY, HTTPS_PROXY is that: if your password has a special char like "$", "%", then it will not be processed correctly by the docker daemon when you execute command like: dock run xxx, you will encounter error. Then you can try to set the special char to others, good luck.
According to the Docs
Add to ~/.docker/config.json proxy configuration
{
"proxies":
{
"default":
{
"httpProxy": "http://127.0.0.1:3001",
"noProxy": "*.test.example.com,.example2.com"
}
}
}
You should replace 127.0.0.1 to your host IP or some public accessible IP
This is an old thread but none of the solutions here worked for me because I am not running docker desktop. What worked is the one in the documentation. You need to make a file /etc/systemd/system/docker.service.d/http-proxy.conf and specify proxies there
[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80"
From what I can tell, docker images are installed to /var/lib/docker as they are pulled. Is there a way to change this location, such as to a mounted volume like /mnt?
With recent versions of Docker, you would set the value of the data-root parameter to your custom path, in /etc/docker/daemon.json
(according to https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file).
With older versions, you can change Docker's storage base directory (where container and images go) using the -goption when starting the Docker daemon. (check docker --help).
You can have this setting applied automatically when Docker starts by adding it to /etc/default/docker
Following advice from comments I utilize Docker systemd documentation to improve this answer.
Below procedure doesn't require reboot and is much cleaner.
First create directory and file for custom configuration:
sudo mkdir -p /etc/systemd/system/docker.service.d
sudo $EDITOR /etc/systemd/system/docker.service.d/docker-storage.conf
For docker version before 17.06-ce paste:
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --graph="/mnt"
For docker after 17.06-ce paste:
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --data-root="/mnt"
Alternative method through daemon.json
I recently tried above procedure with 17.09-ce on Fedora 25 and it seem to not work. Instead of that simple modification in /etc/docker/daemon.json do the trick:
{
"graph": "/mnt",
"storage-driver": "overlay"
}
Despite the method you have to reload configuration and restart Docker:
sudo systemctl daemon-reload
sudo systemctl restart docker
To confirm that Docker was reconfigured:
docker info|grep "loop file"
In recent version (17.03) different command is required:
docker info|grep "Docker Root Dir"
Output should look like this:
Data loop file: /mnt/devicemapper/devicemapper/data
Metadata loop file: /mnt/devicemapper/devicemapper/metadata
Or:
Docker Root Dir: /mnt
Then you can safely remove old Docker storage:
rm -rf /var/lib/docker
For new docker versions we need to use data-root as graph is deprecated in v17.05.0: official deprecated docs
Edit /etc/docker/daemon.json (if it doesn’t exist, create it) and include:
{
"data-root": "/new/path/to/docker-data"
}
Then restart Docker with:
sudo systemctl daemon-reload
sudo systemctl restart docker
A more detailed step-by-step explanation (including moving data) using Docker Storage with data-root can be found in: Blog post
In case of Windows a similar post Windows specific
Much easier way to do so:
Stop docker service
sudo systemctl stop docker
Move existing docker directory to new location
sudo mv /var/lib/docker/ /path/to/new/docker/
Create symbolic link
sudo ln -s /path/to/new/docker/ /var/lib/docker
Start docker service
sudo systemctl start docker
Since I haven't found the correct instructions for doing this in Fedora (EDIT: people pointed in comments that this should also work on CentOS and Suse) (/etc/default/docker isn't used there), I'm adding my answer here:
You have to edit /etc/sysconfig/docker, and add the -g option in the OPTIONS variable. If there's more than one option, make sure you enclose them in "". In my case, that file contained:
OPTIONS=--selinux-enabled
so it would become
OPTIONS="--selinux-enabled -g /mnt"
After a restart (systemctl restart docker) , Docker should use the new directory
Don't use a symbolic Link to move the docker folder to /mnt (for example).
This may cause in trouble with the docker rm command.
Better use the -g Option for docker.
On Ubuntu you can set it permanently in /etc/default/docker.io. Enhance or replace the DOCKER_OPTS Line.
Here an example:
`DOCKER_OPTS="-g /mnt/somewhere/else/docker/"
This solution works on Red Hat 7.2 & Docker 1.12.0
Edit the file
/lib/systemd/system/docker.service in your text editor.
add -g /path/to/docker/ at the end of ExecStart directive. The complete line should look like this.
ExecStart=/usr/bin/dockerd -g /path/to/docker/
Execute the below command
systemctl daemon-reload
systemctl restart docker
Execute the command to check docker directory
docker info | grep "loop file\|Dir"
If you have /etc/sysconfig/docker file in Red Hat or docker 1.7.1 check this answer.
In CentOS 6.5
service docker stop
mkdir /data/docker (new directory)
vi /etc/sysconfig/docker
add following line
other_args=" -g /data/docker -p /var/run/docker.pid"
then save the file and start docker again
service docker start
and will make repository file in /data/docker
Copy-and-paste version of the winner answer :)
Create this file with only this content:
$ sudo vi /etc/docker/daemon.json
{
"graph": "/my-docker-images"
}
Tested on Ubuntu 16.04.2 LTS in docker 1.12.6
The official way of doing this based on this Post-installation steps for Linux guide and what I found while web-crawling is as follows:
Override the docker service conf:
sudo systemctl edit docker.service
Add or modify the following lines, substituting your own values.
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --graph="/mnt/docker"
Save the file. (It creates: /etc/systemd/system/docker.service.d/override.conf)
Reload the systemctl configuration.
sudo systemctl daemon-reload
Restart Docker.
sudo systemctl restart docker.service
After this if you can nuke /var/lib/docker folder if you do not have any images there you care to backup.
For Debian/Ubuntu or Fedora, you can probably use the other answers. But if you don't have files under /etc/default/docker or /etc/sysconfig/docker, and your system is running systemd, you may want to follow this answer by h3nrik. I am using Arch, and this works for me.
Basically, you need to configure systemd to read the new docker image location as an environment variable, and pass that environment variable into the Docker daemon execution script.
For completeness, here is h3nrick's answer:
Do you have a /lib/systemd/system/docker.service file?
If so, edit it so that the Docker service uses the usual /etc/default/docker as an environment file: EnvironmentFile=-/etc/default/docker.
In the /etc/default/docker file then add DOCKER_OPTS="-g /home/rseixas/Programs/Docker/images".
At the end just do a systemctl daemon-reload && systemctl restart docker.
For further information please also have a look at the documentation.
As recommneded by #mbarthelemy this can be done via the -g option when starting the docker daemon directly.
However, if docker is being started as a system service, it is not recommended to modify the /etc/default/docker file. There is a guideline to this located here.
The correct approach is to create an /etc/docker/daemon.json file on Linux (or Mac) systems or %programdata%\docker\config\daemon.json on Windows. If this file is not being used for anything else, the following fields should suffice:
{
"graph": "/docker/daemon_files"
}
This is assuming the new location where you want to have docker persist its data is /docker/daemon_files
A much simpler solution is to create a soft link point to whatever you want, such as
link -s /var/lib/docker /mnt/whatever
It works for me on my CentOS 6.5 server.
I was having docker version 19.03.14. Below link helped me.
Check this Link
in /etc/docker/daemon.json file I added below section:-
{
"data-root": "/hdd2/docker",
"storage-driver": "overlay2"
}
On openSUSE Leap 42.1
$cat /etc/sysconfig/docker
## Path : System/Management
## Description : Extra cli switches for docker daemon
## Type : string
## Default : ""
## ServiceRestart : docker
#
DOCKER_OPTS="-g /media/data/installed/docker"
Note that DOCKER_OPTS was initially empty and all I did was add in the argument to make docker use my new directory
On Fedora 26 and probably many other versions, you may encounter an error after moving your base folder location as described above. This is particularly true if you are moving it to somewhere under /home. This is because SeLinux kicks in and prevents the docker container from running many of its programs from under this location.
The short solution is to remove the --enable-selinux option when you add the -g parameter.
On an AWS Ubuntu 16.04 Server I put the Docker images on a separate EBS, mounted on /home/ubuntu/kaggle/, under the docker dir
This snippet of my initialization script worked correctly
# where are the images initially stored?
sudo docker info | grep "Root Dir"
# ... not where I want them
# modify the configuration files to change to image location
# NOTE this generates an error
# WARNING: Usage of loopback devices is strongly discouraged for production use.
# Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
# see https://stackoverflow.com/questions/31620825/
# warning-of-usage-of-loopback-devices-is-strongly-discouraged-for-production-use
sudo sed -i ' s##DOCKER_OPTS=.*#DOCKER_OPTS="-g /home/ubuntu/kaggle/docker"# ' /etc/default/docker
sudo chmod -R ugo+rw /lib/systemd/system/docker.service
sudo cp /lib/systemd/system/docker.service /etc/systemd/system/
sudo chmod -R ugo+rw /etc/systemd/system/
sudo sed -i ' s#ExecStart.*#ExecStart=/usr/bin/dockerd $DOCKER_OPTS -H fd://# ' /etc/systemd/system/docker.service
sudo sed -i '/ExecStart/a EnvironmentFile=-/etc/default/docker' /etc/systemd/system/docker.service
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo docker info | grep "Root Dir"
# now they're where I want them
For Mac users in the 17.06.0-ce-mac19 version you can simply move the Disk Image location from the user interface in the preferences option Just change the location of the disk image and it will work (by clicking Move disk Image) and restarting the docker. Using this approach I was able to use my external hardisk for storing docker images.
For Those looking in 2020. The following is for Windows 10 Machine:
In the global Actions pane of Hyper-V Manager click Hyper-V
Settings…
Under Virtual Hard Disks change the location from the
default to your desired location.
Under Virtual Machines change the
location from the default to your desired location, and click apply.
Click OK to close the Hyper-V Settings page.
This blog post helps me
Here are the steps to change the directory even after you’ve created Docker containers etc.
Note, you don’t need to edit docker.service or init.d files, as it will read the change from the .json file mentioned below.
Edit /etc/docker/daemon.json (if it doesn't exist, create it)
Add the following
{
"data-root": "/new/path/to/docker-data"
}
Stop docker
sudo systemctl stop docker
Check docker has been stopped
ps aux | grep -i docker | grep -v grep
Copy the files to the new location
sudo rsync -axPS /var/lib/docker/ /new/path/to/docker-data
Start Docker back up
sudo systemctl start docker
Check Docker has started up using the new location
docker info | grep 'Docker Root Dir'
Check everything has started up that should be running
docker ps
Leave both copies on the server for a few days to make sure no issues arise, then feel free to delete it.
sudo rm -r /var/lib/docker