I am trying to move the "/var/lib/docker" folder from one disk to another since that is taking up too much space. I keep running into some errors relating to permissions!
According to these questions:
How do I move a docker container's image to a persistent disk?
How to run docker LXC containers on another partition?
My disk is mounted on "/data" and I copied the "/var/lib/docker" folder to "/data/docker"
This is what I tried:
Tried out the -g flag from DOCKER_OPTS with "/data/docker"
Tried creating a symbolic link from the new disk drive
I tried doing a bind mount from /data/docker
However in all the cases, I get an error when I try to launch services inside my container about missing permissions to write to "/dev/null" (as user root).
I simply did a copy of the folder to the new disk. This copied all the permissions as well (This is an ext4 system with same filesystem level permissions as the original disk on which docker exists now).
Specs:
The fileystem I am using is aufs.
Docker version is 0.7.6
Ubuntu 12.04
How do I move the data properly? Do I need a upgrade first?
I just did the following and it seems to work well:
as root:
service docker stop
mv /var/lib/docker /data/
# reboot and get root
service docker stop
rm -rf /var/lib/docker && ln -s /data/docker /var/lib/
service docker start
To add custom startup options to docker in Debian / Ubuntu (such as using a different data directory):
Edit /lib/systemd/system/docker.service:
[Service]
EnvironmentFile=-/etc/default/docker
ExecStart=/usr/bin/docker -d $DOCKER_OPTS -H fd://
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
In /etc/default/docker set :
DOCKER_OPTS="-g /srv/docker"
In more recent Docker versions on Ubuntu you need to edit /etc/default/daemon.json:
{
"data-root": "/new/location"
}
Related
I want to dynamically get symlinks to devices created by udev running on Host in a docker container
I was able to bind the symlink to the container but it's not dynamically recreated if the device is removed (e.g: usb is disconnected)
Udev rules example:
SUBSYSTEM=="tty", ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="140c", MODE="0666", SYMLINK+="my_dir/gsm-modem0"
docker run example:
sudo docker run -v /dev/my_dir/gsm-modem0:/dev/my_dir/gsm-modem0 my_image my_script.sh
Answer:
Udev rule should symlink to a new directory:
SUBSYSTEM=="tty", ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="140c", MODE="0666", SYMLINK+="my_dir/gsm-modem0"
Running docker must contain --privileged:
sudo docker run --privileged -v /dev/my_dir:/dev/my_dir my_image my_script.sh
and my_script.sh should start by creating a new file in the created symlink directory:
mkdir -p /dev/my_dir
touch /dev/my_dir/keep
Explanation:
For some reason udev may delete the link directory if the directory is empty, and since usually /dev is a tmpfs creating new file won't survive restart. Touching a file on every run will keep the link containing directory on host and if a new link is created it will appear on the container
My server's main drive nearly full. So I move /var/lib/docker directory to second drive location /media/my-username/sec-drive/docker using
sudo -s # enter root mode
service docker stop
rsync -aXS /media/my-username/sec-drive/docker /var/lib/docker
rm -rf /var/lib/docker
ln -s /media/my-username/sec-drive/docker /var/lib/docker
serivce docker start
then I start my all docker container by using docker-compose up -d
all containers works just fine.
But when I reboot or restart docker service, one of my containers lost a bunch of files(other containsers works just fine). One of those files is libmxnet.so(filemode:777) under /opt/myproj/mxnet/
use local mxnet
RuntimeError: Cannot find the files.
List of candidates:
/opt/myproj/mxnet/libmxnet.so
/opt/myproj/mxnet/libmxnet.so
/opt/myproj/mxnet/../../build/libmxnet.so
/usr/local/nvidia/lib/libmxnet.so
/usr/local/nvidia/lib64/libmxnet.so
../../../libmxnet.so
Those files seems lost randomly. In mxnet folder __init__.py lost but __init__.pyc stays fine. That's really wired.
Then I try to remove images and containers and import again, just turn out same result.
UPDATE:
This error occurred on another server again. But this time I've reinstalled the system and haven't move docker to another drive. Seems it has nothing to do with docker directory location
You have to instruct the docker daemon that you change the folder.
In your docker.service you should add a parameter (-g):
FROM:
ExecStart=/usr/bin/docker daemon
TO:
ExecStart=/usr/bin/docker daemon -g /new/path/docker
Some references here:
https://www.rb-associates.co.uk/blog/move-var-lib-docker-to-another-directory/
https://linuxconfig.org/how-to-move-docker-s-default-var-lib-docker-to-another-directory-on-ubuntu-debian-linux
When creating volumes through the volume API, that is, as the container volume pattern is now not necessarily the best practice anymore:
# docker volume inspect test-data
[
{
"Name": "test-data",
"Driver": "local",
"Mountpoint": "/var/lib/docker/volumes/test-data/_data"
}
]
I would like to, for example, have docker volumes exist in /data (which is mounted in a different physical volume).
This is not possible to do with symbolic links, it is possible to do with bind mounts, but would I'm wondering if there is some configuration in Docker to change the default location for each separate volume.
You can change where Docker stores its files including volumes by changing one of its startup parameters called --data-root.
If you're using systemd for service management, the file is usually located at /lib/systemd/system/docker.service. Edit the file as such:
# Old - taken from the generated docker.service file in Ubuntu 16.04's docker.io package
ExecStart=/usr/bin/dockerd -H fd:// $DOCKER_OPTS
# New
ExecStart=/usr/bin/dockerd --data-root /new_location/ -H fd:// $DOCKER_OPTS
Alternatively, you can edit the Docker daemon configuration file which defaults to /etc/docker/daemon.json.
Restart the Docker daemon and your volumes will be under /new_location/volumes/{volume_name}/_data
Note: be careful in production and also locally! You also have to move the existing data from /var/lib/docker/ to the new location for your docker install to work as expected.
You can use symlinks from the new location if you want specific folders to be in specific place.
2017: with 17.05.0-ce (2017-05-04), the PR 28696 deprecates --graph flag in favor or --data-root: commit 1ecaed0
The name "graph" is a legacy term from long ago when there used to be a directory at the default location /var/lib/docker/graph.
However, the flag would indicate the path of the parent directory of the "graph" directory which contains not only image data but also data for volumes, containers, and networks.
In the most recent version of docker, this directory also contains swarm cluster state and node certificates.
With issue 5922 and PR 5978, the documentation has been updated.
Example:
ExecStart=/usr/bin/dockerd -H fd:// --data-root=/mnt/ssd/lib/docker
2016 (now deprecated)
I only know of a docker option to change /var/lib/docker itself, not its subfolders (part of its "graph" used by a docker daemon storage driver)
See docker daemon "Miscellaneous options":
Docker supports softlinks for the Docker data directory (/var/lib/docker) and for /var/lib/docker/tmp.
The DOCKER_TMPDIR and the data directory can be set like this:
DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/docker daemon -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1
# or
export DOCKER_TMPDIR=/mnt/disk2/tmp
/usr/local/bin/docker daemon -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log
As mentioned in "Where are docker images stored on the host machine?" (and that would apply also for containers/volumes):
The contents of the /var/lib/docker directory vary depending on the driver Docker is using for storage.
I successfully moved the storage location of docker by moving the content of /var/lib/docker to a new location and then place a symlink pointing to the new location (I took this solution from here https://askubuntu.com/questions/631450/change-data-directory-of-docker):
Caution - These steps depend on your current /var/lib/docker being an
actual directory (not a symlink to another location).
1) Stop docker: service docker stop. Verify no docker process is running:
ps aux | grep -i [d]ocker
2) Double check docker really isn't running. Take a look at the current docker directory:
ls /var/lib/docker/
2b) Make a backup - tar -zcC /var/lib docker >
/mnt/pd0/var_lib_docker-backup-$(date +%s).tar.gz
3) Move the /var/lib/docker directory to your new partition:
mv /var/lib/docker /mnt/pd0/docker
4) Make a symlink: ln -s /mnt/pd0/docker /var/lib/docker
5) Take a peek at the directory structure to make sure it looks like
it did before the mv: ls /var/lib/docker/ (note the trailing slash)
6) Start docker back up service docker start
7) restart your containers (resolve the symlink)
Worked for me on Ubuntu 18.04.1 LTS on an Azure VM with Docker 18.09.2
If you're on Fedora (tested on 32) just change or add the --data-root flag with your desired path to the OPTIONS variable on /etc/sysconfig/docker, this is the environment file used by systemd to start the dockerd service.
I'm new to docker, and I'm trying mount the root directory of docker container as a NFS mount point.
for example, I had a NFS mount point test:/home/user/3243, and I'm trying:
docker run -it -v "test:/home/user/3243":/ centos7 /bin/bash
absolutely, it's failed. So I tried this:
mount -t nfs test:/home/user/3243 /mnt/nfs/3243
docker run -it -v /mnt/nfs/3243:/ centos7 /bin/bash
but failed again, so how to do this? Could it be worked out?
A couple of issues here:
You cannot mount to the root directory of a container. So docker run -v /foo:/ will never work.
With the syntax of your first attempt, -v test:/foo:bar, Docker would see this as wanting to create a "named" volume called "test".
You should be able to first do the NFS mount, then do docker run -v /mnt/nfs/3243:/foo to have the nfs path mounted to /foo.
But again, you can't mount to /.
That is currently discussed (since mid 2014) in issue 4213.
One recent workaround by Jeroen van Bemmel (jbemmel) was:
It appears that NFS functionality depends on the underlying storage driver ( aufs, devicemapper, etc. ), as well as the sharing of file handles between processes ( see blog post "docker: devicemapper fix for “device or resource busy” (EBUSY)") i.e. 'unshare' may have an impact on NFS mounts.
I've moved away from using the 'MOUNTPOINT=/vm/nfs' as I am not sure if that event is even emitted.
Instead I created an upstart file like this:
cat > /etc/init/ecdn.conf << EOF
description "eCDN container"
author "Jeroen van Bemmel"
# mounted MOUNTPOINT=/vm/nfs doesn't seem to work, at least not the first time
start on started docker and virtual-filesystems
stop on starting rc RUNLEVEL=[016]
respawn
script
exec /usr/bin/docker start -a ecdn
end script
pre-stop script
/usr/bin/docker stop ecdn
# dont /usr/bin/docker rm ecdn
end script
EOF
and then create the container like this:
script -c "docker create -it --name='ecdn' --volume /vm:/usr/share/nginx/html/vm:ro image/name"
So, despite Docker 1.3 now allowing easy access to external storage on OSX through boot2docker for files in /Users/, I still need to access files not in /Users/. I have a settings file in /etc/settings/ that I'd like my container to have access to. Also, the CMD in my container writes logs to /var/log in the container, which I'd rather have it write to /var/log on the host. I've been playing around with VOLUME and passing stuff in with -v at run, but I'm not getting anywhere. Googling hasn't been much help. Can someone who has this working provide help?
As boot2docker now includes VirtualBox Guest Additions, you can now share folders on the host computer (OSX) with guest operating systems (boot2docker-vm). /Users/ is automatically mounted but you can mount/share custom folders. In your host console (OSX) :
$ vboxmanage sharedfolder add "boot2docker-vm" --name settings-share --hostpath /etc/settings --automount
Start boot2docker and ssh into it ($boot2docker up / $boot2docker ssh).
Choose where you want to mount the "settings-share" (/etc/settings) in the boot2docker VM :
$ sudo mkdir /settings-share-on-guest
$ sudo mount -t vboxsf settings-share /settings-share-on-guest
According that /settings is the volume declared in the docker container add -v /settings-share-on-guest:/settings to the docker run command to mount the host directory settings-share-on-guest as a data volume.
Works on Windows, not tested on OSX but should work.