I followed the following tutorial to transfer and permanently move where docker saves data previously inside /usr/bin: https://linuxconfig.org/how-to-move-docker-s-default-var-lib-docker-to-another-directory-on-ubuntu-debian-linux
However upon restarting docker and rebuilding all containers, there seems to be activity in /var/run/docker/containerd/ which I was previously trying to work around. I was hoping to have all things docker saved in a specific directory not in /var/run along with my newly created docker directory to replace /usr/bin/docker
Note: df -h did in fact prove that I am out of space in the base directory where /usr/bin and /var/run exists. I am trying to navigate all docker items to a sub directory under /opt
How do I move all things Docker to a different directory?
(Answer) Found in documentation: https://docs.docker.com/config/daemon/systemd/#runtime-directory-and-storage-driver
As described in the Docker documentation, to set the docker daemon directory to <folder>:
Create /etc/docker/daemon.json with the following contents:
{
"data-root": "<folder>",
"storage-driver": "overlay2"
}
Restart the docker daemon.
Note that this will not move existing docker data over to the target folder - you will need to handle that (or start from scratch).
Related
I have a Docker container running on my PC. The main functionality of the container is to scrape data, and this accumulates 0.3GB/day. I'll only be needing this data for the last 30 days, and after this I plan to store it archived on Hard Disk Drives for historical purposes. However after few hours of trials and errors, I've failed to create a Docker Volume on another partition, and the _data folder always appears in the /var/lib/docker/volumes/<volume_name> folder, while the partition drive is always empty.
I also tried creating the volume with docker run -v , but it still creates the volume in the main volumes folder.
The operating system is Pop!_OS 20.04 LTS
I'll provide data about the partition:
I'll provide data about the partition:
In case of docker volumes, you don't have control over where docker saves it's volumes. all you can do is just to change docker root directory. so it's better to mount your new partition under a directory and then change docker root directory to this mount point. this way you can achieve what you want. also you should consider that by doing this, all of your docker data will be stored in this new partition.
for changing your docker root directory, you should first create a file named daemon.json in address below:
/etc/docker/daemon.json
and then add config below to it:
{
"data-root": "/path/to/new/directory"
}
then restart docker daemon:
systemctl restart docker
then you can run command below to check current docker root directory:
docker info
In case you are using docker volumes all your volumes data are stored in default location (/var/lib/docker) but you can change it in /etc/docker/daemon.json config file:
{
...
"data-root": "/your/path/here",
...
}
Restart docker service to apply changes.
Docker daemon configuration file documentation here.
I would like to have a Docker Volume that mounts to a container. This volume would need to be somewhere other than the default location of volumes, preferably somewhere on the Desktop. This is because I am running a web server and would like some directories to be editable by something like VSCode so I don't always have to go inside the container to edit a file. I am not going to be using Docker Compose and instead will be using a Docker File for the container. The functionality I'm going for is the following equivalent of Docker Compose, but in a Dockerfile or through docker run, whichever is easiest to accomplish:
volumes:
- <local-dir>:<container-dir>
This directory will need to be editable LIVE and using the Dockerfile ADD command will not suffice, because after building, the image gets put into a tar archive and cannot be accessed after that.
with this solution you can move even A live container to new partition:
Add a configuration file to tell the docker daemon what is the location of the data directory
Using your preferred text editor add a file named daemon.json under the directory /etc/docker. The file should have this content:
{
"data-root": "/path/to/your/docker"
}
Copy the current data directory to the new one
sudo rsync -aP /var/lib/docker/ /path/to/your/docker
Rename the old docker directory
sudo mv /var/lib/docker /var/lib/docker.old
Restart the docker daemon
sudo service docker start
resource: https://www.guguweb.com/2019/02/07/how-to-move-docker-data-directory-to-another-location-on-ubuntu/
You can mount a directory from your host inside your container when you launch the docker container, using -v or --volume
docker run -v /path/to/desktop/some-dir:/container-dir/path <docker-image>
Volumes specified in the Dockerfile, as you exemplified, will automatically create those volumes under /var/lib/docker/volumes/ every time a container is launched from that image, but it is NOT recommended have these volumes altered by non-Docker processes.
So I have this remote folder /mnt/shared mounted with fuse. It is mostly available, except there shall be some disconnections from time to time.
The actual mounted folder /mnt/shared becomes available again when the re-connection happens.
The issue is that I put this folder into a docker volume to make it available to my app: /shared. When I start the container, the volume is available.
But if a disconnection happens in between, while the /mnt/shared repo on the host machine is available, the /shared folder is not accessible from the container, and I get:
user#machine:~$ docker exec -it e313ec554814 bash
root#e313ec554814:/app# ls /shared
ls: cannot access '/shared': Transport endpoint is not connected
In order to get it to work again, the only solution I found is to docker restart e313ec554814, which brings downtime to my app, hence is not an acceptable solution.
So my questions are:
Is this somehow a docker "bug" not to reconnect to the mounted folder when it is available again?
Can I execute this task manually, without having to restart the whole container?
Thanks
I would try the following solution.
If you mount the volume to your docker like so:
docker run -v /mnt/shared:/shared my-image
I would create an intermediate directory /mnt/base/shared and mount it to docker like so:
docker run -v /mnt/base/shared:/base/shared my-image
and I will also adjust my code to refer to the new path or creating a link from /base/shared to /shared inside the container
Explanation:
The problem is that the mounted directory /mnt/shared is probably deleted on host machine, when there is a disconnection and a new directory is created after connection is back. But, the container started running with directory mapping for the old directory which was deleted. By creating an intermediate directory and mapping to it instead you avoid this mapping issue.
Another solution that might work is to mount the directory using bind-propagation=shared
e.g:
--mount type=bind,source=/mnt/shared,target=/shared,bind-propagation=shared
See docker docs explaining bind-propogation
If I use:
FROM jenkins/jenkins:lts
RUN ls -l /var/jenkins_home/; touch /var/jenkins_home/isthisworking; echo "================================"; ls -l /var/jenkins_home;
I actually see isthisworking in my final ls -l command during the building of the image. It is upon running the container that this file gets removed. Why?
Use 'USER jenkins' if you want to modify ssh resources for that user
You try to reach out ip from network that your docker container isn't part of.your host machine on your docker containers are two sparted networks
I think I know what is going on. Here is the Dockerfile I was using to figure out what was going on (copied from above):
FROM jenkins/jenkins:lts
RUN ls -l /var/jenkins_home/; touch /var/jenkins_home/isthisworking; echo "================================"; ls -l /var/jenkins_home;
As mentioned, with this file, at build time, I would see the file isthisworking but when I run the container, that file is no longer there.
So I went to jenkins/jenkins:lts github page and looked at their Dockerfile. I saw this on line 26:
# Jenkins home directory is a volume, so configuration and build history
# can be persisted and survive image upgrades
VOLUME $JENKINS_HOME
Here, $JENKINS_HOME is /var/jenkins_home/. So as a Docker noob, I asked myself what is VOLUME (I know what it is from the command line but not inside a Dockerfile)? With googling, I found this and this, which basically say:
The docker run command initializes the newly created volume with any
data that exists at the specified location within the base image.
Since that location is a docker VOLUME at that point in the Dockerfile, no matter what and how files are copied into it, at container runtime that location will be reinitialized to how the base image has it "defined."
To have the file stay when the container is run, make the modification/addition to the directory prior to making it a VOLUME.
I am trying to capture the state of a docker container as an image, in a way that includes files I have added to a volume within the container. So, if I run the original container in this way:
$ docker run -ti -v /cookbook ubuntu:14.04 /bin/bash
root#b78f3599d936:/# cd cookbook
root#b78f3599d936:/cookbook# touch foo.txt
Now, if I either export, or commit the container as a new docker image, and then run a container from the new image, then the file, foo.txt is never included in the /cookbook directory.
My question is whether there is a way to create an image from a container in a way that allows the image to include file content within its volumes.
whether there is a way to create an image from a container in a way that allows the image to include file content within its volumes?
No, because volume is designed to manage data inside and between your Docker containers, it's used to persist and share data. What's in image is usually your program(artifacts, executables, libs. e.g) with its whole environment, building/updating data to image does not make much sense.
And in docs of volumes, they told us:
Changes to a data volume will not be included when you update an image.
Also in docs of docker commit:
The commit operation will not include any data contained in volumes mounted inside the container.
Well, by putting the changes in a volume, you're excluding them from the actual container. The documentation for docker export includes this:
The docker export command does not export the contents of volumes associated with the container. If a volume is mounted on top of an existing directory in the container, docker export will export the contents of the underlying directory, not the contents of the volume.
Refer to Backup, restore, or migrate data volumes in the user guide for examples on exporting data in a volume.
This points to this documentation. Please follow the steps there to export the information stored in the volume.
You're probably looking for something like this:
docker run --rm --volumes-from <containerId> -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /cookbook
This would create a file backup.tar with the contents of the container's /cookbook directory and store it in the current directory of the host. You could then use this tar file to import it in another container.
Essentially, there are three ways to do persistence in Docker:
You can keep files in a volume, which is a filesystem managed by Docker. This is what happens in your example: because the /cookbook directory is part of a volume, your file does not get commited/exported with the image. It does however get stored in the volume, so if you remount the same volume in a different container, you will find your file there. You can list your volumes using docker volume ls. As you can see, you should probably give your volumes names if you plan to reuse them. You can mount an existing volume, or create a new one, if the name does not exist, with
docker run -v name:/directory ubuntu
You can keep files as part of the image. If you commit the container, all changes to its file hierarchy are stored in the new image except those made to mounted volumes. So if you just get rid of the -v flag, your file shows up in the commit.
You can bind mount a directory from the host machine to the container, by using the -v /hostdir:/targetdir syntax. The container then simply has access to a directory of the host machine.
Docker commit allows you to create an image from a container and its data (mounted volumes will be ignored)