Resize disk usage of a Docker container - docker

Every Docker container will be configured with 10 GB disk space, which is the default configuration of devicemapper in CentOS. So how can I configure every container newly created with more than 10 GB disk space in default? (The host server is installed with CentOS 6 and Docker 1.7.1)

Yes you can. Use the dm.basesize attribute when starting the Docker daemon. For example:
docker daemon --storage-opt dm.basesize=50G ...
More info can be found in the official docs.

(optional) If you have already downloaded any image via docker pull you need to clean them first - otherwise they won't be resized
docker rmi your_image_name
Edit the storage config
vi /etc/sysconfig/docker-storage
There should be something like DOCKER_STORAGE_OPTIONS="...", change it to DOCKER_STORAGE_OPTIONS="... --storage-opt dm.basesize=100G"
Restart the docker deamon
service docker restart
Pull the image
docker pull your_image_name
(optional) verification
docker run -i -t your_image_name /bin/bash
df -h
I was struggling with this a lot until I found out this link http://www.projectatomic.io/blog/2016/03/daemon_option_basedevicesize/
turns out you have to remove/pull image after enlarging the basesize.

For those who use Mac, here's an easier solution:
Click "Preference" from Docker icon in the status bar:
Then navigate to "Disk" tab, adjust the disk image size with the slider. Docker will take a moment to restart.
That's it.

Above answers properly suggest we have to edit dm.basesize attribute of devicemapper,
but proposed solutions are out of date or simply does not work in my case.
First make sure your storage dirver is a devicemapper with:
docker info | grep "Storage Driver"
you can also check current max-size of container with: (default 10 gb)
docker info | grep "Base Device Size"
From devicemapper documentation
1) edit dm.basesize in etc/docker/daemon.json file, or create new one if does not exist
{
"storage-opts": [
"dm.basesize=30G"
]
}
2) restart docker deamon
sudo systemctl stop docker
sudo systemctl start docker
3) run again command below to check if size changed
docker info | grep "Base Device Size"
4) Its important to update current images so change is applied

Related

DOCKER ON UBUNTU : failed to register layer: Error processing tar file(exit status 1):

" write /opt/gitlab/embedded/share/terminfo/n/ntconsole-25-w: no space left on device."
but the strange thing is, space is available in system.
clean up space by using
docker system prune
Clean your system by removing container and images
$ docker rm $(docker ps -aq)
This will remove containers
$ docker rmi $(docker images -q)
This will remove the images
Where is your docker installation? /usr?
Where are your Docker Volume data stored currently? /usr?
By default with docker installation the volume data is stored usually on a system directly
To change that configure data-root in your daemon.json
once you change is no need to restart docker, just reload the daemon. systemctl daemon-reload
in case if you have running containers and do now want to stop them while the deamon reloads, then set up live-restore property also in daemon.json and reload the daemon.

dockerd --max-concurrent-downloads 1 command not found [duplicate]

I'm working with a poor internet connection and trying to pull and run a image.
I wanted to download one layer at a time and per documentation tried adding a flat --max-concurrent-downloads like so:
docker run --rm -p 8787:8787 -e PASSWORD=blah --max-concurrent-downloads=1 rocker/verse
But this gives an error:
unknown flag: --max-concurrent-downloads See 'docker run --help'.
I tried typing docker run --help and interestingly did not see the option --max-concurrent-downloads.
I'm using Docker Toolbox since I'm on a old Mac.
Over here under l there's an option for --max-concurrent-downloads however this doesn't appear on my terminal when typing docker run --help
How can I change the default of downloading 3 layers at a time to just one?
From the official documentation: (https://docs.docker.com/engine/reference/commandline/pull/#concurrent-downloads)
You can pass --max-concurrent-downloads during a pull operation.
You can set --max-concurrent-downloads with the dockerd command.
If you're using the docker Desktop GUI for Mac or Windows:
You can edit the .json file directly in docker engine settings:
This setting needs to be passed to dockerd when starting the daemon, not to the docker client CLI. The dockerd process is running inside of a VM with docker-machine (and other docker desktop environments).
With docker-machine that is used in toolbox, you typically pass the engine flags on the docker-machine create command line, e.g.
docker-machine create --engine-opt max-concurrent-downloads=1
Once you have a created machine, you can follow the steps from these answers to modify the config of an already running machine, mainly:
SSH into your local docker VM.
note: if 'default' is not the name of your docker machine then substitute 'default' with your docker machine name $
docker-machine ssh default
Open Docker profile $ sudo vi /var/lib/boot2docker/profile
Then in that profile, you would add your --engine-opt max-concurrent-downloads=1.
Newer versions of docker desktop (along with any Linux install) make this much easier with a configuration menu daemon -> advanced where you can specify your daemon.json entries like:
{
"max-concurrent-downloads": 1
}

Increase Docker container size from default 10GB on rhel7

When I launch a container from rhel7.3 image, the default container size is 10GB. I want to increase it to 20GB. I tried the below ways but I had no luck
1) Added "DOCKER_STORAGE_OPTIONS": "--storage-opt dm.basesize=20G" in /etc/docker/daemon.json file. /etc/docker/daemon.json file is not there by default so I had to add it and tried restarting docker. Restart fails with the below error:
"unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives don't match any configuration option: DOCKER_STORAGE_OPTIONS\n"
2) Added "dm.basesize=20G" parameter while I launch the conatiner
docker run --privileged --storage-opt "dm.basesize=20G" -d IMAGE_ID
but it fails to launch with error
"docker: Error response from daemon: Unknown option dm.basesize."
Any help on how I can achieve to launch a container with 20GB instead of the default 10GB?
Thanks,
Premchand
I changed the storage type to "Overlay" by the following steps
1) Added {"storage-driver": "overlay"} in /etc/docker/daemon.json file. This file was not there in rhel 7.3 so I added it manually.
2) Restarted docker
My issue of increasing the container volume is resolved as each container get total amount of volume available on the host.
Had the same issue as you, after a lot of research i found a simple solution:
stop the docker service:
sudo systemctl stop docker
edit your docker service file, located at:
/usr/lib/systemd/system/docker.service
find the execution line:
ExecStart=/usr/bin/dockerd
and change it to: ExecStart=/usr/bin/dockerd --storage-opt dm.basesize=20G
start docker service again:
sudo systemctl start docker
all done.
You have the correct flag, --storage-opt dm.basesize=some_size, however this is an argument that should be given to dockerd, not docker.
Try reformatting your daemon.json file to contain:
"storage-opt": [ "dm.basesize=20G" ]

Start the docker daemon without starting containers that set to restart automatically

The docker daemon isn't starting anymore on my computer (Linux / Centos 7), and I strongly suspect that a container that is set to auto-restart is to blame in this case. If I start the daemon manually, the last line I see is "Loading containers: start", and then it just hangs.
What I'd like to do is to start the daemon without starting any containers. But I can't find any option to do that. Is there any option in docker to start the daemon without also starting containers set to automatically restart? If not, is there a way to remove the containers manually that doesn't require the docker daemon running?
I wrote this little script to stop all the containers before docker is started. It requires to have jq installed.
for i in /var/lib/docker/containers/*/config.v2.json; do
touch "$i.new" && getfacl -p "$i" | setfacl --set-file=- "$i.new"
cat "$i" | jq -c '.State.Running = false' > "$i.new" && mv -f "$i.new" "$i"
done
I think we need to verify the storage driver for docker that you are using. Devicemapper is known to have some issues similar to what you are describing. I would suggest moving to overlay2 as a storage driver.
If you are not running this on a prod system, you can try to do below steps to see if the daemon is coming up or not,
Stop the daemon process
Clean the docker home directory, default is /var/lib/docker/*
You may not be able to remove everything, in that case safe bet is to stop docker from autostart ,systemctl disable docker and restart the system
Once system is up, execute step-2 again and try to restart the daemon. Hopefully everything will come up.

Error: "error creating aufs mount to" when building dockerfile

I get this error when I try to build a docker file
error creating aufs mount to /var/lib/docker/aufs/mnt
/6c1b42ce1a98b1c0f2d2a7f17c196221445f1054566065d4c607e4f1b99930eb-init:
invalid argument
What does it mean? How do I fix it?
I had some unresolved errors after removing /var/lib/docker/aufs, which a couple extra steps cleared up.
To add to #benwalther answer, since I lack the reputation to comment:
# Cleaning up through docker avoids these errors
# ERROR: Service 'master' failed to build:
# open /var/lib/docker/aufs/layers/<container_id>: no such file or directory
# ERROR: Service 'master' failed to build: failed to register layer:
# open /var/lib/docker/aufs/layers/<container_id>: no such file or directory
docker rm -f $(docker ps -a -q)
docker rmi -f $(docker images -a -q)
# As per #BenWalther's answer above
sudo service docker stop
sudo rm -rf /var/lib/docker/aufs
# Removing the linkgraph.db fixed this error:
# Conflict. The name "/jenkins_data_1" is already in use by container <container_id>.
# You have to remove (or rename) that container to be able to reuse that name.
sudo rm -f /var/lib/docker/linkgraph.db
sudo service docker start
If you try to use docker inside a persistent enable Live CD, you may encounter this error. I guess, it is due to the fact that you can't mount aufs inside overlayfs mounts, which is the persistent layer.
The solution was simply using different driver. I've used vfs in /etc/docker/daemon.json
Here it is
{
"storage-driver": "vfs"
}
I have removed /var/lib/docker/aufs/diff and got the same problem:
error creating aufs mount to /var/lib/docker/aufs/mnt/blah-blah-init: invalid argument
It solved by running the following commands:
docker stop $(docker ps -a -q);
docker rm $(docker ps -a -q);
docker rmi -f $(docker images -a -q)
AUFS is unable to mount the docker container filesystem.
This is either because: the path is already mounted - or - there's a race condition in docker's interaction with AUFS, due to the large amount of existing volumes.
To solve this, try the following:
restart the docker service or daemon and try again.
check mount for aufs mounted on any paths under /var/lib/docker/aufs/. If found, stop docker, then umount them (need sudo).
example:
mount
none on /var/lib/docker/aufs/mnt/55639da9aa959e88765899ac9dc200ccdf363b2f09ea933370cf4f96051b22b9 type aufs (rw,relatime,si=5abf628bd5735419,dio,dirperm1)
then
sudo umount /var/lib/docker/aufs/mnt/55639da9aa959e88765899ac9dc200ccdf363b2f09ea933370cf4f96051b22b9
If that doesn't work, stop docker, then sudo rm -rf /var/lib/docker/aufs. You will lose any existing stopped containers and all images. But this is just about guaranteed to solve the problem.
Unfortunately on my system I could not resolve this with the above answers. The docker administration kept remembering a certain file in the aufs layer that it couldn't reach anymore. Other solutions didn't work either. So if this is an option for you, you could try the following fix: uninstall/purge docker and docker-engine:
apt-get purge docker docker-engine
Then make sure everything from /var/lib/docker is removed.
rm -rf /var/lib/docker
After that install docker again.
I'm using Raspbian with Raspberry 4
Best way to do it..
Check your docker version with:
sudo docker info and check "Storage Driver"
sudo systemctl stop docker
sudo nano /etc/docker/daemon.json
write this code below and save it
{
"storage-driver": "vfs"
}
sudo systemctl start docker
altought vfs... has a performance issue and could not be the best choice... :)
I just had a similar issue on Lubuntu (Ubuntu 4.15.0-20-generic) with Docker CE 18.03. None of the described options helped.
It appears that latest docker versions use the overlay2 storage driver. However some applications require aufs. Thus a possible fix might be to simply use this docker guide to change the storage driver to aufs (simply replace "overlay2" with "aufs") as in this guide.
I am running a container inside another container(also installed docker in that container) and is trying to create an aufs storage on top of an overlayfs mount, which is not possible. So, I also change the host overlyfs storage to aufs. It's solve my issue. To check storage driver use below command.
docker info
The solution was simply using different driver. I've used aufs in /etc/docker/daemon.json
Here it is
{
"storage-driver": "aufs"
}
For detailed explanation read below documentation.
Docker storage documentation
A similar issue arose while I was using Docker in Windows:
ERROR: Service 'daemon' failed to build: error creating overlay mount
to /var/lib/docker/overlay2/83c98f716020954420e8b89e6074b1af6
1b2b86cd51ac6a54724ed263b3663a2-init/merged: no such file or directory
The problem occurred after having removed a volume from the image's Dockerfile, rebuilding the image and then rebooting the PC. Maybe this is a common cause?
I managed to solve the problem by clicking Docker -> Settings -> Reset -> Reset to factory defaults...
All my images were subsequently lost but that didn't matter for me. I also figured that removing the VM disk image (the path to which can be found under the Advanced tab in Settings) could solve the issue. I haven't tried this approach however.
In windows after a restart, docker machine problem is solved for me.
Use these commands:
docker-machine stop
docker-machine start
docker-compose up
I put this answer also here, as the google search lead me here since the #whitebrow's answer contains what term I searched for in google
ERROR: Service '***' failed to build: error creating overlay mount
to /var/lib/docker/overlay2/***/merged: no such file or directory
In my case, the working workaround surprisingly was to restrict the number of 'RUN' docker building commands/layers, since if the number surpassed 60 layers/commands, it always ended up with that missing 'merged' folder error, no matter what was the contents of the command, even simple command such as RUN ls -la ended up with that error, if the total number of such/any commands was higher than about 60, strange. Merged subfolder was always missing, though even when I automatically generated all the merged subfolders, always was created on the fly a new layer with a new hash, which was missing that subfolder.
I faced the same issue.I resolved it by adding the storage driver to /etc/docker/daemon.json
you can refer this link as well to see other driver options.
Visit https://docs.docker.com/storage/storagedriver/select-storage-driver/

Resources