My server is at 0% free space. I just deleted 100gb of data in one of my docker volumes in an actively running container.
How do I free up the space and release it to the host system so that I am not at 0%. Do I need to stop the docker container to release it?
Thanks!
If the container is still running, and deleting files doesn't show any drop in disk usage, odds are good that the process inside your container has those file handles open. The OS won't release the underlying file and reclaim the space until all processes with open file handles close those file handles, or the process exits. In short, you likely need to restart the container.
what you need to do is clean your docker system by using:
docker system prune : https://docs.docker.com/config/pruning/
This will remove unused containers, clean images and more.
Related
What is the best way/tool to monitor an EBS volume available space when mounted inside a Docker container?
I really need to monitor the available disk space in order to prevent crash because of no space left on device.
Do you know of any tool that can monitor that, like datadog, newrelic grafana, prometheus or something opensource?
The telegraf/influxdb/grafana stack can monitor space left on disc. Kapacitor can also be added if you want alerts. If you want to specify a limit, you have to use a dedicated partition / mount point or a btrfs subvolume with quotas.
Another option is to make cron job to clean up unused docker images, unused docker volume, and exited docker container. I use this method myself.
I had problem with storage so I deleted many files and somes files in docker folders (/var/lib/docker).
Is there a way to check the installation of Docker, the link containers and volumes, ... .
This is not the way to deal with storage. I would start from scratch to be sure everything is ok.
To properly deal with storage problems:
How to clean docker devicemapper folder properly ?
If you want to increase capacity:
https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#increase-capacity-on-a-running-device
For earlier versions with devicemapper driver, I used to do following -
Remove untagged & dangling images using simple shell awk command.
Always use docker run --rm parameter if you don't wish to review the stopped container. This will prevent from using additional storage and caches. This is more towards efficient use of Container Life Cycle, which is true for all versions.
Make sure to use -v parameter to remove the Volume associated with stopped container.
Make sure to have a separate mount for /var/lib on Docker Host for more Enterprise Robustness.
I remember one incident, where even after deleting all volumes, I had to run xfs_fsr to reclaim all storage space on the /var/lib mount.
Redhat/Fedora family was affected the most, the only solid solution was to remove Docker, remove /var/lib/docker & reinstall.
Newer version of Docker -
All operations are combined in a wrapper docker system prune -a, which cleans Volume, Image & Container. (Caution - Will remove everything which is not associated with anything).
I have a docker container which does alot of read/write to disk. I would like to test out what happens when my entire docker filesystem is in memory. I have seen some answers here that say it will not be a real performance improvement, but this is for testing.
The ideal solution I would like to test is sharing the common parts of each image and copy to your memory space when needed.
Each container files which are created during runtime should be in memory as well and separated. it shouldn't be more than 5GB fs in idle time and up to 7GB in processing time.
Simple solutions would duplicate all shared files (even those part of the OS you never use) for each container.
There's no difference between the storage of the image and the base filesystem of the container, the layered FS accesses the images layers directly as a RO layer, with the container using a RW layer above to catch any changes. Therefore your goal of having the container running in memory while the Docker installation remains on disk doesn't have an easy implementation.
If you know where your RW activity is occurring (it's fairly easy to check the docker diff of a running container), the best option to me would be a tmpfs mounted at that location in your container, which is natively supported by docker (from the docker run reference):
$ docker run -d --tmpfs /run:rw,noexec,nosuid,size=65536k my_image
Docker stores image, container, and volume data in its directory by default. Container HDs are made of the original image and the 'container layer'.
You might be able set this up using a RAM disk. You would hard allocate some RAM, mount it, and format it with your file system of choice. Then move your docker installation to the mounted RAM disk and symlink it back to the original location.
Setting up a Ram Disk
Best way to move the Docker directory
Obviously this is only useful for testing as Docker and it's images, volumes, containers, etc would be lost on reboot.
Docker makes it easy to stop & restart containers. It also has the ability to pause and then unpause containers. The Docker docs state
When the container is exited, the state of the file system and its exit value is preserved. You can start, stop, and restart a container. The processes restart from scratch (their memory state is not preserved in a container), but the file system is just as it was when the container was stopped.
I tested this out by settting up a container with memcached running, wrote a value to memcache and then
Stopped & then restarted the container - the memcached value was gone
Paused & then unpaused the container - the memcached value was still intact
Somewhere in the docs - I can no longer find the precise document - I read that stopped containers do not consume CPU or memory. However:
I suppose the fact that the file system state is preserved means that the container still does consume some space on the host's file system?
Is there a performance hit (other than host disk space consumption) associated with having 10s, or even 100s, of stopped containers in the system? For instance, does it make it any harder for Docker to startup and manage new containers?
And finally, if Paused containers retain their memory state when Unpaused - as demonstrated by their ability to remember memcached keys - do they have a different impact on CPU and memory?
I'd be most obliged to anyone who might be able to clarify these issues.
I am not an expert about docker core but I will try to answer some of these questions.
I suppose the fact that the file system state is preserved means that the container still does consume some space on the host's file
system?
Yes. Docker save all the container and image data in /var/lib/docker. The default way to save the container and image data is using aufs. The data of each layer is saved under /var/lib/docker/aufs/diff. When a new container is created, a new layer is also created with is folder, and there the changes from the layers of the source image are stored.
Is there a performance hit (other than host disk space consumption) associated with having 10s, or even 100s, of stopped
containers in the system? For instance, does it make it any harder for
Docker to startup and manage new containers?
As far as I know, it should not be any performace hit. When you stop a container, docker daemon sends SIGTERM and SIGKILL to all the process of that container, as described in docker CLI documentation:
Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...]
Stop a running container by sending SIGTERM and then SIGKILL after a
grace period
-t, --time=10 Number of seconds to wait for the container to
stop before killing it. Default is 10 seconds.
3.And finally, if Paused containers retain their memory state when
Unpaused - as demonstrated by their ability to remember memcached
keys - do they have a different impact on CPU and memory?
As #Usman said, docker implements pause/unpause using the cgroup freezer. If I'm not wrong, when you put a process in the freezer (or its cgroup), you block the execution of new task of that process from the kernel task scheduler (i.e.: it stops the process), but you don't kill them and they keep consuming the memory they are using (although the Kernel may move that memory to swap or to solid disk). And the CPU resources used by a paused container I would consider insignificant. For more information about this I would check the pull request of this feature, Docker issue #5948
Assume I am starting a big number of docker containers which are based on the same docker image. It means that each docker container is running the same application. It could be the case that the application is big enough and requires a lot of hard drive memory.
How is docker dealing with it?
Does all docker containers sharing the static part defined in the docker image?
If not does it make sense to copy the application into some directory on the machine which is used to run docker containers and to mount this app directory for each docker container?
Docker shares resources at kernel level. This means application logic is in never replicated when it is ran. If you start notepad 1000 times it is still stored only once on your hard disk, the same counts for docker instances.
If you run 100 instances of the same docker image, all you really do is keep the state of the same piece of software in your RAM in 100 different separated timelines. The hosts processor(s) shift the in-memory state of each of these container instances against the software controlling it, so you DO consume 100 times the RAM memory required for running the application.
There is no point in physically storing the exact same byte-code for the software 100 times because this part of the application is always static and will never change. (Unless you write some crazy self-altering piece of software, or you choose to rebuild and redeploy your container's image)
This is why containers don't allow persistence out of the box, and how docker differs from regular VM's that use virtual hard disks. However, this is only true for the persistence inside the container. The files that are being changed by docker software on the hard disk are "mounted" into containers using the docker volumes and thus arent really part of the docker environments, but just mounted into them. (Read more about this at: https://docs.docker.com/userguide/dockervolumes/)
Another question that you might want to ask when you think about this, is how does docker store changes that it makes to its disk on runtime. What is really sweet to check out, is how docker actually manages to get this working. The original state of the container's hard disk is what is given to it from the image. It can NOT write to this image. Instead of writing to the image, a diff is made of what is changed in the containers internal state in comparison to what is in the docker image.
Docker uses a technology called "Union Filesystem", which creates a diff layer on top of the initial state of the docker image.
This "diff" (referenced as the writable container in the image below) is stored in memory and disappears when you delete your container. (Unless you use the command "docker commit", however: I don't recommend this. The state of your new docker image is not represented in a dockerfile and can not easily be regenerated from a rebuild)