I have four separate pipelines that all run on the same node. Recently, I've been getting errors that look like this:
Disk space is too low. Only 0.315GB left on /var/jenkins.
I've already reconfigured the pipelines to get rid of old logs and builds after 7 days. Aside from this, are there any plugins or shell commands I can run post-build to keep my disk space free?
This is one of those problems that can be fixed/monitoried in multiple ways.
If you're willing to you can set up something like datadog or nagios to monitor your system and alert you when something is starting to fill up your /var/jenkins.
You can also set up a cron that checks and emails you when something is starting to fill up.
But if you'd like to figure why it's filling up, it's possible that your /var partition is too small, but without seeing your disk partition layout it's hard to give a better answer.
I Have faced same issue with the one of the jenkins node
solution: Ssh to your slave and do this df -h, it will show disk info, and available space in /tmp and increase tmp size by
sudo mount -o remount /tmp
Related
My raspberrypi suddenly had no more free space.
By looking at the folder sizes with the following command:
sudo du -h --max-depth=3
I noticed that a docker folder eats an incredible amount of hard disk space. It's the folder
var/lib/docker/containers/*
The folder seems to contain some data for the current running docker containers. The first letters of the filename correspond to the docker container-ID. One folder seems to grow dramatically fast. After stopping the affected container and removed him, the related folder disappeared. So the folder seems to have belonged to it.
Problem solved.
I wonder now what the reason could be that this folder size increases so much. Further, I wonder what is the best way to not run into the same problem again later.
I could write a bash script which removes the related container at boot and run it again. Better ideas are very welcome.
The container ids are directories, so you can look inside to see what is using space in there. The two main reasons are:
Logs from stdout/stdere. These can be limited with added options. You can view these with docker logs.
Filesystem changes. The underlying image filesystem is not changed, so any writes trigger a copy-on-write to a directory within each container id. You can view these with docker diff.
I have a Kubernetes-cluster with 1 master-node and 3 worker-nodes. All Nodes are running on CentOS 7 with Docker 19.06. I'm also running Longhorn for dynamic provisioning of volumes (if that's important).
My problem is that every few days one of the worker nodes grows the HDD-usage to 85% (43GB). This is not a linear increase, but happens over a few hours, sometimes quite rapidly. I can "solve" this problem for a few days by first restarting the docker service and then doing a docker system prune -a. If I don't restart the service first, the prune removes next to nothing (only a few MB).
I also tried to find out which container is taking up all that space, but docker system df says it doesn't use the space. I used df and du to crawl along the /var/lib/docker subdirectories too, and it seems none of the folders (alone or all together) takes up much space either. Continuing this all over the system, I can't find any other big directories either. There are 24GB that I just can't account for. What makes me think this is a docker problem nonetheless is that a restart and prune just solves it every time.
Googling around I found a lot of similar issues where most people just decided to increase disk space. I'm not keen on accepting this as the preferred solution, as it feels like kicking the can down the road.
Would you have any smart ideas on what to do instead of increasing disk space?
It seems like it is expected behavior, from Docker documentation you can read:
Docker takes a conservative approach to cleaning up unused objects
(often referred to as “garbage collection”), such as images,
containers, volumes, and networks: these objects are generally not
removed unless you explicitly ask Docker to do so. This can cause
Docker to use extra disk space. For each type of object, Docker
provides a prune command. In addition, you can use docker system prune to clean up multiple types of objects at once. This topic shows
how to use these prune commands.
So it seems like you have to clean it up manually using docker system/image/container prune. Other issue might be that those containers create too much logs and you might need to clean it up.
I had started Hyperledger-composer from fabric-dev-server, So all images running as regular.
Now after two weeks I had seen that my HDD space is occupied by docker container.
So, Here are some screenshots of my hdd space:
Day-1
Day-2
In 2 days, the hdd available size become 9.8G to 9.3G.
So, How can I resolve this issue?
I think the problem is that the docker container of peer0 is generating too many logs, so if you run that container continuously, it will generate more logs when you access the fabric network.
you can check the file size of the log for particular docker container:
Find container id of peer0.
Goto directory /var/lib/docker/containers/container_id/.
There should be a file named as container_id-json.log.
So in my case:
My fabric was running from 2 weeks, and the logs file is at (example):
/var/lib/docker/containers/a50ea6b441ee327587a73e2a0efc766ff897bed2e187575fd69ff902b56a5830/a50ea6b441ee327587a73e2a0efc766ff897bed2e187575fd69ff902b56a5830-json.log
I had check the size of that file, it was near 6.5GB.
Solution (Temporary):
Run below command, which will delete data of that file (example):
> var/lib/docker/containers/a50ea6b441ee327587a73e2a0efc766ff897bed2e187575fd69ff902b56a5830/a50ea6b441ee327587a73e2a0efc766ff897bed2e187575fd69ff902b56a5830-json.log
Solution (Permanent):
What you can do this just make a script that run everyday and remove data from that log file.
You can use crontab, which give you ability to run script on specific time,day etc.
I'm having problem with space usage of docker. I have
/var/lib/docker/aufs/diff/e20ed0ec78d30267e8cf855c6311b02089b6086ea149c21997a3e6cb9757ecd4/tmp/registry-dev/docker/registry/v2/blobs# du -sh
4.6G .
can I find which container does this folder belong to? I have docker registry running but inside there I have
/var/lib/registry/docker/registry/v2/blobs# du -sh
465M .
I'm suspecting docker upgrade (I used migrate tool https://docs.docker.com/engine/migration/ here) could have left it, or, I was building docker registry myself before, and moving to pre-compiled registry left this.
can I somehow check which container it belogs to? or maybe, does it belong to any?
I had same issue and the spotify/docker-gc fixed it. Clone it then follow "Running as a Docker Image"
The spotify/docker-gc is not going to fix it, but it make things to turn for the better much easier. The first thing you need to do is to stop doing commits on the same image. As I've realized, this is going to build up a huge diff-dependency. What I did is that I've committed all my running containers into different image names and tags, stopped and restarted the containers. After that, I've deleted the old images manually, then ran spotify/docker-gc. I've saved about 20% of disk space. If I ran spotify/docker-gc before the commits into new images, nothing happened.
If you use spotify/docker-gc, please do DRY_RUN.
I am using jenkins 1.520, i have 500mb in /tmp on linux (RHEL6 host), 500mb is plenty of space for our current build size and number of projects, how can i make this message go away and have my master node be online?
I looked around for bugs that apply to this version, did not find any. I tried previous suggestions found in stackoverflow, do not seem to work or simply do not apply.
df -h
Filesystem Size Used Avail Use% Mounted on
/
/dev/mapper/vg00-tmp 504M 23M 456M 5% /tmp
..thanks...
You can configure the Free Disk Space and Free Temp Space thresholds in the Node configuration, http://jenkins:8080/computer/configure (or from the Jenkins main page -> Build Executor Status -> Configure).