Before we used fstab:
/dev/disk /var/lib/lxc/<container>/rootfs/dev/disk none bind,create=dir 0 0
But after upgrade to lxc 1.1.5 this solution doesn't work.
Also if we make parted to create some partitions in host system, forlders "disk" and "block" magicaly appear in lxc container /dev
Related
Before Linux Kernel v4, I was able to obtain the host PID from inside the docker container from the process scheduling information.
For instance, if I run sleep command inside the container and my local PID is 37, then I can check the actual PID on the host via:
root#helloworld-595777cb8b-gjg4j:/# head /proc/37/sched
sleep (27062, #threads: 1)
I can verify on the host that the PID 27062 corresponds to the process within the container.
root 27062 0.0 0.0 4312 352 pts/0 S 16:29 0:00 sleep 3000
I have tried this with on RHEL7 (Kernel: Linux 3.10) with Docker version: 17.09.0-ce.
I am not able to reproduce the same result on RHEL8 (Kernel: Linux 4.18) with Docker version: 20.10. In fact, I always get the local PID from the scheduling information.
/ # head /proc/8/sched
sleep (8, #threads: 1)
I might be wrong but my assumption is that something is changed within the Kernel which forbids to obtain the host PID?
So the question is how to obtain the host PID from within the container?
The bug (or "feature" if you prefer) that allowed the host PID to be discovered from /proc/PID/sched in the container was fixed (or "broken" if you prefer) in Linux kernel 4.14 by commit 74dc3384fc79 ("sched/debug: Use task_pid_nr_ns in /proc/$pid/sched").
As a result of the change, the container cannot get the host PID of a process (at least via /proc/PID/sched) if PID namespaces are in use.
I was wondering if Docker Swarm was possible to be a load balancer with GlusterFS as the local filesystem? And use Pacemaker to hold the VIP (because I understand Docker cannot create a VIP).
My idea - which I'm hoping can be verified or suggested better :)
System:
2x CentOS 8 servers
- 192.168.0.1
---- /dev/sda (OS)
---- /dev/sdb (data)
- 192.168.0.2
---- /dev/sda (OS)
---- /dev/sdb (data)
Install Pacemaker, Corosync
dnf --enablerepo=HighAvailability -y install pacemaker pcs psmisc policycoreutils-python-utils
systemctl start pcsd
Add a VIP to both servers
pcs resource create vip IPaddr2 ip=192.168.0.100 cidr_netmask=24 op monitor interval=30s
Set up both storage
mkfs.xfs /dev/sdb
Make the directory and add to startup
mkdir -p /my-data/
echo "/dev/sdb /my-data xfs defaults 0 0" >> /etc/fstab
Install GlusterFS on both nodes
dnf install -y glusterfs-server
Setup Gluster for the volume
gluster volume create gfs replica 2 transport tcp node01:/my-data node02:/my-data force
gluster volume start gfs
Make it accessible for the replication
echo 'node01:/my-data /mnt glusterfs defaults,_netdev 0 0' >> /etc/fstab
echo 'node02:/my-data /mnt glusterfs defaults,_netdev 0 0' >> /etc/fstab
Install Docker and Docker-Compose
Initialise Swarm
- on node01 use IP 192.168.0.1 -> manager
- on node02 use IP 192.168.0.2 -> manager
Create the directories
mkdir /mnt/html
mkdir /mnt/mysql
In the docker-compose.yml file:
volumes:
- "/mnt/html:/var/www/html/wp-content"
volumes:
- "/mnt/mysql:/var/lib/mysql"
Apart of the docker-compose.yml - apache:
Use IP 192.168.0.100 as the access on 80
My thoughts are that as 192.168.0.100 is only accessible on one of the Pacemaker resources, that the secondary Manager wouldn't be hit on the front end. If that node went down on the IP .100 then the other node02 would take that IP and the Swarm would still be active.
Is this something that would work? I cant find anything about having a VIP on the Swarm - at least working solutions.
I have them both as Managers because I assume if the manager goes off then its not going to work? Then if I had a 3rd, 4th, etc. I'd add them as Workers.
Below is the file system in overlay2 eating disk space, on Ubuntu Linux 18.04 LTS
Disk space of server 125GB
overlay 124G 6.0G 113G 6% /var/lib/docker/overlay2/9ac0eb938cd2a50bb87e8ed13605d3f09214fdd9c8967f18dfc3f9432701fea7/merged
overlay 124G 6.0G 113G 6% /var/lib/docker/overlay2/397b099799212060ee7a4718660aa13aba8aa1fbb92f4d88d86fbad94e572847/merged
shm 64M 0 64M 0% /var/lib/docker/containers/7ffb129016d187a61a31c33f9e468b98d0ac7ab1771b87631f6caade5b84adc6/mounts/shm
overlay 124G 6.0G 113G 6% /var/lib/docker/overlay2/df7c4acee73f7aa2536d2a8929a48241bc8e92a5f7b9cb63ab70cea731b52cec/merged
Another solution if the above doesn't work is setup a log rotation.
nano /etc/docker/daemon.json
if not found
cat > daemon.json
Add the following lines to file:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
}
}
Restart the docker daemon: systemctl restart docker
Please refer: How to setup log rotation post installation
In case someone else runs into this, here's what's happening:
Your container may be writing data (logs, deployables, downloads...) to its local filesystem, and overlay2 will create a diff on each append/create/delete, so the container's filesystem will keep growing until it fills all available space on the host.
There are a few workarounds that won't require changing the storage driver:
first of all, make sure the data saved by the container may be discarded (you probably don't want to delete your database or anything similar)
periodically stop the container, prune the system docker system prune and restart the container
make sure the container doesn't write to its local filesystem, but if you can't:
replace any directories the container writes to with volumes or mounts.
Follow the Steps if your Server is Linux Ubuntu 18.04 LTS (should work for others too)
Docker info for Overlay2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
if you got the following lines when you enter df -h --total
19M /var/lib/docker/overlay2/00d82017328c49c661c78ce14550c4073c50a550fe5004911bd3488b085aea76/diff
5.9M /var/lib/docker/overlay2/00e3e4fa0cbff7c242c38cfc9501ef1a523158d69b50779e08a773e7e22a01f1/diff
44M /var/lib/docker/overlay2/0e8e7e893b2c8aa17b4875d421670e058e4d97de066c970bbeab6cba566a44ba/diff
28K /var/lib/docker/overlay2/12a4c4e4877d35e9db657e4acff32e513042cb44119cca5c43fc19ad81c3915f/diff
............
............
then do the changes as follows:
First stop docker : sudo systemctl stop docker
Next: got to path /etc/docker
Check file daemon.json if not found
cat > daemon.json
and enter the following inside:
{
"storage-driver": "aufs"
}
and close
Finally restart docker : sudo systemctl start docker
Check if the changes have been made:
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 0
Dirperm1 Supported: true
Changing the file system can help you to resolve this issue.
Please if check your docker version supports aufs here:
Please do check the Linux distribution and what storage drivers supported here :
I had a similar issue with the docker swarm.
The docker system prune --volume, restarting the server, removing the swarm stack and recreation was not helping.
In my case, I was hosting RabbitMQ where docker-compose config was:
services:
rabbitmq:
image: rabbitmq:.....
....
volumes:
- "${PWD}/queues/data:/var/lib/rabbitmq"
In such a case each container restart, each server reboot, just all that leads to restarting the rabbitmq container takes more and more hard drive space.
Initial value:
ls -ltrh queues/data/mnesia/ | wc -l
61
du -sch queues/data/mnesia/
7.8G queues/data/mnesia/
7.8G total
After restart:
ls -ltrh queues/data/mnesia/ | wc -l
62
du -sch queues/data/mnesia/
8.3G queues/data/mnesia/
8.3G total
My solution was to stop the rabbitmq and remove directories in queues/data/mnesia/. Then restart the rabbitmq.
Maybe sth is wrong with my config... But if you have such an issue then worth checking your volumes of containers whether do not leave some trash there.
If you are troubled by that /var/lib/docker/overlay2 directory is taking too much space(use du command to check space usage), then the answer below may be suitable for you.
docker xxx prune commands will clean up something unused, such as all stopped containers(in /var/lib/docker/containers), files in the virtual filesystems of stopped containers(in /var/lib/docker/overlay2), unmounted volumes(in /var/lib/docker/volumes) and images that don't have related containers(in /var/lib/docker/images). But all of this will not touch the containers which are in running.
limiting the size of logs in configurations will limit the size of /var/lib/docker/containers/*/*-json.log, but it doesn't involve the overlay2 directory.
you can find two folders called merged and diff in /var/lib/docker/overlay2/<hash>/. If these folders are big. That means there are high disk usage in your containers SELVES but not the docker host. In this case, you have to attach a terminal into relevant containers, find the high usage locations in the containers, and take your own solutions.
Just like Nick M said.
I have a server where I run some containers with volumes. All my volumes are in /var/lib/docker/volumes/ because docker is managing it. I use docker-compose to start my containers.
Recently, I tried to stop one of my container but it was impossible :
$ docker-compose down
[17849] INTERNAL ERROR: cannot create temporary directory!
So, I checked how the data is mounted on the server :
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7,8G 0 7,8G 0% /dev
tmpfs 1,6G 1,9M 1,6G 1% /run
/dev/md3 20G 19G 0 100% /
tmpfs 7,9G 0 7,9G 0% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 7,9G 0 7,9G 0% /sys/fs/cgroup
/dev/md2 487M 147M 311M 33% /boot
/dev/md4 1,8T 1,7G 1,7T 1% /home
tmpfs 1,6G 0 1,6G 0% /run/user/1000
As you can see, the / is only 20Go, so it is full and I can't stop my containers using docker-compose.
My questions are :
There is a simple solution to increase the available space in the
/, using /dev/md4 ?
Or can I move volumes to another place without losing data ?
This part of the Docker Daemon is confirgurable. Best practices would have you change the data folder; this can be done with OS-level Linux commands like a symlink... I would say it's better to actually configure the Docker Daemon to store the data elsewhere!
You can do that by editing the Docker command line (e.g. the systemd script that starts the Docker daemon), or change /etc/docker/daemon.json.
The file should have this content:
{
"data-root": "/path/to/your/docker"
}
If you add a new hard drive, partition, or mount point you can add it here and docker will store its data there.
I landed here as I had the very same issue. Even though some sources suggest you could do it with a symbolic link this will cause all kinds of issues.
Depending on the OS and Docker version I had malformed images, weird errors or the docker-daemon refused to start.
Here is a solution, but it seems it varies a little from version to version. For me the solution was:
Open
/lib/systemd/system/docker.service
And change this line
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
to:
ExecStart=/usr/bin/dockerd -g /mnt/WHATEVERYOUR/PARTITIONIS/docker --containerd=/run/containerd/containerd.sock
I solved it creating a symbolic link to a partition with bigger size:
ln -s /scratch/docker_meta /var/lib/docker
/scratch/docker_meta is the folder that I have in a bigger partition.
Do a bind mount.
For example, moving /docker/volumes to /mnt/large.
Append line into /etc/fstab.
/mnt/large /docker/volumes none bind 0 0
And then.
mv /docker/volumes/* /mnt/large/
mount /docker/volumes
Do not forget chown and chmod of /mnt/large first, if you are using non-root docker.
I set up swarm cluster with two machine. It work as desired. I'm able to launch container on desired node based on constraint filter. However when I'm trying to ping one container in one node from container in another node, it fail. It does not recognize it. Is it as expected or I did something wrong in setting swarm cluster?
Other Details:
machine 1(10.0.0.4) as both host/node
machine 2(10.0.0.21) as node
Swarm agent is 10.0.0.4:2374(ip :port)
The output of info command it's:
docker -H tcp://10.0.0.4:2374 info
Containers: 11
strategy: spread
Filters: affinity, health, constraint, port, dependency
Nodes: 2
machine1: 10.0.0.4:2375
└ Containers: 6
└ Reserved CPUs: 0 / 25
└ Reserved Memory: 0 B / 24.76 GiB
machine2: 10.0.0.21:2375
└ Containers: 5
└ Reserved CPUs: 0 / 25
└ Reserved Memory: 0 B / 24.76 GiB
Overlay networks was introduced in Docker 1.9 (Nov 2015). It allows containers in different nodes (hosts) to be part of the same network and communicate.
Yes, from the docs "Docker Swarm is native clustering for Docker. It turns a pool of Docker hosts into a single, virtual host." https://docs.docker.com/swarm/
It looks docker swarm is more like managing tool and scheduler. I have to use some other tool like weave or ambassador to connect two container in different host . Anyhow, docker swarm is good clustering tool and help me setup thing as i desire.