in my bash script I got the following command
ulimit -s unlimited
However, when I launch my job by sbatch job.sh and then ssh to one of the computer nodes to check the stack size ulimit -a I clearly see the stack size is:
stack size (kbytes, -s) 8192
This is my full script
#!/bin/bash -l
#SBATCH --job-name=test
#SBATCH --nodes=13
#SBATCH --ntasks-per-node=32
#SBATCH --mem=120GB
#SBATCH --time=999:99:00
#SBATCH --propagate=STACK
ulimit -s unlimited
mpirun ./pres.a
ulimit is a shell built-in command. Resource limits set with it are not system-wide and only apply to processes started in the same shell session and their descendants. When you SSH into a node and execute ulimit, it shows you the limits in that particular shell session, not the limits applied to the processes in the job, even if some of them are running on the same node.
Also, --propagate=STACK propagates the resource limits of the shell session where you execute the sbatch command and not the limits set in the job script:
PropagateResourceLimits
A list of comma separated resource limit names. The slurmd daemon uses these names to obtain the associated (soft) limit values from the users process environment on the submit node. These limits are then propagated and applied to the jobs that will run on the compute nodes.
Thus, the ulimit -s unlimited inside the job script only applies to the shell process started by SLURM when the job is executed and unless mpirun propagates the limits further onto the processes it spawns, they will inherit the system default stack size limit instead. But if you do:
$ ulimit -s unlimited
$ sbatch --propagate=STACK foo.sh
(or have #SBATCH --propagate=STACK inside foo.sh as you do), then all processes spawned by SLURM for that job will already have their stack size limit set to unlimited.
Related
We are running a docker swarm and using Monit to see resources utilisation. The
Process memory for dockerd keeps on growing over time. This happens on all nodes that at least perform a docker action e.g docker inspect or docker exec. I'm suspecting it might be something related to this these actions but I'm not sure how to replicate it. I have a script like
#!/bin/sh
set -eu
containers=$(docker container ls | awk '{if(NR>1) print $NF}')
# Loop forever
while true;
do
for container in $containers; do
echo "Running Inspect on $container"
CONTAINER_STATUS="$(docker inspect $container -f "{{.State}}")"
done
done
but I'm open to other suggestions
Assuming you can run ansible to run a command via ssh on all servers:
ansible swarm -a "docker stats --no-stream"
A more SRE solution is containerd + Prometheus + AlerManager / Grafana to gather metrics from the swarm nodes and then implement alerting when container thresholds are exceeded.
Don't forget you can simply set a resource constraint on Swarm services to limit the amount of memory and cpu service tasks can consume or be restarted. Then just look for services that keep getting OOM killed.
I've got a Jenkins setup using Docker containers as build slaves.
The containers are provisioned as agents via the Docker plugin - rather than the Jenkinsfiles instantiating the container.
Machine is an 8 thread i7 with 16GB ram and 16GB swapfile. Currently I'm running just a single container.
If the build in the container uses all 8 threads, it seems to cause OS OOM kills of some of the GCC processes when it is building. There is a mix of build tasks, some I can explicitly control the number of make threads, others it will query the system for the number of cores and go wide. When there are the maximum number of threads active it appears to run out of RAM without throttling or stalling the work - it also doesn't seem to make full use of the swap.
I want to limit the number of CPU cores the Docker slave is allowed to use but I can't find a way to pass the --cpus=2 argument to the Docker run command. the shares argument doesn't appear to have the effect I want.
I'm happy to divvy up the resources explicitly from the container configuration in order to make the server more reliable, but I don't want it running into hard limits and getting OOM killed.
I am not quite sure how you start the Docker container but if you do it with help of the plug-in, I am guessing that you make a call in your Jenkinsfile like myImage.inside("..."). In that case you should be able to add "--cpus=2" as part of the string argument. This works at least for us.
The drawback with this solution is however that all Jenkins jobs have to be updated and you can never be sure that all creators of Jenkins jobs remember this. Hence we implemented a different solution which should work for you if the above solution does not. We basically created a script to work as a wrapper when anyone called docker. It injects "--cpus=2" if the "docker run" is called. Otherwise it just forwards the call to Docker. We accomplished this by:
Put the following script in /home/jenkins/bin/docker . ("docker" being the file, not a folder)
#!/bin/bash
echo COMMAND: $0 with arguments: $# >&2
if [ "$1" = 'run' ] ; then
echo 'INFO: I found a "run" command, limiting to 2 cpus' >&2
echo RUNNING: /usr/bin/docker "$1" --cpus=2 "${#:2}" >&2
/usr/bin/docker "$1" --cpus=2 "${#:2}"
else
echo RUNNING: /usr/bin/docker "$#" >&2
/usr/bin/docker "$#"
fi
Make it executable
chmod a+x /home/jenkins/bin/docker
Modify the path by updating ~/.bashrc . (~/.bash_profile only takes affect when you log in, meaning that Jenkins will not pick it up.) Put "/home/jenkins/bin" FIRST in the path, e. g.
PATH=/home/jenkins/bin:$PATH
I have a Docker container running with this command in my Jenkins job:
docker run --name="mydoc" reportgeneration:1.0 start=$START end=$END config=$myfile
This works very well. The image is created from a DockerFile which is executing a shell script with ENTRYPOINT.
Now I want to know how much CPU and memory has been utilized by this container. I am using a Jenkins job, where in the "execute shell command", I am running the above Docker run command.
I saw about 'docker stats' command. It works very well in my Ubuntu machine. But I want it to run via Jenkins as my container is running via Jenkins console. So here follows the limitations I have.
I don't know if there is any way to stop docker stats command. In Ubuntu command line, we hit 'ctrl+c' to stop it. How will I do it in Jenkins?
Even if I figure out a way to stop docker stats, once the 'docker run' command gets executed, the container will not be active and will be exited. For exited container, CPU and memory utilisation will be zero.
docker run 'image'
docker stats container id/name
With the above two lines, docker stats command will only get an exited container and I don't think docker stats will even work with Jenkins console as it cannot be stopped.
Is there any way that I can get container's resource utilization (CPU, memory) in a better way via Jenkins console?
Suggestion is to not run docker stats interactively, but have a piece of a shell script with a loop like this:
#!/bin/sh
# First, start the container
CONTAINER_ID=$(docker run -d ...)
# Then start watching that it's running (with inspect)
while [ "$(docker inspect -f {{.State.Running}} $CONTAINER_ID 2>/dev/null)" = "true" ]; do
# And while it's running, check stats
docker stats --no-stream $CONTAINER_ID
sleep 1
done
# When the script reaches this point, the container had stopped.
# For example, let's clean it up (assuming you haven't used --rm in run).
docker rm $CONTAINER_ID
The condition checks whenever the container is running or not, and docker stats --no-stream prints stats once then exits, making it suitable for non-interactive use.
I believe you can use a variant of such shell script file (obviously, updated to do something useful, rather than just starting the container and watching its stats) as a build step.
But if you need/want/have an interactive process that you want to stop, kill is the command you're looking for. Ctrl-C in a terminal just sends a SIGINT to the process.
You need to know an PID, of course. I'm not sure about Jenkins, but if you've just started a child process from a shell script with child-process & (e.g. docker stats &), then its PID would be in the $! variable. Or you can try to figure it using pidof or ps commands, but that may be error-prone in case of concurrent jobs (unless they're all isolated).
Here I've assumed that your Jenkins jobs are shell scripts that do the actual work. If your setup is different (e.g. if you use some plugins so Jenkins talk to Docker directly), things may be different and more complicated.
I need to set the file descriptor limit correctly on the docker container
I connect to container with ssh (https://github.com/phusion/baseimage-docker)
Already tried:
edit limits.conf the container ignore this file
upstart procedure found at https://coderwall.com/p/myodcq but this docker image has different kind of init process. (runit)
I tried to modify configuration of pam library in /etc/pam.d
try to enabled pam for ssh in sshd_config
The output it always the same.
bash: ulimit: open files: cannot modify limit: Operation not permitted
The latest docker supports setting ulimits through the command line and the API. For instance, docker run takes --ulimit <type>=<soft>:<hard> and there can be as many of these as you like. So, for your nofile, an example would be --ulimit nofile=262144:262144
After some searching I found this on a Google groups discussion:
docker currently inhibits this capability for enhanced safety.
That is because the ulimit settings of the host system apply to the docker container. It is regarded as a security risk that programs running in a container can change the ulimit settings for the host.
The good news is that you have two different solutions to choose from.
Remove sys_resource from lxc_template.go and recompile docker. Then
you'll be able to set the ulimit as high as you like.
or
Stop the docker demon. Change the ulimit settings on the host. Start
the docker demon. It now has your revised limits, and its child
processes as well.
I applied the second method:
sudo service docker stop;
changed the limits in /etc/security/limits.conf
reboot the machine
run my container
run ulimit -a in the container to confirm the open files limit has been inherited.
See: https://groups.google.com/forum/#!searchin/docker-user/limits/docker-user/T45Kc9vD804/v8J_N4gLbacJ
If using the docker-compose file,
Based on docker compose version 2.x
We can set like as below, by overriding the default config.
ulimits:
nproc: 65535
nofile:
soft: 26677
hard: 46677
https://docs.docker.com/compose/compose-file/compose-file-v3/
I have tried many options and unsure as to why a few solutions suggested above work on one machine and not on others.
A solution that works and that is simple and can work per container is:
docker run --ulimit memlock=819200000:819200000 -h <docker_host_name> --name=current -v /home/user_home:/user_home -i -d -t docker_user_name/image_name
Actually, I have tried the above answer, but it did not seem to work.
To get my containers to acknowledge the ulimit change, I had to update the docker.conf file before starting them:
$ sudo service docker stop
$ sudo bash -c "echo \"limit nofile 262144 262144\" >> /etc/init/docker.conf"
$ sudo service docker start
Here is what I did.
set
ulimit -n 32000
in the file /etc/init.d/docker
and restart the docker service
docker run -ti node:latest /bin/bash
run this command to verify
user#4d04d06d5022:/# ulimit -a
should see this in the result
open files (-n) 32000
[user#ip ec2-user]# docker run -ti node /bin/bash
user#4d04d06d5022:/# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 58729
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 58729
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
The docker run command has a --ulimit flag you can use this flag to set the open file limit in your docker container.
Run the following command when spinning up your container to set the open file limit.
docker run --ulimit nofile=<softlimit>:<hardlimit> the first value before the colon indicates the soft file limit and the value after the colon indicates the hard file limit. you can verify this by running your container in interactive mode and executing the following command in your containers shell ulimit -n
PS: check out this blog post for more clarity
For boot2docker, we can set it on /var/lib/boot2docker/profile, for instance:
ulimit -n 2018
Be warned not to set this limit too high as it will slow down apt-get! See bug #1332440. I had it with debian jessie.
I added something in /etc/security/limits.conf in a docker container to limit the max number of user processes for user1, but when I run bash in the container under the user user1, ulimit -a doesn't reflect that limits defined in the pam limits file (/etc/security/limits.conf).
How can I get this to work?
I've also added the line session required pam_limits.so to /etc/pam.d/common-session, so that's not the problem.
I start the docker container with something like sudo docker run --user=user1 --rm=true <container-name> bash
Also, sudo docker run ... --user=user1 ... cmd doesn't apply the pam limits, but sudo docker run ... --user=root ... su user1 -c 'cmd' does
The /etc/security/limits.conf is just a file that is read by PAM infrastructure on boot. Docker containers are clones of the kernel in pristine state after it just started. This means that none of the inherited initializations of the environment will apply to container. You have to use the 'limit' command directly to set the process limits.
Better way to do that would be to use container limits, unfortunately current version of docker doesn't support limits on number of processes. Looks like the support will be coming in version 1.6 when it comes out, as #thaJeztah has mentioned.