I have couple of python docker containers, processing data from kafka topics every X seconds.
If the data isn't appearing for i.e. more than 2X time, I should log it somewhere (so I can check which kafka producers to restart - doesn't matter here).
I was thinking about creating some file, where I would keep last message timestamp, and another script/container would check it every X seconds, to find out, if the data should already have appeared. I could check docker logs aswell, if "amount" of logs is increasing (if not, then it means, that no data is appearing).
While those solutions would probably work, I'm not sure, if there isn't already another "ready-made" solution for that issue. Is there something, that I can use to check if the data is being processed?
You can use some sort of health check mechanism by putting logs inside: <filename_1>
then the healthcheck would be:
if [[ `[ -r <filename_1> ] && cat <filename_1> | wc -l` -ge 1 ]]; then > <filename_1> && echo 0; else echo 1; fi
By doing this you check if the file exist then check the number of lines. If greater/equal than 1 you empty it then return 0 .If empty or does not exist you return 1
This will mark your docker as healthy or unheathly.
If you want to restart by itself you can replace echo 1 by kill 1 kill 1 will kill process with pid 1 in container and force container exit. Usually the process started by CMD or ENTRYPOINT has pid 1.
[[ `[ -r <filename_1> ] && cat <filename_1> | wc -l` -ge 1 ]] && > <filename_1> && echo 0 || kill 1
Don't forget --restart always option.
Related
I have a Docker image which contains a file, say /usr/bin/foo. What's the easiest way to find out which step of the Dockerfile added that path? (Which I thought was equivalent to the question, of which layer of the Docker image does that path come from?)
I wrote a script which prints out all the paths in the image, prefixed by layer ID. It appears to work, but is quite slow:
#!/bin/bash
die() { echo 1>&2 "ERROR: $*"; exit 1; }
dir=$(mktemp -d)
trap "rm -rf $dir" EXIT
img="$1"
[[ -n "$img" ]] || die "wrong arguments"
docker image save "$img" | (cd $dir && tar xf -) ||
die "failed extracting docker image $img"
(cd $dir && find . -name '*.tar' | while read f; do layer=$(echo $f | cut -d/ -f2); tar tf $f | sed -e "s/^/$layer:/"; done) ||
die "failed listing layers"
(It could be made faster if it didn't write anything to disk. The problem is while tar tf - prints the paths in the TAR, it doesn't do the same for the nested layer.tar files. I am thinking I could use the Python tarfile module - but surely somebody else out there has done this already?)
However, I don't know how to translate the layer ID it gives me to a step in the Docker image. I thought I'd correlate it with the layer IDs reported by docker inspect:
docker image inspect $IMAGE | jq -r '.[].RootFS.Layers[]' | nl
But the layer ID which my script reports as containing the path, I can't find in the output of the above command. (Is that a consequence of BuildKit???)
In the end, I gave up on this whole approach. Instead I just made some educated guesses as to which Dockerfile line was probably creating that path, tested each guess by commenting it out (and all the lines after it), and soon I found the answer. Still, there must be a better way, surely? Ideally, what I'd like is something like a --contains-path= option to docker image history – which doesn't exist, but maybe there is something else which does the equivalent?
While dlayer does not have any searching function built-in, it is straight-forward to implement by combining it with a Perl one-liner:
docker image save $IMAGE |
dlayer -n 999999 |
perl -ne 'chomp;$query=quotemeta("usr/bin/foo");$cmd=$_ if $_ =~ m/ [\$] /;print "$cmd\n\t$_\n" if m/ $query/;'
This will print something like:
13 MB $ /opt/bar/install.sh # buildkit
637 B usr/bin/foo
-n 999999 is to increase limit of number of file names output from the default 100, otherwise the path will only be found if it is in the first 100 from that layer.
(I submitted a PR to add a built-in search function to dlayer, which removes the need for this one-line Perl script.)
As for linux, I can get the real-time of a process through /proc (process file system). But how can I get the data of the process on MacOS 12.4 since there is no /proc dir.
When using top command, I want to save the read-time output of a process to a text file (only the process info), but the interval is too large to get accurate result and I cannot strip the header of top via options. Is there any possible way to achieve this? On a phone, I can get the output of the top via the following script:
#!/bin/sh
export LANG="en_US.UTF-8"
if [ "$#" -ne 1 ]; then
echo "Illegal number of parameters" $#
exit 0
fi
while true; do
pid=`pgrep COMMAND`
if [ "$pid" -ne 0 ]; then
echo "${pid}\t`top -q -p $pid -n 1 -d 1`" >> $1
fi
usleep 100
done
I wonder whether it is possible to get the detailed info via /proc-like file system on MacOS?
I am running docker stats $CONTAINER_ID from a shell script to monitor my Docker container memory and CPU usage over a period of 1 hour. I have the below shell script.
#!/bin/sh
# First, start the container
CONTAINER_ID=abcadsasdasd
# Then start watching that it's running (with inspect)
while [ "$(docker inspect -f {{.State.Running}} $CONTAINER_ID 2>/dev/null)" = "true" ]; do
# And while it's running, check stats
#docker stats $CONTAINER_ID 2>&1 | tee "$1"
docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}\t{{.BlockIO}}\t{{.PIDs}}" $CONTAINER_ID 2>&1 | tee "$1"
sleep 5
done
It is running fine. But, it seems it is running at every second. I need that it outputs at every 5 seconds. I found that there is no such option like specifying timeperiod with docker stats command. Request some suggestions to achieve it. It seems sleep 5 is not having any effect.
Edit
Even with 1 second delay, I expected 60 lines in my log file, With 5 seconds, I expected 12 lines over 1 minute period. But, I am getting close to 150 lines for 1 minute.
Without sleep, just read current values from the file ($1) and average them every period you choose:
#!/bin/bash
multiLine="";
format="table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}";
docker stats --all --format "$format" --no-trunc | (while read line; do
sedLine=$(echo "$line" | sed "s/^.*name.*cpu.*mem.*$/_divider_/i")
if [ "$sedLine" != "_divider_" ];
then
multiLine="${multiLine}"'\n'"${line}";
else
echo -e $multiLine > $1;
multiLine="";
fi;
done);
docker stats $container won't exit when running, so your sleep 5 don't have chance to execute.
For you, you should use next:
--no-stream Disable streaming stats and only pull the first result
Then, you could use something like next fake code to control the rate:
while xxx; do
docker stats $container --no-stream
sleep 5
done
With docker stats you can see the memory usage of a container over time.
Is there a way to find what the highest value of memory usage was while running docker stats?
If you need to find the peak usage you are better off requesting the .MemPerc option and calculating based on the total memory (unless you restricted the memory available to the container). .MemUsage has units which change during the life of the container which mess with the result.
docker stats --format 'CPU: {{.CPUPerc}}\tMEM: {{.MemPerc}}'
You can stream an ongoing log to a file (or script).
To get just the max memory as originally requested:
(timeout 120 docker stats --format '{{.MemPerc}}' <CONTAINER_ID> \
| sed 's/\x1b\[[0-9;]*[a-zA-Z]//g' ; echo) \
| tr -d '%' | sort -k1,1n | tail -n 1
And then you can ask the system for its total RAM (again assuming you didn't limit the RAM available to docker) and calculate:
awk '/MemTotal/ {print $2}' /proc/meminfo
You would need to know how long the container is going to run when using timeout as above, but if docker stats was run without this in background submitted by a script it could kill it once the container completed.
...
This command allows you to generate a time-series of the cpu/memory load:
(timeout 20 docker stats --format \
'CPU: {{.CPUPerc}}\tMEM: {{.MemPerc}}' <CONTAINER_ID> \
| sed 's/\x1b\[[0-9;]*[a-zA-Z]//g' ; echo) \
| gzip -c > monitor.log.gz
Note that it pipes into gzip. In this form you get ~2 rows per second so the file would get large rapidly if you don't.
I'd advise this for benchmarking and trouble shooting rather than use on production containers
I took a sampling script from here and aggregated data by #pl_rock. But be careful - the sort command only compares string values - so the results are usually wrong (but ok for me).
Also mind that docker is sometimes reporting wrong numbers (ie. more allocated mem than physical RAM).
Here is the script:
#!/bin/bash
"$#" & # Run the given command line in the background.
pid=$!
echo "" > stats
while true; do
sleep 1
sample="$(ps -o rss= $pid 2> /dev/null)" || break
docker stats --no-stream --format "{{.MemUsage}} {{.Name}} {{.Container}}" | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0 }' >> stats
done
for containerid in `awk '/.+/ { print $7 }' stats | sort | uniq`
do
grep "$containerid" stats | sort -r -k3 | tail -n 1
# maybe: | sort -r -k3 -h | head -n 1
# see comment below (didnt tested)
done
In my case I wanted to monitor a docker container which runs tests for my web application. The test suite is pretty big, it includes javascript tests in a real browser and consume significant amount of both, memory and time.
Ideally, I wanted to watch the current memory usage real time, but to also keep the history for later analysis.
I ended up using a modified and simplified version of the Keiran's solution:
CONTAINER=$(docker ps -q -f name=CONTAINER_NAME)
FORMAT='{{.MemPerc}}\t{{.MemUsage}}\t{{.Name}}'
docker stats --format $FORMAT $CONTAINER | sed -u 's/\x1b\[[0-9;]*[a-zA-Z]//g' | tee stats
Notes:
CONTAINER=$(docker ps -q -f name=NAME) find container by name, but there are other options
FORMAT='{{.MemPerc}} ...}} MemPerc goes first (for sorting); otherwise you can be creative
sed -u the -u flag is important, it turns off buffering
| sed -u 's/\x1b\[[0-9;]*[a-zA-Z]//g' removes ANSI escape sequences
| tee stats not only display in real time, but also write into the stats file
I Ctrl-C manually when it's ready – not ideal, but OK for me
after that it's easy to find the max with something like sort -n stats | tail
you can use command:
docker stats --no-stream | awk '{ print $3 }' | sed '1d'|sort | tail -1
It will give highest memory by container.
Let me Explain command:
--no-stream : Disable streaming stats and only pull the first result
awk '{ print $3 }' : will print MEM USAGE
sed '1d' : will delete first entry that is %
sort : it will sort the result
tail -1 : it will give last entry that is highest.
After some crashes with a docker container with a too low mem_limit, how can i check in a container the mem_limit of this container? I want to print an error message on startup and exit if the mem_limit is set to low.
The memory limit is enforced via cgroups. Therefore you need to use cgget to find out the memory limit of the given cgroup.
To test this you can run a container with a memory limit:
docker run --memory 512m --rm -it ubuntu bash
Run this within your container:
apt-get update
apt-get install cgroup-bin
cgget -n --values-only --variable memory.limit_in_bytes /
# will report 536870912
Docker 1.13 mounts the container's cgroup to /sys/fs/cgroup (this could change in future versions). You can check the limit using:
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
On the host you can run docker stats to get a top like monitor of your running containers. The output looks like:
$ docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
729e4e0db0a9 dev 0.30% 2.876GiB / 3.855GiB 74.63% 25.3MB / 4.23MB 287kB / 16.4kB 77
This is how I discovered that docker run --memory 4096m richardbronosky/node_build_box npm run install was not getting 4G of memory because Docker was configured to limit to 2G of memory. (In the example above this has been corrected.) Without that insite I was totally lost as to why my process was ending with simply "killed".
Worked for me in the container, thanks for the ideas Sebastian
#!/bin/sh
function memory_limit
{
awk -F: '/^[0-9]+:memory:/ {
filepath="/sys/fs/cgroup/memory"$3"/memory.limit_in_bytes";
getline line < filepath;
print line
}' /proc/self/cgroup
}
if [[ $(memory_limit) < 419430400 ]]; then
echo "Memory limit was set too small. Minimum 400m."
exit 1
fi
Previously the /sys/fs/cgroup/memory/memory.limit_in_bytes worked for me, but in my ubuntu with kernel 5.8.0-53-generic seems that the correct endpoint now is /sys/fs/cgroup/memory.max to recover the memory limit from inside the container.
You have to check all values from the path defined in /prof/self/cgroup (example: /sys/fs/cgroup/memory/user.slice/user-1501.slice/session-99.scope) up to /sys/fs/cgroup/memory and look for the minimum. Here is the script:
#!/bin/bash
function memory_limit {
[ -r /proc/self/cgroup ] || { echo >&2 "Cannot read /proc/self/cgroup" ; return 1; }
path=$(grep -Poh "memory:\K.*" /proc/self/cgroup)
[ -n "$path" ] || { echo >&2 "Cannot get memory constrains from /proc/self/cgroup" ; return 1; }
full_path="/sys/fs/cgroup/memory${path}"
cd "$full_path" || { echo >&2 "cd $full_path failed" ; return 1; }
[ -r memory.limit_in_bytes ] || { echo >&2 "Cannot read 'memory.limit_in_bytes' at $(pwd)" ; return 1; }
min=$(cat memory.limit_in_bytes)
while [[ $(pwd) != /sys/fs/cgroup/memory ]]; do
cd .. || { echo >&2 "cd .. failed in $(pwd)" ; return 1; }
[ -r memory.limit_in_bytes ] || { echo >&2 "Cannot read 'memory.limit_in_bytes' at $(pwd)" ; return 1; }
val=$(cat memory.limit_in_bytes)
(( val < min )) && min=$val
done
echo "$min"
}
memory_limit
21474836480
In my situation, I have
cat /proc/self/cgroup
3:memory:/user.slice/user-1501.slice/session-99.scope
cat /sys/fs/cgroup/memory/user.slice/user-1501.slice/session-99.scope/memory.limit_in_bytes
9223372036854771712
cat /sys/fs/cgroup/memory/user.slice/user-1501.slice/memory.limit_in_bytes
21474836480 <= actual limit
cat /sys/fs/cgroup/memory/user.slice/memory.limit_in_bytes
9223372036854771712
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
9223372036854771712
Thanks to Mandragor for the original idea.