Docker stats for n seconds period - docker

I am running docker stats $CONTAINER_ID from a shell script to monitor my Docker container memory and CPU usage over a period of 1 hour. I have the below shell script.
#!/bin/sh
# First, start the container
CONTAINER_ID=abcadsasdasd
# Then start watching that it's running (with inspect)
while [ "$(docker inspect -f {{.State.Running}} $CONTAINER_ID 2>/dev/null)" = "true" ]; do
# And while it's running, check stats
#docker stats $CONTAINER_ID 2>&1 | tee "$1"
docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}\t{{.BlockIO}}\t{{.PIDs}}" $CONTAINER_ID 2>&1 | tee "$1"
sleep 5
done
It is running fine. But, it seems it is running at every second. I need that it outputs at every 5 seconds. I found that there is no such option like specifying timeperiod with docker stats command. Request some suggestions to achieve it. It seems sleep 5 is not having any effect.
Edit
Even with 1 second delay, I expected 60 lines in my log file, With 5 seconds, I expected 12 lines over 1 minute period. But, I am getting close to 150 lines for 1 minute.

Without sleep, just read current values from the file ($1) and average them every period you choose:
#!/bin/bash
multiLine="";
format="table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}";
docker stats --all --format "$format" --no-trunc | (while read line; do
sedLine=$(echo "$line" | sed "s/^.*name.*cpu.*mem.*$/_divider_/i")
if [ "$sedLine" != "_divider_" ];
then
multiLine="${multiLine}"'\n'"${line}";
else
echo -e $multiLine > $1;
multiLine="";
fi;
done);

docker stats $container won't exit when running, so your sleep 5 don't have chance to execute.
For you, you should use next:
--no-stream Disable streaming stats and only pull the first result
Then, you could use something like next fake code to control the rate:
while xxx; do
docker stats $container --no-stream
sleep 5
done

Related

Mechanism to check if works is being done on docker containers

I have couple of python docker containers, processing data from kafka topics every X seconds.
If the data isn't appearing for i.e. more than 2X time, I should log it somewhere (so I can check which kafka producers to restart - doesn't matter here).
I was thinking about creating some file, where I would keep last message timestamp, and another script/container would check it every X seconds, to find out, if the data should already have appeared. I could check docker logs aswell, if "amount" of logs is increasing (if not, then it means, that no data is appearing).
While those solutions would probably work, I'm not sure, if there isn't already another "ready-made" solution for that issue. Is there something, that I can use to check if the data is being processed?
You can use some sort of health check mechanism by putting logs inside: <filename_1>
then the healthcheck would be:
if [[ `[ -r <filename_1> ] && cat <filename_1> | wc -l` -ge 1 ]]; then > <filename_1> && echo 0; else echo 1; fi
By doing this you check if the file exist then check the number of lines. If greater/equal than 1 you empty it then return 0 .If empty or does not exist you return 1
This will mark your docker as healthy or unheathly.
If you want to restart by itself you can replace echo 1 by kill 1 kill 1 will kill process with pid 1 in container and force container exit. Usually the process started by CMD or ENTRYPOINT has pid 1.
[[ `[ -r <filename_1> ] && cat <filename_1> | wc -l` -ge 1 ]] && > <filename_1> && echo 0 || kill 1
Don't forget --restart always option.

Makefile docker wait for database to be ready

I'm attempting to create a makefile that will launch my db container, wait for it to complete before launching the rest of my app.
I have 2 compose files.
docker-compose.db.yml
docker-compose.yml
My make file is as follows:
default:
#echo "Preparing database"
docker-compose -f docker-compose.db.yml build
docker-compose -f docker-compose.db.yml pull
docker-compose -f docker-compose.db.yml up -d
#echo ""
#echo "Waiting for database \"ready for connections\""
#while [ -z "$(shell docker logs $(PROJECT_NAME)_mariadb 2>&1 | grep -o "ready for connections")" ]; \
do \
sleep 5; \
done
#echo "Database Ready for connections!"
#echo ""
#echo "Launching App Containers"
docker-compose build
docker-compose pull
docker-compose up -d
What happens is that it immediately goes to "Database Ready for connections!" even before the database is ready. If I run the same command in terminal it response with empty for about the first 20 seconds and then finally returns "ready for connections".
Thank you in advance
The GNU make $(shell ...) function gets run once when the Makefile is processed. So when your rule has
#while [ -z "$(shell docker logs $(PROJECT_NAME)_mariadb 2>&1 | grep -o "ready for connections")" ]
Make first runs the docker logs command on its own, then substitutes the result in the shell command it runs
while [ -z "ready for connections" ]
which is trivially false, and the loop exits immediately.
Instead you probably want to escape the $ in the shell substitution command
#while [ -z "$$(docker-compose logs mariadb ...) "]
It's fairly typical to configure containers to be able to wait for the database startup themselves, and to run the application and database from the same docker-compose.yml file. Docker Compose wait for container X before starting Y describes this setup.

how to load all saved docker images in parallel

I have 20 images TARed, now I want to load those images on another system. However, loading itself is taking 30 to 40 minutes. All images are independent of each other so all images loading should happen in parallel, I believe.
I tried solution like running load command in background(&) and wait till loading finishes, but observed that it is taking even more time. Any help here is highly appreciated.
Note:- not sure about the option -i to docker load command.
Try
find /path/to/image/archives/ -iname "*.tar" -o -iname "*.tar.xz" |xargs -r -P4 -i docker load -i {}
This will load Docker image archives in parallel (adjust -P4 to the desired number of parallel loads or set to -P0 for unlimited concurrency).
For speeding up the pulling/saving processes, you can use ideas from the snippet below:
#!/usr/bin/env bash
TEMP_FILE="docker-compose.image.pull.yaml"
image_name()
{
local name="$1"
echo "$name" | awk -F '[:/]' '{ print $1 }'
}
pull_images_file_gen()
{
local from_file="$1"
cat <<EOF >"$TEMP_FILE"
version: '3.4'
services:
EOF
while read -r line; do
cat <<EOF >>"$TEMP_FILE"
$(image_name "$line"):
image: $line
EOF
done < "$from_file"
}
save_images()
{
local from_file="$1"
while read -r line; do
docker save -o /tmp/"$(image_name "$line")".tar "$line" &>/dev/null & disown;
done < "$from_file"
}
pull_images_file_gen "images"
docker-compose -f $TEMP_FILE pull
save_images "images"
rm -f $TEMP_FILE
images - contains needed Docker images names list line by line.
Good luck!

Openwrt Script - Autostartup Shadowsocks

I would like to create a script for openwrt that every day changes some variables inside the Shadowsocks service. This is the script but i don't know where to put it or how to manage to call it every day or every reboot of router.
#!/bin/sh /etc/rc.common
restart=0
for i in `uci show shadowsocks | grep alias | sed -r 's/.*\[(.*)\].*/\1/'`
do
server=$(uci get shadowsocks.#servers[${i}].alias)
result=$(nslookup $server)
new_ip=$(echo "${result}" | tail -n +3 | awk -F" " '/^Address 1/{ print $3}')
if [ -n "$new_ip" ]; then
logger -t shadowsocks "nslookup $server -> $new_ip"
old_ip=$(uci get shadowsocks.#servers[${i}].server)
if [ "$old_ip" != "$new_ip" ]; then
logger -t shadowsocks "detect $server ip address change ($old_ip -> $new_ip)"
restart=1
uci set shadowsocks.#servers[${i}].server=${new_ip}
fi
else
logger -t shadowsocks "nslookup $server fail"
fi
done
if [ $restart -eq 1 ]; then
logger -t shadowsocks "restart for server ip address change"
uci commit shadowsocks
/etc/init.d/shadowsocks restart
fi
You can use cron utility. Cron is a time-based job scheduler in Unix-like computer OS. It allows to run jobs/programs/scripts at specified times.
OpenWrt comes with a cron system by default, provided by busybox.
Cron is not enabled by default, so your jobs won't be run. To activate cron in Openwrt:
/etc/init.d/cron start
/etc/init.d/cron enable
Ref: https://oldwiki.archive.openwrt.org/doc/howto/cron
Now considering your question, if you want to run mentioned script every day:
Edit cron file using crontab -e command. And write below line.
0 0 * * * sh /path/to/your/script.sh
This command will run your script at 00:00 (every day mid night). You can easily modify the above command to schedule your job at any other time. Good reference to generate cron job entry: https://crontab.guru/
To see if crontab is working properly:
tail -f /var/log/syslog | grep CRON
Now coming to your second question "Run script at every reboot of router":
You can put your script in /etc/rc.local. This file will be executed as as a shell script on every boot up by /etc/rc.d/S95done in Openwrt. So just edit /etc/rc.local with sh /path/to/your/script.sh Make sure your script is executable and doing your task properly.

how to find MAX memory from docker stats?

With docker stats you can see the memory usage of a container over time.
Is there a way to find what the highest value of memory usage was while running docker stats?
If you need to find the peak usage you are better off requesting the .MemPerc option and calculating based on the total memory (unless you restricted the memory available to the container). .MemUsage has units which change during the life of the container which mess with the result.
docker stats --format 'CPU: {{.CPUPerc}}\tMEM: {{.MemPerc}}'
You can stream an ongoing log to a file (or script).
To get just the max memory as originally requested:
(timeout 120 docker stats --format '{{.MemPerc}}' <CONTAINER_ID> \
| sed 's/\x1b\[[0-9;]*[a-zA-Z]//g' ; echo) \
| tr -d '%' | sort -k1,1n | tail -n 1
And then you can ask the system for its total RAM (again assuming you didn't limit the RAM available to docker) and calculate:
awk '/MemTotal/ {print $2}' /proc/meminfo
You would need to know how long the container is going to run when using timeout as above, but if docker stats was run without this in background submitted by a script it could kill it once the container completed.
...
This command allows you to generate a time-series of the cpu/memory load:
(timeout 20 docker stats --format \
'CPU: {{.CPUPerc}}\tMEM: {{.MemPerc}}' <CONTAINER_ID> \
| sed 's/\x1b\[[0-9;]*[a-zA-Z]//g' ; echo) \
| gzip -c > monitor.log.gz
Note that it pipes into gzip. In this form you get ~2 rows per second so the file would get large rapidly if you don't.
I'd advise this for benchmarking and trouble shooting rather than use on production containers
I took a sampling script from here and aggregated data by #pl_rock. But be careful - the sort command only compares string values - so the results are usually wrong (but ok for me).
Also mind that docker is sometimes reporting wrong numbers (ie. more allocated mem than physical RAM).
Here is the script:
#!/bin/bash
"$#" & # Run the given command line in the background.
pid=$!
echo "" > stats
while true; do
sleep 1
sample="$(ps -o rss= $pid 2> /dev/null)" || break
docker stats --no-stream --format "{{.MemUsage}} {{.Name}} {{.Container}}" | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0 }' >> stats
done
for containerid in `awk '/.+/ { print $7 }' stats | sort | uniq`
do
grep "$containerid" stats | sort -r -k3 | tail -n 1
# maybe: | sort -r -k3 -h | head -n 1
# see comment below (didnt tested)
done
In my case I wanted to monitor a docker container which runs tests for my web application. The test suite is pretty big, it includes javascript tests in a real browser and consume significant amount of both, memory and time.
Ideally, I wanted to watch the current memory usage real time, but to also keep the history for later analysis.
I ended up using a modified and simplified version of the Keiran's solution:
CONTAINER=$(docker ps -q -f name=CONTAINER_NAME)
FORMAT='{{.MemPerc}}\t{{.MemUsage}}\t{{.Name}}'
docker stats --format $FORMAT $CONTAINER | sed -u 's/\x1b\[[0-9;]*[a-zA-Z]//g' | tee stats
Notes:
CONTAINER=$(docker ps -q -f name=NAME) find container by name, but there are other options
FORMAT='{{.MemPerc}} ...}} MemPerc goes first (for sorting); otherwise you can be creative
sed -u the -u flag is important, it turns off buffering
| sed -u 's/\x1b\[[0-9;]*[a-zA-Z]//g' removes ANSI escape sequences
| tee stats not only display in real time, but also write into the stats file
I Ctrl-C manually when it's ready – not ideal, but OK for me
after that it's easy to find the max with something like sort -n stats | tail
you can use command:
docker stats --no-stream | awk '{ print $3 }' | sed '1d'|sort | tail -1
It will give highest memory by container.
Let me Explain command:
--no-stream : Disable streaming stats and only pull the first result
awk '{ print $3 }' : will print MEM USAGE
sed '1d' : will delete first entry that is %
sort : it will sort the result
tail -1 : it will give last entry that is highest.

Resources