Docker stats show memory usage less than output of top command - docker

Question

I run my service app in docker container app-api
Result of top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6420 root 20 0 30.572g 0.028t 38956 S 47.8 92.5 240:40.95 app
...
Result of htop:
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
6420 root 20 0 30.6G 29.0G 38956 S 47.1 92.5 4h21:53 app
6554 root 20 0 30.6G 29.0G 38956 S 6.6 92.5 23:04.15 app
6463 root 20 0 30.6G 29.0G 38956 S 2.0 92.5 27:29.53 app
6430 root 20 0 30.6G 29.0G 38956 S 0.0 92.5 25:30.61 app
6429 root 20 0 30.6G 29.0G 38956 S 5.3 92.5 26:36.17 app
6428 root 20 0 30.6G 29.0G 38956 S 10.0 92.5 23:56.10 app
6426 root 20 0 30.6G 29.0G 38956 S 6.0 92.5 8:09.12 app
6427 root 20 0 30.6G 29.0G 38956 S 0.0 92.5 23:03.81 app
6425 root 20 0 30.6G 29.0G 38956 S 0.0 92.5 0:00.00 app
6424 root 20 0 30.6G 29.0G 38956 S 0.0 92.5 25:42.46 app
6423 root 20 0 30.6G 29.0G 38956 S 4.6 92.5 26:10.82 app
6422 root 20 0 30.6G 29.0G 38956 S 12.0 92.5 23:24.68 app
6421 root 20 0 30.6G 29.0G 38956 S 2.0 92.5 4:32.47 app
2202 gitlab-ru 20 0 231M 70132 53620 S 5.3 0.2 4h54:21 nginx: worker process
2203 gitlab-ru 20 0 228M 59040 47680 S 0.7 0.2 54:44.83 nginx: worker process
281 root 19 -1 175M 58104 47728 S 0.0 0.2 0:17.76 /lib/systemd/systemd-journald
1036 root 20 0 1893M 38164 13332 S 0.0 0.1 0:38.17 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
...
Result of docker stats:
$ docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
14654b8a4bfb app-letsencrypt 13.08% 244.5MiB / 31.41GiB 0.76% 183GB / 192GB 12.4GB / 4.64MB 23
a932dabbced8 app-api 60.50% 7.258GiB / 31.41GiB 23.10% 53.2GB / 10.6GB 48.1MB / 0B 14
2cebc542dda6 app-redis 0.12% 3.902MiB / 31.41GiB 0.01% 24.2kB / 0B 1.84GB / 655kB 4
As you are can see 0.028t (~29G) in top in much more than 7.258GiB in docker stats. Difference is about 29 - 7.258 > 20G of RAM.
Help me please to understand how to detect what is this phantom that takes 20G of RAM? Or maybe point me where to dig, into problems with my application or with docker (20.10.1) or with operation system (Ubuntu 18.04)?
UPD
Output in pprof (heap)
# runtime.MemStats
# Alloc = 7645359160
# TotalAlloc = 2552206192400
# Sys = 31227357832
# Lookups = 0
# Mallocs = 50990505448
# Frees = 50882282691
# HeapAlloc = 7645359160
# HeapSys = 29526425600
# HeapIdle = 21707890688
# HeapInuse = 7818534912
# HeapReleased = 9017090048
# HeapObjects = 108222757
# Stack = 1474560 / 1474560
# MSpan = 101848496 / 367820800
# MCache = 13888 / 16384
# BuckHashSys = 10697838
# GCSys = 1270984696
# OtherSys = 49937954
# NextGC = 11845576832
# LastGC = 1615583458331895138
# PauseNs = ..................
# NumGC = 839
# NumForcedGC = 0
# GCCPUFraction = 0.027290987331299785
# DebugGC = false
# MaxRSS = 31197982720

Answer 1

docker stats is reporting the cgroup resource usage of the container's cgroup:
$ docker run -it -m 1g --cpus 1.5 --name test-stats busybox /bin/sh
/ # cat /sys/fs/cgroup/memory/memory.usage_in_bytes
2629632
/ # cat /sys/fs/cgroup/memory/memory.limit_in_bytes
1073741824
/ # cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
150000
/ # cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
100000
From another window (there's a small variation with the cat command stopped):
$ docker stats --no-stream test-stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
9a69d1323422 test-stats 0.00% 2.395MiB / 1GiB 0.23% 5.46kB / 796B 3MB / 0B 1
Note that this is will differ from the overall host memory and cpu if you have specified limits with your containers. Without limits, the cpu quota will be -1 to be unrestricted, and the memory limit will set to the page counter max value.
Trying to add up memory usage from the top command is very error prone. There is different types of memory in the linux kernel (including disk cache), memory gets shared between multiple threads (which is why you likely see multiple pids for app, each with the exact same memory), some memory may be mmap that is not backed with ram, and a long list of other challenges. People that know much more about this than me will say that the kernel doesn't even know when it's actually out of memory until it attempts to reclaim memory from many process and those attempts all fail.

Answer 2

You are comparing top/htop RES mem (man):
The non-swapped physical memory a task has used.
RES = CODE + DATA.
with docker stats CLI output (doc):
On Linux, the Docker CLI reports memory usage by subtracting cache usage from the total memory usage.
Use docker stats API and you will get much more granular view, e.g. stat for memory:
{
"total_pgmajfault": 0,
"cache": 0,
"mapped_file": 0,
"total_inactive_file": 0,
"pgpgout": 414,
"rss": 6537216,
"total_mapped_file": 0,
"writeback": 0,
"unevictable": 0,
"pgpgin": 477,
"total_unevictable": 0,
"pgmajfault": 0,
"total_rss": 6537216,
"total_rss_huge": 6291456,
"total_writeback": 0,
"total_inactive_anon": 0,
"rss_huge": 6291456,
"hierarchical_memory_limit": 67108864,
"total_pgfault": 964,
"total_active_file": 0,
"active_anon": 6537216,
"total_active_anon": 6537216,
"total_pgpgout": 414,
"total_cache": 0,
"inactive_anon": 0,
"active_file": 0,
"pgfault": 964,
"inactive_file": 0,
"total_pgpgin": 477
}
You can see - memory is not just one, but it has many types and each tool may report own set&combination of memory types. I guess you will find missing memory in app cache memory allocation.
You can check overall basic memory allocations with free command:
$ free -m
total used free shared buff/cache available
Mem: 2000 1247 90 178 662 385
Swap: 0 0 0
It is a normal state, when Linux uses unused memory for buff/cache.