Docker's --memory switch does not work as expected - memory

Preamble
I'm writing a little presentation to list some of the "gotchas" when using Docker, and I've run into one of my own.
While explaining the perils of letting Docker run without memory restrictions I've found it doesn't behave the way I expected.
I created a docker image with a PHP script that will recursively create arrays to use up memory, reporting on how much memory is currently used. The script kills itself at 1gb if no memory restriction is put in place.
You can get the image here: https://hub.docker.com/r/gisleburt/my-memory-hog
You can see the source code here: https://github.com/Gisleburt/my-memory-hog
My expectation was that the following would prevent the script going beyond 128mb of memory.
docker run -it --memory=128m --memory-swap=0 gisleburt/my-memory-hog
However, on OSX (both native, and docker-machine using the virtual box driver), the docker actually the script just before 250mb. On Ubuntu the script reaches 1gb and kills itself.
Weirder still, if we inspect what's happening with docker stats, we can see that the container actually doesn't go over it's memory limit. In fact, running with --memory=4m on Ubuntu, docker stats still says it doesn't use more than the 4mb, even though the script completes at 1gb. On Mac, restricting to 4mb kills the script around the 5-6mb mark.
I assume I'm missing something about the way that memory is allocated, but I can't work out what it is.
Question
Why does the script in the docker container appear to use more memory than the docker container? What is happening with the memory inside and outside of the container?
Version Info
Mac Client:
Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Thu Oct 27 00:09:21 2016
OS/Arch: darwin/amd64
Experimental: true
Mac Native:
Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 23:26:11 2016
OS/Arch: linux/amd64
Mac Virtual Box Machine:
Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Thu Oct 27 00:09:21 2016
OS/Arch: linux/amd64
Experimental: true
Ubuntu Client:
Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 22:01:48 2016
OS/Arch: linux/amd64
Ubuntu Engine:
Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 22:01:48 2016
OS/Arch: linux/amd64

With the help of a number of people in our tech department, and in particular one member of our platform operations team, we worked out what's going on, and it is mentioned in the reference guide.
--memory-swap Total memory limit (memory + swap, format: <number>[<unit>]). Number is a positive integer. Unit can be one of b, k, m, or g.
To put is plainly, --memory-swap is the total amount of memory available, both resident, and swap.
In order to get the desired effect, you must run:
docker run -it --memory=128m --memory-swap=128m gisleburt/my-memory-hog
In this example memory = 128mb, and memory + swap = 128mb therefore swap = 0.
It also makes sense that you can not specify a number for --memory-swap that is greater than 0, but less that --memory.
Additionally, while --memory-swap=-1 gives you an "infinite" amount of memory via swap, for some reason, setting --memory-swap=0 seems to make memory and swap the same size, giving you a total amount of memory twice what you were expecting.
You can also turn off swapping using --memory-swappiness=0. Although the swap file might be created, it will not be written to, and therefore will not take up any further disk space than an empty file.
TL;DR:
--memory = RES
--memory-swap = VIRT = RES + SWAP

Related

Why docker container update is not stable?

I use the command docker container update --cpus 1 target_container to update the setting. But in most cases it not working and response "you must provide one or more flags when using this command", but somethings it working
Any one know the reason?
Some log is like,
$ docker container update --cpus 0.5 target_container
you must provide one or more flags when using this command
$ docker container update --cpus 0.5 target_container
you must provide one or more flags when using this command
$ docker container update --cpus 0.5 target_container
target_container
the docker version is
Client:
Version: 18.09.7
API version: 1.39
Go version: go1.10.4
Git commit: 2d0083d
Built: Fri Aug 16 14:19:38 2019
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.09.7
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 2d0083d
Built: Thu Aug 15 15:12:41 2019
OS/Arch: linux/amd64
Experimental: false
The docker update man page includes:
--cpus API 1.29+
A docker version can show you if your API is equal or above the API 29, which was with Docker 17.05.
For testing, try --cpus=1 or --cpus=0.5, considering the argument is supposed to be "number of CPUs"
As usual with commands including an hyphen: don't copy-paste it, copy it manually.

Remove Docker limit-cpu, limit-memory, reserve cpu and memory

I am currently using a Docker Swarm with limit-cpu, limit-memory and reserve cpu and memory.
After done my testing I would like to remove these configurations. I couldn't find any article for how to remove these.
Is there anyway to remove these setting by update without delete and recreate the docker service?
Docker version
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
Built: Tue Oct 2 18:06:45 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a/18.06.1-ce
Built: Tue Oct 2 18:08:26 2018
OS/Arch: linux/amd64
Experimental: false
Please help.
Thanks
You can use docker service update full documentation can be found here -> https://docs.docker.com/engine/reference/commandline/service_update/
To update or remove the cpu and memory limits you could run this command
docker service update --limit-cpu 0 --limit-memory 0 --reserve-cpu 0 --reserve-memory 0 service_name

Running windows docker in virtual machine crashing

I am running windows docker on a virtual machine and when i try to run docker run, it crashes the vm as well saying unhandled exception. And get logged out of the vm
The vm is windows server 2016 running on a host that is 2012.
The docker version info
Client:
Version: 17.06.2-ee-6
API version: 1.30
Go version: go1.8.3
Git commit: e75fdb8
Built: Mon Nov 27 22:46:09 2017
OS/Arch: windows/amd64
Server:
Version: 17.06.2-ee-6
API version: 1.30 (minimum version 1.24)
Go version: go1.8.3
Git commit: e75fdb8
Built: Mon Nov 27 22:55:16 2017
OS/Arch: windows/amd64
Experimental: false
The docker logs get corrupted for this specific container I am trying to run each time that I try to run the docker. So i cant diagnose what happened. Please suggest what i can do.
In a windows OS I ended up running docker inside a vagrant debian/centos box.
It's ligthweight and never had a problem.

Docker push incredibly slow to google registry

I have Google Instance and a Google Docker Registry configured.
Things where working quite smoothly and suddenly when I try to push to the
registry, it became practically unusable because of the speed.
$ docker push gcr.io/<ID>/my-registry/image:44]
The push refers to a repository [gcr.io/<ID>/my-registry/image:44]
3bc2db43a686: Pushing [=======> ] 3.409MB/23.32MB
1302d8ee9a47: Layer already exists
29d71372a492: Layer already exists
And if I measure the bandwidth of the docker push I get
NetHogs version 0.8.1
PID USER PROGRAM DEV SENT RECEIVED
1521 root /usr/bin/dockerd ens4 24.183 0.458 KB/sec
But my actual connection to the internet is good enough.
$ speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Google Cloud (x.x.x.x)...
Selecting best server based on latency...
Hosted by KamaTera INC (Santa Clara, CA) [10.61 km]: 20.436 ms
Testing download speed........................................
Download: 2046.49 Mbit/s
Testing upload speed..................................................
Upload: 114.68 Mbit/s
I tried restarting the docker daemon and nothing.
This are the tools I am using
$ docker version
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:18 2017
OS/Arch: linux/amd64
Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:40:56 2017
OS/Arch: linux/amd64
Experimental: false
$ gcloud version
Google Cloud SDK 176.0.0
alpha 2017.10.13
beta 2017.10.13
bq 2.0.27
core 2017.10.13
gsutil 4.27
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
Any idea what could be the problem?
Use gcloud builds submit instead of docker push
Docs: https://cloud.google.com/sdk/gcloud/reference/builds/submit
I tried using docker push to get images to GCR, but it was incredibly slow and would eventually grind to a halt. I then started using GCE instances to docker push from, which was very fast. Later, I stumbled upon gcloud builds submit. It runs fast locally for me.

Can't run the elasticsearch image on Raspberry Pi

I get an error when trying to run the elasticsearch image on my raspberry pi. It runs fine on my mac, so it is probably operating system specific. I can run other images.
pi#raspberrypi:~/folder/folder$ docker run
node:latest
pi#raspberrypi:~/folder/folder$ docker run
docker.elastic.co/elasticsearch/elasticsearch:5.6.3
standard_init_linux.go:195: exec user process caused "exec format
error"
Having searched extensively for standard_init_linux.go:195 it has yielded very little, apart from maybe having something to do with the qemu libray, from this discussion, but don't know enough about linux OS to really understand that.
pi#raspberrypi:~/folder/folder$ docker version
Client:
Version: 17.10.0-ce
API version: 1.33
Go version: go1.8.3
Git commit: f4ffd25
Built: Tue Oct 17 19:13:44 2017
OS/Arch: linux/arm
Server:
Version: 17.10.0-ce
API version: 1.33 (minimum version 1.12)
Go version: go1.8.3
Git commit: f4ffd25
Built: Tue Oct 17 19:06:18 2017
OS/Arch: linux/arm
Experimental: false
pi#raspberrypi:~/folder/folder$ uname -a
Linux raspberrypi 4.9.41-v7+ #1023 SMP Tue Aug 8 16:00:15 BST 2017 armv7l GNU/Linux
Could it be to do with the Go version?
The docker image is for i386 architectures. The raspberry pi is arm and it's not supported.
Probably you'll find something from hypriot images or you can try to use charlesyan/rpi-elasticsearch image.

Resources