I am using db2 on docker with a self non-root installation.
Even if I set the INSTANCE_MEMORY to the minimum, it seems to reserve 4G of RAM on the server.
How can DB2 be aware of the limits setted in the docker-compose file as I run the database in a docker swarm cluster as a STACK?
The DB2 version is 11.1.4FP4.
docker --version
Docker version 18.03.1-ce, build 9ee9f40
When I look at the docker stats, it uses only about 80MiB.
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
8282c5d0c9e7 db2wse_db2.1.waalf6vljuapnxlvzhf2cb0uv 0.21% 76.83MiB / 1GiB 7.50% 0B / 0B 408MB / 6.86GB 56
My docker-compose.yml file
version: "3.3"
services:
db2:
image: docker-registry.ju.globaz.ch:5000/db2wse:11.1.4fp4
networks:
- network-db
deploy:
mode: replicated
replicas: 1
resources:
limits:
memory: 1G
networks:
network-db:
external: true
Any idea ? It is very frustrating.
Thanks a lot
I found the problem.
It was a kernel configuration in the sysctl.conf.
I had this :
cat /etc/sysctl.conf |grep vm.
vm.swappiness=10
vm.overcommit_memory=2
vm.dirty_ratio=2
vm.dirty_background_ratio=1
I removed everything setted for DB2 (put back the default configuration) and now I can take advantage of all the RAM of the hosts.
I kept this :
cat /etc/sysctl.conf |grep vm.
vm.swappiness=10
vm.max_map_count=262144
Thanks
Related
I've run mongodb service via docker-compose like this:
version: '2'
services:
mongo:
image: mongo
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
mem_limit: 4GB
If I run docker stats I can see 4 GB allocated:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
cf3ccbd17464 michal_mongo_1 0.64% 165.2MiB / 4GiB 4.03% 10.9kB / 4.35kB 0B / 483kB 35
But I run this command I get RAM from my laptop which is 32 GB:
~$ docker exec michal_mongo_1 free -g
total used free shared buff/cache available
Mem: 31 4 17 0 9 24
Swap: 1 0 1
How does mem_limit affect the memory size then?
free (and other utilities like top) will not report correct numbers inside a memory-constraint container because it gathers its information from /proc/meminfo which is not namespaced.
If you want the actual limit, you must use the entries populated by cgroup pseudo-filesystem under /sys/fs/cgroup.
For example:
docker run --rm -i --memory=128m busybox cat /sys/fs/cgroup/memory/memory.limit_in_bytes
The real-time usage information is available under /sys/fs/cgroup/memory/memory.stat.
You will probably need the resident-set-size (rss), for example (inside the container):
grep -E -e '^rss\s+' /sys/fs/cgroup/memory/memory.stat
For a more in-depth explanation, see also this article
I have swarm of two nodes
[ra#speechanalytics-test ~]$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
mlwwmkdlzbv0zlapqe1veq3uq speechanalytics-preprod Ready Active 18.09.3
se717p88485s22s715rdir9x2 * speechanalytics-test Ready Active Leader 18.09.3
I am trying to run container with elastic in stack. Here is my docker-compose.yml file
version: '3.4'
services:
elastic:
image: docker.elastic.co/elasticsearch/elasticsearch:6.7.0
environment:
- cluster.name=single-node
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata:/usr/share/elasticsearch/data
deploy:
placement:
constraints:
- node.hostname==speechanalytics-preprod
volumes:
esdata:
driver: local
after start with docker stack
docker stack deploy preprod -c docker-compose.yml
container crashes in 20 seconds
docker service logs preprod_elastic
...
| OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
| OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
| [2019-04-03T16:41:30,044][WARN ][o.e.b.JNANatives ] [unknown] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
| [2019-04-03T16:41:30,049][WARN ][o.e.b.JNANatives ] [unknown] This can result in part of the JVM being swapped out.
| [2019-04-03T16:41:30,049][WARN ][o.e.b.JNANatives ] [unknown] Increase RLIMIT_MEMLOCK, soft limit: 16777216, hard limit: 16777216
| [2019-04-03T16:41:30,050][WARN ][o.e.b.JNANatives ] [unknown] These can be adjusted by modifying /etc/security/limits.conf, for example:
| OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
| # allow user 'elasticsearch' mlockall
| OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
| elasticsearch soft memlock unlimited
| [2019-04-03T16:41:02,949][WARN ][o.e.b.JNANatives ] [unknown] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
| elasticsearch hard memlock unlimited
| [2019-04-03T16:41:02,954][WARN ][o.e.b.JNANatives ] [unknown] This can result in part of the JVM being swapped out.
| [2019-04-03T16:41:30,050][WARN ][o.e.b.JNANatives ] [unknown] If you are logged in interactively, you will have to re-login for the new limits to take effect.
| [2019-04-03T16:41:02,954][WARN ][o.e.b.JNANatives ] [unknown] Increase RLIMIT_MEMLOCK, soft limit: 16777216, hard limit: 16777216
preprod
on both nodes I have
ra#speechanalytics-preprod:~$ sysctl vm.max_map_count
vm.max_map_count = 262144
Any ideas how to fix ?
The memlock errors you're seeing from Elasticsearch is a common issue not unique to having used Docker, but occurs when Elasticsearch is told to lock its memory, but is unable to do so. You can circumvent the error by removing the following environment variable from the docker-compose.yml file:
- bootstrap.memory_lock=true
Memlock may be used with Docker Swarm Mode, but with some caveats.
Not all options that work with docker-compose (Docker Compose) work with docker stack deploy (Docker Swarm Mode), and vice versa, despite both sharing the docker-compose YAML syntax. One such option is ulimits:, which when used with docker stack deploy, will be ignored with a warning message, like so:
Ignoring unsupported options: ulimits
My guess is that with your docker-compose.yml file, Elasticsearch runs fine with docker-compose up, but not with docker stack deploy.
With Docker Swarm Mode, by default, the Elasticsearch instance as you have defined will have trouble with memlock. Currently, setting of ulimits for docker swarm services is not yet officially supported. There are ways to get around the issue, though.
If the host is Ubuntu, unlimited memlock can be enabled across the docker service (see here and here). This can be achieved via the commands:
echo -e "[Service]\nLimitMEMLOCK=infinity" | SYSTEMD_EDITOR=tee systemctl edit docker.service
systemctl daemon-reload
systemctl restart docker
However, setting memlock to infinity is not without its drawbacks, as spelt out by Elastic themselves here.
Based on my testing, the solution works on Docker 18.06, but not on 18.09. Given the inconsistency and the possibility of Elasticsearch failing to start, the better option would be to not use memlock with Elasticsearch when deploying on Swarm. Instead, you can opt for any of the other methods mentioned in Elasticsearch Docs to achieve similar results.
I used docker-compose to run a service but it crashed, I entered the container and got resource info by 'top' as below.
top - 13:43:25 up 1 day, 6:46, 0 users, load average: 1.82, 0.74, 0.52
Tasks: 3 total, 1 running, 2 sleeping, 0 stopped, 0 zombie
%Cpu(s): 32.2 us, 22.4 sy, 0.0 ni, 40.1 id, 3.0 wa, 0.0 hi, 2.3 si, 0.0 st
KiB Mem: 2047040 total, 1976928 used, 70112 free, 172 buffers
KiB Swap: 1048572 total, 1048572 used, 0 free. 14588 cached Mem
So I think my docker is out of memory.
I've tried add
mem_limit: 5g
memswap_limit: 5g
mem_reservation: 5g
into docker-compose.yml
But it seems not work. My question is, how to enlarge docker's memory limit by docker-compose.
The docker engine has a compatiblity mode which aims to make transition from compose v2 files to v3 easier. As a result of that, it is possible to partially use the swarm config (deploy) to specifiy resource limits for standard docker-compose usage:
To run in compatiblity mode just add the --compatiblity flag like this:
docker-compose --compatibility up myservice
and you can use a compose file like this:
version: '3.5'
services:
myservice:
image: postgres:12-alpine
deploy:
resources:
limits:
cpus: '0.50'
memory: 50M
If this is on a machine running Docker Desktop, then you would have to open the Docker Desktop preferences and go to Resources section to tweak how much of your host resources the Docker Engine can use.
As stated in the documentation the following fields can be used in docker-compose to control memory and cpu resources.
deploy:
resources:
limits:
cpus: '0.001'
memory: 50M
reservations:
cpus: '0.0001'
memory: 20M
Note however, by default there are no limits for the container memory usage, so setting the memory flags will unlikely help.
Trying to run cluster application on different virtual machines with use of Swarm stand alone and docker-compose version '2'. Overlay network is set. But want to force certain containers to run on specific hosts.
In documentation there is following advice, but with this parameter I was not able to start any container at all:
environment:
- "constraint:node==node-1"
ERROR: for elasticsearch1 Cannot create container for service elasticsearch1: Unable to find a node that satisfies the following conditions
[available container slots]
[node==node-1]
Should we register hosts as node-1 node-2... or it is done by default.
[root#ux-test14 ~]# docker node ls
Error response from daemon: 404 page not found
[root#ux-test14 ~]# docker run swarm list
[root#ux-test14 ~]#
[root#ux-test14 ~]# docker info
Containers: 8
Running: 6
Paused: 0
Stopped: 2
Images: 8
Server Version: swarm/1.2.5
Role: primary
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint
Nodes: 2
ux-test16.rs: 10.212.212.2:2375
â ID: JQPG:GKFF:KJZJ:AY3N:NHPZ:HD6J:SH36:KEZR:2SSH:XF65:YW3N:W4DG
â Status: Healthy
â Containers: 4 (4 Running, 0 Paused, 0 Stopped)
â Reserved CPUs: 0 / 2
â Reserved Memory: 0 B / 3.888 GiB
â Labels: kernelversion=3.10.0-327.28.3.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
â UpdatedAt: 2016-09-05T11:11:31Z
â ServerVersion: 1.12.1
ux-test17.rs: 10.212.212.3:2375
â ID: Z27V:T5NU:QKSH:DLNK:JA4M:V7UX:XYGH:UIL6:WFQU:FB5U:J426:7XIR
â Status: Healthy
â Containers: 4 (2 Running, 0 Paused, 2 Stopped)
â Reserved CPUs: 0 / 2
â Reserved Memory: 0 B / 3.888 GiB
â Labels: kernelversion=3.10.0-327.28.3.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
â UpdatedAt: 2016-09-05T11:11:17Z
â ServerVersion: 1.12.1
Plugins:
Volume:
Network:
Swarm:
NodeID:
Is Manager: false
Node Address:
Security Options:
Kernel Version: 3.10.0-327.28.3.el7.x86_64
Operating System: linux
Architecture: amd64
CPUs: 4
Total Memory: 7.775 GiB
Name: 858ac2fdd225
Docker Root Dir:
Debug Mode (client): false
Debug Mode (server): false
WARNING: No kernel memory limit support
My first answer is about "swarm mode". You'd since clarified that you're using legacy Swarm and added more info, so here:
The constraint you list assumes that you have a host named node-1. Your hosts are named ux-test16.rs and ux-test17.rs. Just use that instead of node-1 in your constraint. Eg:
environment:
- "constraint:node==ux-test16.rs"
The environment variable constraint is only valid for the legacy (stand alone) version of Swarm. The newer "Swarm Mode" uses either mode or constraints options (not environment variables).
To enforce one and only one task (container) per node, use mode=global.
docker service create --name proxy --mode global nginx
The default mode is replicated which means that the swarm manager will create tasks (containers) across all available nodes to meet the number specified in the --replicas option. Eg:
docker service create --name proxy --replicas 5 nginx
To enforce other constraints based on hostname (node), label, role, id's use the --constraint option. Eg:
docker service create --name proxy --constraint "node.hostname!=node01" nginx
See https://docs.docker.com/engine/reference/commandline/service_create/#/specify-service-constraints
EDIT sept 2016:
Something else. docker-compose is not currently supported in "swarm mode". Swarm mode understands the new dab format instead. There is a way to convert docker-compose files to dab but it's experimental and not to be relied on at this point. It's better to create a bash script that calls all the docker service create ... directly.
EDIT March 2017:
As of docker 1.13 (17.03), docker-compose can now be used to provision swarm environments directly without having to deal with the dab step.
Related issue - I had a recent Swarm project with a mixture of worker nodes (3 x Linux + 4 x Windows). My containers needed to run on a specific OS, but not on any specific node. Swarm mode now supports specifying an OS under "constraints" in docker-compose files. No need to create labels for each node:
version: '3'
services:
service_1:
restart: on-failure
image: 'service_1'
deploy:
placement:
constraints:
- node.platform.os == windows
junittestsuite:
restart: on-failure
image: 'junit_test_suite:1.0'
command: ant test ...
deploy:
placement:
constraints:
- node.platform.os == linux
mem_limit is supported by docker-compose? How can I test it?
I have a following docker-compose.yml
repository:
image: myregistry/my_nginx_image
mem_limit: 60m
volumes:
- /etc/localtime:/etc/localtime
ports:
- "80:80"
How can I prove that the container actually does not exceed 60 mb of RAM?
I am using:
docker 1.3.1
docker-compose 1.1.0
Yes. Memory limitation is supported by docker-compose and value can be set as in your example with "m" for megabytes.
It is possible to check what is the memory limit set for running Docker container using "docker stats" command.
If your container name is "repository_1" then use this command:
docker stats repository_1
The result of this will be simillar to this one:
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
repository_1 1.93% 4.609 MiB/60 MiB 7.20% 1.832 KiB/648 B
According to documentation, simple
mem_limit: 1000000000
should be enough. I guess, you should drop "m", and use bytes instead of megabytes.
If you use docker compose v3 instead of:
mem_limit: 60m
you need to specify memory limit in deploy section:
deploy:
resources:
limits:
memory: 60m
This change is described here: https://docs.docker.com/compose/compose-file/compose-versioning/#version-2x-to-3x
You can find how configure docker to limit resources (CPU & MEMORY) and how test your restrictions in this post written last year: resource-management-in-docker.