I need to import a large csv file into postges using copy.
I find that the processing time is a bit long. So I look at docker stats:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
682795abfc17 postgres 28.83% 1.034GiB / 15.63GiB 6.62% 23.7kB / 17.7kB 15.5GB / 14.4GB 11
I want it to be able to use more ram than 1Gib so I put this but no improvement:
database:
container_name: postgres_container
image: postgis/postgis:14-3.3
cpus: 4
how to make the container use more ram?
thank you
Related
I've run mongodb service via docker-compose like this:
version: '2'
services:
mongo:
image: mongo
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
mem_limit: 4GB
If I run docker stats I can see 4 GB allocated:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
cf3ccbd17464 michal_mongo_1 0.64% 165.2MiB / 4GiB 4.03% 10.9kB / 4.35kB 0B / 483kB 35
But I run this command I get RAM from my laptop which is 32 GB:
~$ docker exec michal_mongo_1 free -g
total used free shared buff/cache available
Mem: 31 4 17 0 9 24
Swap: 1 0 1
How does mem_limit affect the memory size then?
free (and other utilities like top) will not report correct numbers inside a memory-constraint container because it gathers its information from /proc/meminfo which is not namespaced.
If you want the actual limit, you must use the entries populated by cgroup pseudo-filesystem under /sys/fs/cgroup.
For example:
docker run --rm -i --memory=128m busybox cat /sys/fs/cgroup/memory/memory.limit_in_bytes
The real-time usage information is available under /sys/fs/cgroup/memory/memory.stat.
You will probably need the resident-set-size (rss), for example (inside the container):
grep -E -e '^rss\s+' /sys/fs/cgroup/memory/memory.stat
For a more in-depth explanation, see also this article
I have a few micro-services running in docker containers (One service in each container).
How do I find out the network usage between every two pair of docker containers? So that I can make a graph such that I have containers as vertices and on edges I have the amount of bytes transmitted/received.
I used cAdvisor, but it gives me the overall network usage of each container.
Start with docker stats:
$ docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
a645ca0d8feb wp_wordpress_1 0.00% 135.5MiB / 15.55GiB 0.85% 12.2MB / 5.28MB 2.81MB / 9.12MB 11
95e1649c5b79 wp_db_1 0.02% 203MiB / 15.55GiB 1.28% 6.44MB / 3.11MB 0B / 1.08GB 30
I am using db2 on docker with a self non-root installation.
Even if I set the INSTANCE_MEMORY to the minimum, it seems to reserve 4G of RAM on the server.
How can DB2 be aware of the limits setted in the docker-compose file as I run the database in a docker swarm cluster as a STACK?
The DB2 version is 11.1.4FP4.
docker --version
Docker version 18.03.1-ce, build 9ee9f40
When I look at the docker stats, it uses only about 80MiB.
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
8282c5d0c9e7 db2wse_db2.1.waalf6vljuapnxlvzhf2cb0uv 0.21% 76.83MiB / 1GiB 7.50% 0B / 0B 408MB / 6.86GB 56
My docker-compose.yml file
version: "3.3"
services:
db2:
image: docker-registry.ju.globaz.ch:5000/db2wse:11.1.4fp4
networks:
- network-db
deploy:
mode: replicated
replicas: 1
resources:
limits:
memory: 1G
networks:
network-db:
external: true
Any idea ? It is very frustrating.
Thanks a lot
I found the problem.
It was a kernel configuration in the sysctl.conf.
I had this :
cat /etc/sysctl.conf |grep vm.
vm.swappiness=10
vm.overcommit_memory=2
vm.dirty_ratio=2
vm.dirty_background_ratio=1
I removed everything setted for DB2 (put back the default configuration) and now I can take advantage of all the RAM of the hosts.
I kept this :
cat /etc/sysctl.conf |grep vm.
vm.swappiness=10
vm.max_map_count=262144
Thanks
I'm trying to limit my container so that it doesn't take up all the RAM on the host. From the Docker docs I understand that --memory limits the RAM and --memory-swap limits (RAM+swap). From the docker-compose docs it looks like the terms for those are mem_limit and memswap_limit, so I've constructed the following docker-compose file:
> cat docker-compose.yml
version: "2"
services:
stress:
image: progrium/stress
command: '-m 1 --vm-bytes 15G --vm-hang 0 --timeout 10s'
mem_limit: 1g
memswap_limit: 2g
The progrium/stress image just runs stress, which in this case spawns a single thread which requests 15GB RAM and holds on to it for 10 seconds.
I'd expect this to crash, since 15>2. (It does crash if I ask for more RAM than the host has.)
The kernel has cgroups enabled, and docker stats shows that the limit is being recognised:
> docker stats
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
7624a9605c70 0.00% 1024MiB / 1GiB 99.99% 396B / 0B 172kB / 0B 2
So what's going on? How do I actually limit the container?
Update:
Watching free, it looks like the RAM usage is effectively limited (only 1GB of RAM is used) but the swap is not: the container will gradually increase swap usage until it's eaten though all of the swap and stress crashes (it takes about 20secs to get through 5GB of swap on my machine).
Update 2:
Setting mem_swappiness: 0 causes an immediate crash when requesting more memory than mem_limit, regardless of memswap_limit.
Running docker info shows WARNING: No swap limit support
According to https://docs.docker.com/engine/installation/linux/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities this is disabled by default ("Memory and swap accounting incur an overhead of about 1% of the total available memory and a 10% overall performance degradation.") You can enable it by editing the /etc/default/grub file:
Add or edit the GRUB_CMDLINE_LINUX line to add the following two key-value pairs:
GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"
then update GRUB with update-grub and reboot.
mem_limit is supported by docker-compose? How can I test it?
I have a following docker-compose.yml
repository:
image: myregistry/my_nginx_image
mem_limit: 60m
volumes:
- /etc/localtime:/etc/localtime
ports:
- "80:80"
How can I prove that the container actually does not exceed 60 mb of RAM?
I am using:
docker 1.3.1
docker-compose 1.1.0
Yes. Memory limitation is supported by docker-compose and value can be set as in your example with "m" for megabytes.
It is possible to check what is the memory limit set for running Docker container using "docker stats" command.
If your container name is "repository_1" then use this command:
docker stats repository_1
The result of this will be simillar to this one:
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
repository_1 1.93% 4.609 MiB/60 MiB 7.20% 1.832 KiB/648 B
According to documentation, simple
mem_limit: 1000000000
should be enough. I guess, you should drop "m", and use bytes instead of megabytes.
If you use docker compose v3 instead of:
mem_limit: 60m
you need to specify memory limit in deploy section:
deploy:
resources:
limits:
memory: 60m
This change is described here: https://docs.docker.com/compose/compose-file/compose-versioning/#version-2x-to-3x
You can find how configure docker to limit resources (CPU & MEMORY) and how test your restrictions in this post written last year: resource-management-in-docker.