A Confluent Kafka instance is running via docker-compose on my Debian 9 server. I followed this tutorial to get it up and running. However, Control Center is shutting down periodically.
sudo docker-compose ps gives the following output:
control-center /etc/confluent/docker/run Exit 1
The rest of the Confluent services stay up and running.
When checking docker logs (sudo docker-compose logs) I can see that it is spamming the following error:
control-center | INFO [Consumer clientId=_confluent-controlcenter-5-3-0-1-9de26cca-62ca-42d6-9d46-86731fc8109a-StreamThread-5-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions (org.apache.kafka.clients.consumer.KafkaConsumer)
EDIT: discovered some more logs:
control-center | [2019-08-30 23:10:02,304] INFO [Consumer clientId=_confluent-controlcenter-5-3-0-1-39ae65e2-457c-4696-b592-504fe320038e-StreamThread-3-consumer, groupId=_confluent-controlcenter-5-3-0-1] Group coordinator broker:29092 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
control-center | [2019-08-30 22:38:39,291] INFO [Consumer clientId=_confluent-controlcenter-5-3-0-1-39ae65e2-457c-4696-b592-504fe320038e-StreamThread-8-consumer, groupId=_confluent-controlcenter-5-3-0-1] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
EDIT 2: memory available to docker containers:
NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
ksql-datagen 0.00% 3.312MiB / 7.792GiB 0.04% 18.2kB / 2.11kB 92.8MB / 65.5kB 1
control-center 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0
ksql-cli 0.00% 484KiB / 7.792GiB 0.01% 19.6kB / 0B 41kB / 0B 1
ksql-server 0.36% 136MiB / 7.792GiB 1.70% 39.8MB / 34.5MB 210MB / 147kB 30
rest-proxy 0.12% 107.2MiB / 7.792GiB 1.34% 22.2kB / 2.41kB 72.6MB / 81.9kB 28
connect 0.60% 1.571GiB / 7.792GiB 20.16% 124MB / 110MB 1.04GB / 81.9kB 36
schema-registry 0.20% 176.8MiB / 7.792GiB 2.22% 40.2MB / 38.4MB 93.7MB / 156kB 32
broker 7.59% 621MiB / 7.792GiB 7.78% 573MB / 791MB 171MB / 335MB 73
zookeeper 0.10% 80.9MiB / 7.792GiB 1.01% 9.56MB / 8.99MB 38.4MB / 410kB 21
System memory (command: free):
total used free shared buff/cache available
Mem: 8366596096 6770286592 160227328 219533312 1436082176 1099268096
Swap: 34356588544 2301014016 32055574528
Any ideas how to fix this?
This error comes up when the memory allocated is very less.
If you are using Docker Desktop then increase the memory . Go to Docker Desktop->Dashboard->Settings->Preferences->Resources->Advanced and increase the memory and you will be all set.
Related
I am running command docker stats <container_id> > performance.txt over a period of 1 hour during multi-user testing. Some stats like memory, CPU increase, then normalize. But, it is with NET I/O value that it kept on increasing.
At the start, the O/P was:
NAME CPU % MEM USAGE / LIMIT NET I/O BLOCK I/O PIDS
my-service 0.10% 5.63GiB / 503.6GiB 310MB / 190MB 0B / 0B 80
NAME CPU % MEM USAGE / LIMIT NET I/O BLOCK I/O PIDS
my-service 0.20% 5.63GiB / 503.6GiB 310MB / 190MB 0B / 0B 80
After 1 hour, it is:
NAME CPU % MEM USAGE / LIMIT NET I/O BLOCK I/O PIDS
my-service 116.26% 11.54GiB / 503.6GiB 891MB / 523MB 0B / 0B 89
NAME CPU % MEM USAGE / LIMIT NET I/O BLOCK I/O PIDS
my-service 8.52% 11.54GiB / 503.6GiB 892MB / 523MB 0B / 0B 89
As above, the value of NET I/O is always increasing. What can it probably mean?
Docker documentation says it is the input received and output given by the container. If so, then why is it increasing? Is there some issue with the image running in the container?
NET I/O is a cumulative counter. It only goes up (when your app receives and sends data).
https://docs.docker.com/engine/reference/commandline/stats/
Column name
Description
NET I/O
The amount of data the container has sent and received over its network interface
So it's accumulated over time. Unlike, say, CPU % which is how much CPU the container is using right now.
The docker stats command returns a live data stream for running containers.
It's considered the total amount of data passed over the network since the container started. From the definition of stream:
computing
a continuous flow of data or instructions,
It doesn't say that explicitly but you can obviously infer this by the term continuous or stream. Perhaps the documentation could be made a bit more clear in that respect.
This question already has answers here:
Check Resources Used by each Docker Container
(4 answers)
Closed 2 years ago.
How to see resource usage by docker containers?
I am using docker-compose version 2.1 and 2.4.
I would like to see running containers and their resources usage: RAM, CPU.
docker-compose ps returns containers, images, but not resources.
$ docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------------------------------------------------------
service-1 /app.sh Up (healthy) 8001/tcp
service-2 /app.sh Up (healthy) 0.0.0.0:80->4000/tcp
kafka /etc/confluent/docker/run Up 0.0.0.0:29092->29092/tcp, 0.0.0.0:7071->7071/tcp, 9092/tcp
mongo /init.sh Up (unhealthy) 0.0.0.0:27017->27017/tcp
zookeeper /etc/confluent/docker/run Up 2181/tcp, 2888/tcp, 0.0.0.0:32181->32181/tcp, 3888/tcp
You can see container resource usages with docker stats command.
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
95f5cdca9700 service-1 0.01% 35.39MiB / 381.5MiB 9.28% 141kB / 78kB 19.7MB / 0B 3
df49fdac6009 service-2 10.43% 278.6MiB / 381.5MiB 73.02% 126MB / 183MB 435MB / 0B 151
9dff501eb008 kafka 12.62% 348.1MiB / 572.2MiB 60.83% 610MB / 188MB 223MB / 0B 224
c00000a1d397 zookeeper 21.27% 313.7MiB / 381.5MiB 82.23% 600MB / 220MB 209MB / 0B 281
6d2d5000bb2e mongo 14.56% 325.8MiB / 381.5MiB 85.39% 531MB / 211MB 259MB / 0B 245
I have a few micro-services running in docker containers (One service in each container).
How do I find out the network usage between every two pair of docker containers? So that I can make a graph such that I have containers as vertices and on edges I have the amount of bytes transmitted/received.
I used cAdvisor, but it gives me the overall network usage of each container.
Start with docker stats:
$ docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
a645ca0d8feb wp_wordpress_1 0.00% 135.5MiB / 15.55GiB 0.85% 12.2MB / 5.28MB 2.81MB / 9.12MB 11
95e1649c5b79 wp_db_1 0.02% 203MiB / 15.55GiB 1.28% 6.44MB / 3.11MB 0B / 1.08GB 30
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
48c16e180af6 0.20% 91.48MiB / 31.31GiB 0.29% 3.86kB / 0B 85.3MB / 0B 33
f734efe5a249 0.00% 472KiB / 31.31GiB 0.00% 3.97kB / 0B 12.3kB / 0B 1
165a7b031093 0.00% 480KiB / 31.31GiB 0.00% 9.49kB / 0B 3.66MB / 0B 1
Does anyone know how to get resource consumption of a specific Docker container within its running environment?
Outside of a container, we can get it easily by typing a command "docker stats". Besides, if I try to get resource consumption inside a container, it will get the consumption (RAM, CPU) of the physical computer which the container runs on.
Another option is using 'htop' command, but it does not show the result exactly compared to 'docker stats' command.
If you want the processes consumption inside the container, you can go into the container and monitor the processes.
docker exec -it <container-name> watch ps -aux
Notice that after running the above command, the container doesn't know about any docker processes running.
Got a rails app currently running and stable on heroku server (512mb ram)
I took the app as is and put it on dokku (with intercity) on a ubuntu server 14gb ram 2 cpu(azure).
The app spins and works very fast, everything looks fine.
After 1 min of inactivity I refresh the browser and get a
504 Gateway Time-out
I try search for errors or any memory issues but the only thing looks wrong is the
17/01/18 11:24:18 [error] 61198#61198: *2071 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 79.184.17.155, server: cltvf.site, request: "GET /campaigns/5874e4d14bc3600a4a19566/details HTTP/1.1", upstream: "http://172.11.0.3:5000/campaigns/587f4e4d4bc3600a4a19566/details", host: "cltvf.site", referrer: "http://cltv.site/an/u_request_approve"
I got from the
nginx:error-logs
command
the 172.11.0.3 is an internal ip, if helps.
when trying to check if there is a memory issue I saw
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
ac513d4dd4ea 0.00% 199.8 MiB / 13.69 GiB 1.43% 296.7 kB / 156.5 kB 0 B / 0 B 13
a296ec88b1ef 0.01% 254.2 MiB / 13.69 GiB 1.81% 282.5 kB / 111.4 kB 0 B / 614.4 kB 52
beb69ddc4351 0.13% 254.3 MiB / 13.69 GiB 1.81% 286.9 kB / 112.5 kB 0 B / 614.4 kB 51
43665198a31b 0.00% 231.8 MiB / 13.69 GiB 1.65% 19.33 MB / 21.8 MB 0 B / 0 B 12
7d374f36b240 0.00% 231.6 MiB / 13.69 GiB 1.65% 19.34 MB / 21.81 MB 0 B / 0 B 13
04e98f7914b0 0.01% 343.9 MiB / 13.69 GiB 2.45% 14.37 MB / 9.091 MB 0 B / 614.4 kB 51
1255e7837b19 0.20% 231.5 MiB / 13.69 GiB 1.65% 19.34 MB / 21.78 MB 0 B / 0 B 12
378302bbdb84 0.00% 55.11 MiB / 13.69 GiB 0.39% 64.81 kB / 4.737 kB 0 B / 225.3 kB 40
5b8eb7a5423e 0.01% 52.47 MiB / 13.69 GiB 0.37% 71.75 kB / 8.718 kB 0 B / 225.3 kB 40
You can see nothing serious
same for disk usage
dev/sda1 28G 7403M 21G 25.5 [##########............................] /
/dev/sdb1 27G 44M 26G 0.2 [......................................] /mnt
/dev/sda1 28G 7403M 21G 25.5 [##########............................] /var/lib/docker/aufs
none 28G 7403M 21G 25.5 [##########............................] /var/lib/docker/aufs/mnt/4631d50385f25bf480fc18f5f2c7d93052b0f2ffecd6d04a14076513344b7338
none 28G 7403M 21G 25.5 [##########............................] /var/lib/docker/aufs/mnt/4f8488bdd0a683fda71a6789165d44626215ef4ce00f7d6c70c7ff64d7d89c14
none 28G 7403M 21G 25.5 [##########............................] /var/lib/docker/aufs/mnt/553fb1ea82841dd534450e9929513b90d17e4be73e271b861716d8f240ef8d17
none 28G 7403M 21G 25.5 [##########............................] /var/lib/docker/aufs/mnt/6909bba1bea70a3781f55bea3d059a014ddae8638021bf4f9a82edffab63cc94
none 28G 7403M 21G 25.5 [##########............................] /var/lib/docker/aufs/mnt/7200a36e8f3ca4e9358f83aad1ac5de562068f6458045f291812b8ab9e769abf
none 28G 7403M 21G 25.5 [##########............................] /var/lib/docker/aufs/mnt/bd289b0106072a2946e40a60bacb2b1024d1075996aff5bb3388290617ad85b2
none 28G 7403M 21G 25.5 [##########............................] /var/lib/docker/aufs/mnt/bd4d4632764af3a8e61b6da8d5f137addc2044615a5a36e72f675a180e6f7c7c
none 28G 7403M 21G 25.5 [##########............................] /var/lib/docker/aufs/mnt/e050fcacaeb0d9cb759bc72e768b2ceabd2eb95350f7c9ba6f20933c4696d1ef
none 28G 7403M 21G 25.5 [##########............................] /var/lib/docker/aufs/mnt/ffd758a6189aab5eac81950df15779f84f7c93a2a81b1707b082cee2202ece4d
I'm posting this question after hours of googling.
thanks
You could start by checking application logs: dokku logs <app>
You can try to connect directly to curl http://172.11.0.3:5000/
You can try to enter the container that is running the web process: dokku enter <app> web
You can use gdb, strace to connect to the process and use standard linux debugging tools
Most probably the request is taking to long to execute and it fails with 504 Gateway Timeout.
This usually happens when your server response time becomes longer than 60s (depending on the setting in nginx.conf, in dokku 60s is default).
The solutions is:
The easiest: increase the proxy_read_timeout in /home/dokku/:your_app_name/nginx.conf and reload nginx config.
Or find the root cause why your request takes too long to execute and make it respond faster (enable caching for example, or split time expensive tasks in separate workers processes and make web service just respond the status of jobs, allowing frontend to just poll the status of job until its finished)