I'm trying to setup Kafka locally and facing an issue. Whenever I run docker compose up, all containers are up correctly. After sometime, the broker container stops running for some reason. There is no error in the container logs.
Below is the status of all docker containers:
0c27a63bb0e7 confluentinc/ksqldb-examples:5.5.1 "bash -c 'echo Waiti…" 6 minutes ago Up 6 minutes ksql-datagen
4e4a30204ccc confluentinc/cp-ksqldb-cli:5.5.1 "/bin/sh" 6 minutes ago Up 6 minutes ksqldb-cli
61b86ff2a6d6 confluentinc/cp-ksqldb-server:5.5.1 "/etc/confluent/dock…" 6 minutes ago Up 6 minutes (health: starting) 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp ksqldb-server
2e022b64a760 cnfldemos/kafka-connect-datagen:0.3.2-5.5.0 "/etc/confluent/dock…" 6 minutes ago Exited (137) 5 minutes ago connect
3c7d273683fb confluentinc/cp-kafka-rest:5.5.1 "/etc/confluent/dock…" 6 minutes ago Exited (137) 5 minutes ago rest-proxy
6b6d36fb9d88 confluentinc/cp-schema-registry:5.5.1 "/etc/confluent/dock…" 6 minutes ago Up 6 minutes 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp schema-registry
3bb20335ecd1 confluentinc/cp-kafka:5.5.1 "/etc/confluent/dock…" 6 minutes ago Exited (137) 5 minutes ago broker
7b2f922ef8ef confluentinc/cp-zookeeper:5.5.1 "/etc/confluent/dock…" 6 minutes ago Up 6 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp zookeeper
All I want is a single node Kafka cluster with Schema Registry up and running locally. Any pointers are appreciated.
137 exit code is memory related.
If all you want is Kafka and the Schema Registry, remove KSQLDB stuff, REST Proxy, and DataGen containers
I'd also suggest using a later image tag, such as 7.3.1
Related
I am trying to set up a test hyperledger fabric environment. I am using docker approach.
Orderer and 2 peer nodes started successfully. However, channel creation fails with error. Any help would be greatly appreciated.
Error: Post "https://localhost:7053/participation/v1/channels": dial tcp [::1]:7053: connect: connection refused
===> List out hyperledger docker images
hyperledger/fabric-tools 2.4 545af418d284 3 months ago 489MB
hyperledger/fabric-tools 2.4.7 545af418d284 3 months ago 489MB
hyperledger/fabric-tools latest 545af418d284 3 months ago 489MB
hyperledger/fabric-peer 2.4 d77dd7cfbcd7 3 months ago 64.2MB
hyperledger/fabric-peer 2.4.7 d77dd7cfbcd7 3 months ago 64.2MB
hyperledger/fabric-peer latest d77dd7cfbcd7 3 months ago 64.2MB
hyperledger/fabric-orderer 2.4 77c489caa81b 3 months ago 36.7MB
hyperledger/fabric-orderer 2.4.7 77c489caa81b 3 months ago 36.7MB
hyperledger/fabric-orderer latest 77c489caa81b 3 months ago 36.7MB
hyperledger/fabric-ccenv 2.4 4eb7ee7f4af5 3 months ago 520MB
hyperledger/fabric-ccenv 2.4.7 4eb7ee7f4af5 3 months ago 520MB
hyperledger/fabric-ccenv latest 4eb7ee7f4af5 3 months ago 520MB
hyperledger/fabric-baseos 2.4 b20d2dde6941 3 months ago 6.82MB
hyperledger/fabric-baseos 2.4.7 b20d2dde6941 3 months ago 6.82MB
hyperledger/fabric-baseos latest b20d2dde6941 3 months ago 6.82MB
hyperledger/fabric-ca 1.5 93f19fa873cb 7 months ago 76.5MB
hyperledger/fabric-ca 1.5.5 93f19fa873cb 7 months ago 76.5MB
hyperledger/fabric-ca latest 93f19fa873cb 7 months ago 76.5MB
sudo ./network.sh up
above is conducted successful
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0eec45f8ecd8 hyperledger/fabric-tools:latest "/bin/bash" 3 seconds ago Up Less than a second cli
d511a671b457 hyperledger/fabric-peer:latest "peer node start" 3 seconds ago Exited (1) 1 second ago peer0.org1.example.com
dfd6cf1917ee hyperledger/fabric-peer:latest "peer node start" 3 seconds ago Exited (1) 1 second ago peer0.org2.example.com
05d550a97a73 hyperledger/fabric-orderer:latest "orderer" 3 seconds ago Exited (2) Less than a second ago orderer.example.com
but when comes to the channel creating, error appears
system:mac
Have retried many times, but it is still not working.
thanks!
solutions for this question i searched in stackoverflow, that retry cannot help, include:
reload docker
re-clone
etc.
Im trying to setup airflow on my machine using docker and the docker-compose file provided by airflow here : https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#docker-compose-yaml
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d4d8de8f7782 apache/airflow:2.2.0 "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 8080/tcp airflow_airflow-scheduler_1
3315f125949c apache/airflow:2.2.0 "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 8080/tcp airflow_airflow-worker_1
2426795cb59f apache/airflow:2.2.0 "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp airflow_airflow-webserver_1
cf649cd645bb apache/airflow:2.2.0 "/usr/bin/dumb-init …" About a minute ago Up About a minute (unhealthy) 8080/tcp airflow_airflow-triggerer_1
fa6b181113ae apache/airflow:2.2.0 "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp airflow_flower_1
b6e05f63aa2c postgres:13 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes (healthy) 5432/tcp airflow_postgres_1
177475be25a3 redis:latest "docker-entrypoint.s…" 2 minutes ago Up 2 minutes (healthy) 6379/tcp airflow_redis_1
I followed all steps as described in this URL, every airflow component is working great but the airflow trigerrer shows an unhealthy status :/
Im kinda new to docker i just know the basics and i don't really know how to debug that, can anyone help me up ?
Try to follow all steps on their website including mkdir ./dags ./logs ./plugins echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env.
I don't know but it works then, but still unhealthy,
airflow.apache.org/docs/apache-airflow/stable/start/docker.html
I am following this example:
I find the namenode as follows:
(dash_apps_36) idf#ubvm:~/Documents/docker-hadoop$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2ce71013488f bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 8088/tcp resourcemanager
7fc72b887e23 bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 0.0.0.0:9000->9000/tcp, 0.0.0.0:9870->9870/tcp namenode
9c3f8c435b87 bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 8042/tcp nodemanager
4d57c5bf2f9e bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 9864/tcp datanode
c7eef33905e8 bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 8188/tcp historyserver
2b1144867924 redis "docker-entrypoint.s\u2026" 6 days ago Exited (137) 6 days ago serene_keldysh
9d9463166b63 redis "docker-entrypoint.s\u2026" 6 days ago Created eloquent_merkle
47d6488e29bf redis:alpine "docker-entrypoint.s\u2026" 4 weeks ago Exited (137) 4 weeks ago wikipedia-playground_redis_1
be5ea0df0401 mcr.microsoft.com/mssql/server:2019-latest "/opt/mssql/bin/perm\u2026" 18 months ago Exited (255) 18 months ago 0.0.0.0:1433->1433/tcp mssql
Using 7fc72b887e23 as the ID of the namenode, when I get to the command
$ docker cp ../hadoop-mapreduce-examples-2.7.1-sources.jar cb0c13085cd3:hadoop-mapreduce-examples-2.7.1-sources.jar
I get an error:
$: command not found
Not sure what I am doing wrong.
Remove the $ at the beginning. That's what $: command not found means. Easy to miss when copy pasting code
docker cp ../hadoop-mapreduce-examples-2.7.1-sources.jar cb0c13085cd3:hadoop-mapreduce-examples-2.7.1-sources.jar
I setup my kafka containers by follow the tutorial from here: https://success.docker.com/article/getting-started-with-kafka
Then I found that I can't remove the container anymore, even though I tried docker container prune. The containers are still running.
What should I do?
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
caa0b94c1d98 qnib/plain-kafka:1.1.0 "/usr/local/bin/entr…" 3 minutes ago Up 3 minutes (healthy) broker.2.3fij6pt90qt9sb9aco0i2dpys
b888cb6f783a qnib/plain-kafka:1.1.0 "/usr/local/bin/entr…" 3 minutes ago Up 3 minutes (healthy) broker.3.xqmjnfnfg7ha46lf6drlrq4ki
dcdda2d778c2 qnib/plain-kafka:1.1.0 "/usr/local/bin/entr…" 3 minutes ago Up 3 minutes (healthy) broker.1.gtgluxt6q58z2irzgfmu969ba
843def0b24fb qnib/plain-zkui "/usr/local/bin/entr…" 3 minutes ago Up 3 minutes (healthy) zkui.1.7zks618eae8sp4woc7araydix
d7ced19be88c qnib/plain-kafka-manager:2018-04-25 "/usr/local/bin/entr…" 3 minutes ago Up 3 minutes (healthy) manager.1.jdu5gnprhr4d982vz50511rhg
a67ac962e682 qnib/plain-zookeeper:2018-04-25 "/usr/local/bin/entr…" 3 minutes ago Up 3 minutes (healthy) 2181/tcp, 2888/tcp, 3888/tcp zookeeper.1.xar7cmdgozdj79orow0bmj3ev
880121f2fee5 qnib/golang-kafka-producer:2018-05-01.5 "kafka-producer" 3 minutes ago Up 3 minutes (healthy) producer.2.hety8za590v1twdgj2byvrmse
b6487d29812e qnib/golang-kafka-producer:2018-05-01.5 "kafka-producer" 3 minutes ago Up 3 minutes (healthy) producer.1.5oz02c8cw5oefc97xbarq5qoa
8b3a81905e90 qnib/golang-kafka-producer:2018-05-01.5 "kafka-producer" 3 minutes ago Up 3 minutes (healthy) producer.3.p8uh3hzr22fgm7u4gl1p3fiyw
I found I have to use docker service rm to remove the service due to the replicas settings.
The prune command doesn't work against running containers. You will need to either stop or kill them.
docker kill caa0b94c1d98
I follow the official tutorial of deploying docker services https://docs.docker.com/get-started/part5/#add-a-new-service-and-redeploy, in the first time i tried this the containers was running as expected but after that the containers shutdown and restarting (i notice that using the visualizer service provided by docker )
and when i execute the command :
docker stack ps getstartedlab
NAME DESIRED STATE CURRENT STATE
ERROR
getstartedlab_web.1 Running Preparing 2 seconds ago
\_ getstartedlab_web.1 Shutdown Failed 4 minutes ago
"task: non-zero exit (2)"
i read in this post https://github.com/docker/machine/issues/3747 that the problem came from the firewall that may block the icmp, i tried to ping docker.com and i had 100% loss packet but when i ping google.com it's ok with no loss packets.
the result of docker ps --all is :
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9f67c82da7c7 username/repo:tag "python app.py" 2 minutes ago Exited (2) 2 minutes ago getstartedlab_web.2.zcnr0ld9bioy0dffsxmn8zss5
f59e413b0780 username/repo:tag "python app.py" 4 minutes ago Exited (2) 4 minutes ago getstartedlab_web.5.ymxgnsf7n8306yr8963xyyljv
9ab631c4057c username/repo:tag "python app.py" 10 minutes ago Exited (2) 10 minutes ago getstartedlab_web.5.zr3gsvgbyxs8c51fiko5h9jxp
bee5816ce1f2 dockersamples/visualizer:stable "npm start" 15 minutes ago Up 15 minutes 8080/tcp getstartedlab_visualizer.1.oyiqwb5esq6zakcdtiw4txh8a
cadca594f8cd username/repo:tag "python app.py" 24 hours ago Exited (2) 24 hours ago getstartedlab_web.1.zehutsl9cefrccqrj86dz4ap7
576b1a6db0b0 username/repo:tag "python app.py" 24 hours ago Exited (2) 24 hours ago getstartedlab_web.5.za9xvxpo5yvl20kha9sjcimmz
2804ebc4fc0c username/repo:tag "python app.py" 24 hours ago Exited (2) 24 hours ago getstartedlab_web.1.zlk42chspvym3jxkzs2nc8k2d
03efb2b04489 dockersamples/visualizer:stable "npm start" 24 hours ago Exited (255) 16 minutes ago 8080/tcp getstartedlab_visualizer.1.qyp4egtu9vcd31kf2jxtzxko3
b85fd1600955 username/repo:tag "python app.py" 2 days ago Exited (2) 2 days ago getstartedlab_web.5.kzrj3m5c3jgkuox0ulpszizee
and the
docker logs 9f67c82da7c7
python: can't open file 'app.py': [Errno 2] No such file or directory
exuse me for representation of the results of those commands because when i copy past the results the lines breaks, how to copy past and preserve the same displaying ?
can anyone have a fix to this problem ? thanks.