I am following this example:
I find the namenode as follows:
(dash_apps_36) idf#ubvm:~/Documents/docker-hadoop$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2ce71013488f bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 8088/tcp resourcemanager
7fc72b887e23 bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 0.0.0.0:9000->9000/tcp, 0.0.0.0:9870->9870/tcp namenode
9c3f8c435b87 bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 8042/tcp nodemanager
4d57c5bf2f9e bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 9864/tcp datanode
c7eef33905e8 bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 8188/tcp historyserver
2b1144867924 redis "docker-entrypoint.s\u2026" 6 days ago Exited (137) 6 days ago serene_keldysh
9d9463166b63 redis "docker-entrypoint.s\u2026" 6 days ago Created eloquent_merkle
47d6488e29bf redis:alpine "docker-entrypoint.s\u2026" 4 weeks ago Exited (137) 4 weeks ago wikipedia-playground_redis_1
be5ea0df0401 mcr.microsoft.com/mssql/server:2019-latest "/opt/mssql/bin/perm\u2026" 18 months ago Exited (255) 18 months ago 0.0.0.0:1433->1433/tcp mssql
Using 7fc72b887e23 as the ID of the namenode, when I get to the command
$ docker cp ../hadoop-mapreduce-examples-2.7.1-sources.jar cb0c13085cd3:hadoop-mapreduce-examples-2.7.1-sources.jar
I get an error:
$: command not found
Not sure what I am doing wrong.
Remove the $ at the beginning. That's what $: command not found means. Easy to miss when copy pasting code
docker cp ../hadoop-mapreduce-examples-2.7.1-sources.jar cb0c13085cd3:hadoop-mapreduce-examples-2.7.1-sources.jar
Related
I'm trying to setup Kafka locally and facing an issue. Whenever I run docker compose up, all containers are up correctly. After sometime, the broker container stops running for some reason. There is no error in the container logs.
Below is the status of all docker containers:
0c27a63bb0e7 confluentinc/ksqldb-examples:5.5.1 "bash -c 'echo Waiti…" 6 minutes ago Up 6 minutes ksql-datagen
4e4a30204ccc confluentinc/cp-ksqldb-cli:5.5.1 "/bin/sh" 6 minutes ago Up 6 minutes ksqldb-cli
61b86ff2a6d6 confluentinc/cp-ksqldb-server:5.5.1 "/etc/confluent/dock…" 6 minutes ago Up 6 minutes (health: starting) 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp ksqldb-server
2e022b64a760 cnfldemos/kafka-connect-datagen:0.3.2-5.5.0 "/etc/confluent/dock…" 6 minutes ago Exited (137) 5 minutes ago connect
3c7d273683fb confluentinc/cp-kafka-rest:5.5.1 "/etc/confluent/dock…" 6 minutes ago Exited (137) 5 minutes ago rest-proxy
6b6d36fb9d88 confluentinc/cp-schema-registry:5.5.1 "/etc/confluent/dock…" 6 minutes ago Up 6 minutes 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp schema-registry
3bb20335ecd1 confluentinc/cp-kafka:5.5.1 "/etc/confluent/dock…" 6 minutes ago Exited (137) 5 minutes ago broker
7b2f922ef8ef confluentinc/cp-zookeeper:5.5.1 "/etc/confluent/dock…" 6 minutes ago Up 6 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp zookeeper
All I want is a single node Kafka cluster with Schema Registry up and running locally. Any pointers are appreciated.
137 exit code is memory related.
If all you want is Kafka and the Schema Registry, remove KSQLDB stuff, REST Proxy, and DataGen containers
I'd also suggest using a later image tag, such as 7.3.1
The airflow cluster accesses the worker log error, and accesses the worker in the master node will not report an error.I can't solve my proplem.I used the docker develoment my airflow and celery.can you help me?Thanks!!!
*** Log file does not exist: /opt/airflow/logs/dag_id=example_bash_operator/run_id=scheduled__2022-09-23T00:00:00+00:00/task_id=runme_1/attempt=1.log
*** Fetching from: http://eosbak01.zzz.ac.cn:8793/log/dag_id=example_bash_operator/run_id=scheduled__2022-09-23T00:00:00+00:00/task_id=runme_1/attempt=1.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url 'http://eosbak01.zzz.ac.cn:8793/log/dag_id=example_bash_operator/run_id=scheduled__2022-09-23T00:00:00+00:00/task_id=runme_1/attempt=1.log'
For more information check: https://httpstatuses.com/403
WorkerNode
[airflow#eosbak01 deploy]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2ef39a54de97 apache/airflow:2.3.4 "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 8080/tcp deploy_airflow-worker_
MainNode
[airflow#eosbak02 deploy]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1e6a7e50831d apache/airflow:2.3.4 "/usr/bin/dumb-init …" 26 minutes ago Up 26 minutes (healthy) 0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp deploy_flower_1
9afb5985b9f3 apache/airflow:2.3.4 "/usr/bin/dumb-init …" 27 minutes ago Up 27 minutes (healthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp deploy_airflow-webserver_1
80132177ae3d apache/airflow:2.3.4 "/usr/bin/dumb-init …" 27 minutes ago Up 27 minutes (healthy) 8080/tcp deploy_airflow-triggerer_1
6ea5a0ed7dec apache/airflow:2.3.4 "/usr/bin/dumb-init …" 27 minutes ago Up 27 minutes (healthy) 8080/tcp deploy_airflow-scheduler_1
2787acb189ad mysql:8.0.27 "docker-entrypoint.s…" 29 minutes ago Up 29 minutes (healthy) 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp deploy_mysql_1
057af26f6070 redis:latest "docker-entrypoint.s…" 29 minutes ago Up 29 minutes (healthy) 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp deploy_redis_1
I had created a lot of containers. I am new to this docker container system.
I do a docker ps -a then I get following result
debian#osboxes:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a6a62181dcf0 python "/bin/bash" About a minute ago Exited (127) 8 seconds ago try1
ed5ef0b8155d 1d1a162a72a6 "/bin/bash" 22 hours ago Exited (0) 22 hours ago pedantic_ellis
19a4eff3b5e5 1d1a162a72a6 "/bin/bash" 22 hours ago Exited (0) 22 hours ago assignment4
30933891f08c eeadc22d21a9 "python3" 22 hours ago Exited (0) 22 hours ago python
ccdab94fd32f 1d1a162a72a6 "/bin/bash" 28 hours ago Exited (0) 24 hours ago confident_wu
ce462ecfc5f2 1d1a162a72a6 "/bin/bash -v /home/…" 28 hours ago Exited (127) 28 hours ago stupefied_grothendieck
6123f134934c 1d1a162a72a6 "/bin/bash" 28 hours ago Exited (1) 28 hours ago stupefied_taussig
0ed23a8112a4 1d1a162a72a6 "/bin/bash" 29 hours ago Exited (0) 29 hours ago vigilant_bartik
c343731b7cde 1d1a162a72a6 "/bin/bash" 30 hours ago Exited (0) 29 hours ago gallant_ardinghelli
2f95d3b4c1b8 1d1a162a72a6 "/bin/bash" 30 hours ago Created nice_hermann
5ebe9f18c744 1d1a162a72a6 "/bin/bash" 46 hours ago Created pensive_easley
c1b43edfafb9 1d1a162a72a6 "/bin/bash" 46 hours ago Exited (1) 22 hours ago adoring_williams
42dea69d1d4e 1d1a162a72a6 "/bin/bash" 46 hours ago Created funny_austin
6f736902e650 1d1a162a72a6 "/bin/bash" 46 hours ago Exited (1) 46 hours ago strange_ride
09306e5ec5d1 1d1a162a72a6 "--name=kaushik" 2 days ago Created pensive_shtern
699fb2a23e1c 1d1a162a72a6 "--name=kaushik" 2 days ago Created sharp_feistel
9f7b29ab512e 1d1a162a72a6 "--name=kaushik" 2 days ago Created elastic_payne2
25bfc74fab3b 1d1a162a72a6 "/bin/bash" 2 days ago Exited (1) 2 days ago festive_einstein
e658dd320297 1d1a162a72a6 "/bin/bash" 2 days ago Exited (255) 2 minutes ago objective_napier
ebae378d9152 1d1a162a72a6 "/bin/bash" 2 days ago Exited (1) 2 days ago brave_ritchie
23c7f4293b30 hadoop-build-1001 "/bin/bash" 2 days ago Exited (0) 2 days ago suspicious_lumiere
5090081f6809 hadoop-build-1001 "/bin/bash" 2 days ago Exited (1) 2 days ago quizzical_keller
425d59be9cbf hadoop-build-1001 "/bin/bash" 2 days ago Exited (0) 2 days ago distracted_lederberg
11c55ce7f011 hadoop-build-1001 "/bin/bash" 2 days ago Created elastic_noyce
1ccaf0477995 hadoop-build-1001 "/bin/bash" 2 days ago Created mystifying_tu
62528115f4b7 hadoop-build-1001 "/bin/bash" 2 days ago Created determined_meninsky
fca64af2f595 hadoop-build-1001 "/bin/bash" 2 days ago Created elastic_goldwasser
eecb3153bded hadoop-build-1001 "/bin/bash" 2 days ago Created cool_cray
30b6d61fcac9 hadoop-build-1001 "/bin/bash" 2 days ago Exited (1) 2 days ago quirky_kapitsa
cf992a8b8286 hadoop-build-1001 "/bin/bash" 2 days ago Created hungry_goldstine
0f9a951f7593 hadoop-build-1001 "/bin/bash" 2 days ago Created crazy_wright
e25dcf8a8be8 hadoop-build-1001 "/bin/bash" 2 days ago Created bold_pasteur
73d068e0d756 hadoop-build-1001 "/bin/bash" 2 days ago Exited (0) 2 days ago condescending_goodall
adda325294cd hadoop-build-1001 "/bin/bash" 2 days ago Exited (0) 2 days ago serene_wilson
75a9a3262505 hadoop-build-1001 "/bin/bash" 2 days ago Exited (0) 2 days ago hardcore_khorana
e38726a74e9b hadoop-build-1001 "/bin/bash" 3 days ago Exited (255) 2 days ago beautiful_clarke
4060dbc85d2d hadoop-build-1001 "/bin/bash" 3 days ago Exited (0) 3 days ago strange_yonath
174509213b30 hadoop-build-1001 "/bin/bash" 8 days ago Exited (255) 7 days ago hadoop-c
fa82c595e214 1d1a162a72a6 "/bin/bash" 8 days ago Exited (0) 8 days ago agitated_edison
4b07fcc45271 python "python3" 2 weeks ago Exited (255) 8 days ago pyC
1cc06f213eb7 abee520343a4 "/bin/sh -c 'apt-get…" 3 months ago Exited (100) 3 months ago compassionate_kowalevski
8c1eb67f7325 1b1c7b120b48 "/bin/sh -c 'cd /opt…" 3 months ago Exited (255) 3 months ago dreamy_jepsen
1406d7476a28 1180f37ef8b1 "/bin/sh -c 'mkdir -…" 3 months ago Exited (35) 3 months ago upbeat_taussig
e88bcf7743e2 f0f5acc11f91 "/bin/sh -c 'apt-get…" 3 months ago Exited (100) 3 months ago tender_cohen
I had made some changes in a container which was created using one of hadoop image as below
debian#osboxes:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
python latest eeadc22d21a9 2 weeks ago 933MB
hadoop-build-1001 latest 1d1a162a72a6 3 months ago 2.02GB
hadoop-build latest 5c1480006f78 3 months ago 1.96GB
ubuntu xenial 5f2bf26e3524 3 months ago 123MB
I powered off the machine and rebooted etc.
I want to know that I made changes to one of the containers say in above output of docker ps -a the container was
25bfc74fab3b 1d1a162a72a6 "/bin/bash" 2 days ago Exited (1)
how can after a reboot etc I connect or start the same container again.
What I am currently doing is
docker run -it 1d1a162a72a6 /bin/bash
I make some changes and run a few python programms. But after a reboot all the changes are gone that means those python files which I had edited I have to copy paste them again in the container and do every thing from scratch.I have to install vim each time and then each time I do apt-get update install softwares and then check edit the programe.
I am not able to understand how to go back to same container which was created moments ago before exiting. What mistake I am doing here?
docker run always starts a new container. You can re-attach to the existing container provided it wasn't started with --rm using docker start:
docker start -ai <CONTAINER_ID>
Notice that you should use container ID, not image ID as a parameter to docker start. See more in Docker help:
$ docker start --help
Usage: docker start [OPTIONS] CONTAINER [CONTAINER...]
Start one or more stopped containers
Options:
-a, --attach Attach STDOUT/STDERR and forward signals
--detach-keys string Override the key sequence for detaching a container
-i, --interactive Attach container's STDIN
I had made some changes in a container... after a reboot all the changes are gone that means those python files which I had edited I have to copy paste them again in the container and do every thing from scratch.I have to install vim each time and then each time I do apt-get update install softwares and then check edit the programe.
Containers are, by definition, immutable. What you were editing was a temporary runtime layer. If the container stopped or exited, you lose those changes.
In order to preserve changes, you must edit the Dockerfile and rebuild the container in order to create those immutable layers that the image contains
Realistically, you'd use a volume mount from the host into the container and edit code locally, which syncs into the container. No need for text editors or repeated apt updates
I follow the official tutorial of deploying docker services https://docs.docker.com/get-started/part5/#add-a-new-service-and-redeploy, in the first time i tried this the containers was running as expected but after that the containers shutdown and restarting (i notice that using the visualizer service provided by docker )
and when i execute the command :
docker stack ps getstartedlab
NAME DESIRED STATE CURRENT STATE
ERROR
getstartedlab_web.1 Running Preparing 2 seconds ago
\_ getstartedlab_web.1 Shutdown Failed 4 minutes ago
"task: non-zero exit (2)"
i read in this post https://github.com/docker/machine/issues/3747 that the problem came from the firewall that may block the icmp, i tried to ping docker.com and i had 100% loss packet but when i ping google.com it's ok with no loss packets.
the result of docker ps --all is :
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9f67c82da7c7 username/repo:tag "python app.py" 2 minutes ago Exited (2) 2 minutes ago getstartedlab_web.2.zcnr0ld9bioy0dffsxmn8zss5
f59e413b0780 username/repo:tag "python app.py" 4 minutes ago Exited (2) 4 minutes ago getstartedlab_web.5.ymxgnsf7n8306yr8963xyyljv
9ab631c4057c username/repo:tag "python app.py" 10 minutes ago Exited (2) 10 minutes ago getstartedlab_web.5.zr3gsvgbyxs8c51fiko5h9jxp
bee5816ce1f2 dockersamples/visualizer:stable "npm start" 15 minutes ago Up 15 minutes 8080/tcp getstartedlab_visualizer.1.oyiqwb5esq6zakcdtiw4txh8a
cadca594f8cd username/repo:tag "python app.py" 24 hours ago Exited (2) 24 hours ago getstartedlab_web.1.zehutsl9cefrccqrj86dz4ap7
576b1a6db0b0 username/repo:tag "python app.py" 24 hours ago Exited (2) 24 hours ago getstartedlab_web.5.za9xvxpo5yvl20kha9sjcimmz
2804ebc4fc0c username/repo:tag "python app.py" 24 hours ago Exited (2) 24 hours ago getstartedlab_web.1.zlk42chspvym3jxkzs2nc8k2d
03efb2b04489 dockersamples/visualizer:stable "npm start" 24 hours ago Exited (255) 16 minutes ago 8080/tcp getstartedlab_visualizer.1.qyp4egtu9vcd31kf2jxtzxko3
b85fd1600955 username/repo:tag "python app.py" 2 days ago Exited (2) 2 days ago getstartedlab_web.5.kzrj3m5c3jgkuox0ulpszizee
and the
docker logs 9f67c82da7c7
python: can't open file 'app.py': [Errno 2] No such file or directory
exuse me for representation of the results of those commands because when i copy past the results the lines breaks, how to copy past and preserve the same displaying ?
can anyone have a fix to this problem ? thanks.
I was surprised when command sudo docker ps -a thrown list of containers:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
efd4879e92e0 ubuntu "/bin/bash" About an hour ago Exited (0) 20 minutes ago elated_newton
7db432be894c ubuntu "/bin/bash" About an hour ago Exited (0) 18 minutes ago nervous_meninsky
aef01293fdc9 ubuntu "/bin/bash" About an hour ago Up About an hour priceless_ramanujan
87cfd461b465 ubuntu "/bin/bash" About an hour ago Up About an hour festive_bhabha
1dcb6f9618d8 ubuntu "/bin/bash" About an hour ago Up About an hour mystifying_poitras
0f1636eb9cc1 hello-world "/hello" About an hour ago Exited (0) About an hour ago sharp_murdock
2a3a1fb03ada hello-world "/hello" About an hour ago Exited (0) About an hour ago admiring_brattain
77c8e7ef5770 hello-world "/hello" 16 hours ago Exited (0) 16 hours ago unruffled_shannon
808805085c59 centos "6.9" 17 hours ago Created youthful_aryabhata
a1438a7811c2 centos "/bin/bash" 17 hours ago Exited (0) 17 hours ago sleepy_neumann
fce6a95f8f5a ubuntu "/bin/bash" 17 hours ago Exited (0) 17 hours ago cranky_ritchie
e17b5f98bbc5 ubuntu "/bin/bash" 17 hours ago Exited (0) 17 hours ago unruffled_ritchie
cafbbe52dad5 hello-world "/hello" 17 hours ago Exited (0) 17 hours ago hungry_pike
d40ac9c16e9c ubuntu "/bin/bash" 17 hours ago Exited (0) 17 hours ago brave_babbage
5efd22250d0e ubuntu "/bin/bash" 17 hours ago Exited (0) 17 hours ago confident_lalande
471e22652bc3 ubuntu "/bin/bash" 17 hours ago Exited (0) 17 hours ago vigorous_wing
b68f43ebae50 ubuntu "bash" 17 hours ago Exited (127) 17 hours ago confident_euler
a83eae486f76 ubuntu "bash" 17 hours ago Exited (0) 17 hours ago infallible_murdock
88b3b4e73e5e hello-world "/hello" 17 hours ago Exited (0) 17 hours ago thirsty_bhabha
deb88e38d347 hello-world "/hello" 17 hours ago Exited (0) 17 hours ago nervous_ptolemy
I was surprised twice when found that some of them alive. Since festive_bhabha is up I would like attach to it:
sudo docker attach festive_bhabha
But command prints nothing and not returns. Why I can't attach to container?
Docker containers are not automatically removed when they exist. You can clean dangling images, exited containers... using:
docker system prune
The Docker container festive_bhabha seems to be an ubuntu image that might not be generating and output, which explains why no output is shown when running the attach command.
You can check what processes are running inside this container by running:
docker container top festive_bhabha
Alternatively, you can exec inside the container.
docker exec -it festive_bhabha bash