The airflow cluster accesses the worker log error, and accesses the worker in the master node will not report an error.I can't solve my proplem.I used the docker develoment my airflow and celery.can you help me?Thanks!!!
*** Log file does not exist: /opt/airflow/logs/dag_id=example_bash_operator/run_id=scheduled__2022-09-23T00:00:00+00:00/task_id=runme_1/attempt=1.log
*** Fetching from: http://eosbak01.zzz.ac.cn:8793/log/dag_id=example_bash_operator/run_id=scheduled__2022-09-23T00:00:00+00:00/task_id=runme_1/attempt=1.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url 'http://eosbak01.zzz.ac.cn:8793/log/dag_id=example_bash_operator/run_id=scheduled__2022-09-23T00:00:00+00:00/task_id=runme_1/attempt=1.log'
For more information check: https://httpstatuses.com/403
WorkerNode
[airflow#eosbak01 deploy]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2ef39a54de97 apache/airflow:2.3.4 "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 8080/tcp deploy_airflow-worker_
MainNode
[airflow#eosbak02 deploy]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1e6a7e50831d apache/airflow:2.3.4 "/usr/bin/dumb-init …" 26 minutes ago Up 26 minutes (healthy) 0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp deploy_flower_1
9afb5985b9f3 apache/airflow:2.3.4 "/usr/bin/dumb-init …" 27 minutes ago Up 27 minutes (healthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp deploy_airflow-webserver_1
80132177ae3d apache/airflow:2.3.4 "/usr/bin/dumb-init …" 27 minutes ago Up 27 minutes (healthy) 8080/tcp deploy_airflow-triggerer_1
6ea5a0ed7dec apache/airflow:2.3.4 "/usr/bin/dumb-init …" 27 minutes ago Up 27 minutes (healthy) 8080/tcp deploy_airflow-scheduler_1
2787acb189ad mysql:8.0.27 "docker-entrypoint.s…" 29 minutes ago Up 29 minutes (healthy) 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp deploy_mysql_1
057af26f6070 redis:latest "docker-entrypoint.s…" 29 minutes ago Up 29 minutes (healthy) 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp deploy_redis_1
Related
I'm trying to setup Kafka locally and facing an issue. Whenever I run docker compose up, all containers are up correctly. After sometime, the broker container stops running for some reason. There is no error in the container logs.
Below is the status of all docker containers:
0c27a63bb0e7 confluentinc/ksqldb-examples:5.5.1 "bash -c 'echo Waiti…" 6 minutes ago Up 6 minutes ksql-datagen
4e4a30204ccc confluentinc/cp-ksqldb-cli:5.5.1 "/bin/sh" 6 minutes ago Up 6 minutes ksqldb-cli
61b86ff2a6d6 confluentinc/cp-ksqldb-server:5.5.1 "/etc/confluent/dock…" 6 minutes ago Up 6 minutes (health: starting) 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp ksqldb-server
2e022b64a760 cnfldemos/kafka-connect-datagen:0.3.2-5.5.0 "/etc/confluent/dock…" 6 minutes ago Exited (137) 5 minutes ago connect
3c7d273683fb confluentinc/cp-kafka-rest:5.5.1 "/etc/confluent/dock…" 6 minutes ago Exited (137) 5 minutes ago rest-proxy
6b6d36fb9d88 confluentinc/cp-schema-registry:5.5.1 "/etc/confluent/dock…" 6 minutes ago Up 6 minutes 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp schema-registry
3bb20335ecd1 confluentinc/cp-kafka:5.5.1 "/etc/confluent/dock…" 6 minutes ago Exited (137) 5 minutes ago broker
7b2f922ef8ef confluentinc/cp-zookeeper:5.5.1 "/etc/confluent/dock…" 6 minutes ago Up 6 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp zookeeper
All I want is a single node Kafka cluster with Schema Registry up and running locally. Any pointers are appreciated.
137 exit code is memory related.
If all you want is Kafka and the Schema Registry, remove KSQLDB stuff, REST Proxy, and DataGen containers
I'd also suggest using a later image tag, such as 7.3.1
So I try running some docker application on My local network it works fine without any problem.
But when I try access it from public IP address it doesn't allow Me to connect
Ex: http://SomePublicIp then browser return with
SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u1
but when I try with local IP it works normally
the return of docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
yacht
352d4a07c5aa ghcr.io/ajnart/homarr:latest "docker-entrypoint.s…" 13 hours ago Up 13 hours 0.0.0.0:7575->7575/tcp, :::7575->7575/tcp
Homepage
8ba96af5b79b linuxserver/sickgear:latest "/init" 13 hours ago Up 13 hours 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp
sickgear
cb50149d8921 n8nio/n8n:latest "tini -- /docker-ent…" 13 hours ago Up 13 hours 0.0.0.0:5678->5678/tcp, :::5678->5678/tcp
n8n
86d99e6877cc portainer/portainer-ce:latest "/portainer" 14 hours ago Up 13 hours 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp, 9000/tcp
portainer
4bc1c1c244a1 linuxserver/homeassistant:latest "/init" 2 days ago Up 14 hours
HomeAssistant
b712b08105c7 grafana/grafana:8.4.5 "/run.sh" 4 days ago Up 14 hours 0.0.0.0:41117->3000/tcp, :::41117->3000/tcp
Smokeping
8b98f0161300 linuxserver/syncthing:latest "/init" 4 days ago Up 14 hours 0.0.0.0:8384->8384/tcp, 0.0.0.0:21027->21027/udp, :::8384->8384/tcp, :::21027->21027/udp, 0.0.0.0:22000->22000/tcp, :::22000->22000/tcp, 0.0.0.0:22000->22000/udp, :::22000->22000/udp
Syncthing
a0622f05ac9f nodered/node-red:2.2.2-12 "npm --no-update-not…" 10 days ago Up 14 hours (healthy) 0.0.0.0:1880->1880/tcp, :::1880->1880/tcp
Im trying to setup airflow on my machine using docker and the docker-compose file provided by airflow here : https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#docker-compose-yaml
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d4d8de8f7782 apache/airflow:2.2.0 "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 8080/tcp airflow_airflow-scheduler_1
3315f125949c apache/airflow:2.2.0 "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 8080/tcp airflow_airflow-worker_1
2426795cb59f apache/airflow:2.2.0 "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp airflow_airflow-webserver_1
cf649cd645bb apache/airflow:2.2.0 "/usr/bin/dumb-init …" About a minute ago Up About a minute (unhealthy) 8080/tcp airflow_airflow-triggerer_1
fa6b181113ae apache/airflow:2.2.0 "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp airflow_flower_1
b6e05f63aa2c postgres:13 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes (healthy) 5432/tcp airflow_postgres_1
177475be25a3 redis:latest "docker-entrypoint.s…" 2 minutes ago Up 2 minutes (healthy) 6379/tcp airflow_redis_1
I followed all steps as described in this URL, every airflow component is working great but the airflow trigerrer shows an unhealthy status :/
Im kinda new to docker i just know the basics and i don't really know how to debug that, can anyone help me up ?
Try to follow all steps on their website including mkdir ./dags ./logs ./plugins echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env.
I don't know but it works then, but still unhealthy,
airflow.apache.org/docs/apache-airflow/stable/start/docker.html
I am following this example:
I find the namenode as follows:
(dash_apps_36) idf#ubvm:~/Documents/docker-hadoop$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2ce71013488f bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 8088/tcp resourcemanager
7fc72b887e23 bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 0.0.0.0:9000->9000/tcp, 0.0.0.0:9870->9870/tcp namenode
9c3f8c435b87 bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 8042/tcp nodemanager
4d57c5bf2f9e bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 9864/tcp datanode
c7eef33905e8 bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8 "/entrypoint.sh /run\u2026" 17 minutes ago Up 17 minutes (healthy) 8188/tcp historyserver
2b1144867924 redis "docker-entrypoint.s\u2026" 6 days ago Exited (137) 6 days ago serene_keldysh
9d9463166b63 redis "docker-entrypoint.s\u2026" 6 days ago Created eloquent_merkle
47d6488e29bf redis:alpine "docker-entrypoint.s\u2026" 4 weeks ago Exited (137) 4 weeks ago wikipedia-playground_redis_1
be5ea0df0401 mcr.microsoft.com/mssql/server:2019-latest "/opt/mssql/bin/perm\u2026" 18 months ago Exited (255) 18 months ago 0.0.0.0:1433->1433/tcp mssql
Using 7fc72b887e23 as the ID of the namenode, when I get to the command
$ docker cp ../hadoop-mapreduce-examples-2.7.1-sources.jar cb0c13085cd3:hadoop-mapreduce-examples-2.7.1-sources.jar
I get an error:
$: command not found
Not sure what I am doing wrong.
Remove the $ at the beginning. That's what $: command not found means. Easy to miss when copy pasting code
docker cp ../hadoop-mapreduce-examples-2.7.1-sources.jar cb0c13085cd3:hadoop-mapreduce-examples-2.7.1-sources.jar
I follow the official tutorial of deploying docker services https://docs.docker.com/get-started/part5/#add-a-new-service-and-redeploy, in the first time i tried this the containers was running as expected but after that the containers shutdown and restarting (i notice that using the visualizer service provided by docker )
and when i execute the command :
docker stack ps getstartedlab
NAME DESIRED STATE CURRENT STATE
ERROR
getstartedlab_web.1 Running Preparing 2 seconds ago
\_ getstartedlab_web.1 Shutdown Failed 4 minutes ago
"task: non-zero exit (2)"
i read in this post https://github.com/docker/machine/issues/3747 that the problem came from the firewall that may block the icmp, i tried to ping docker.com and i had 100% loss packet but when i ping google.com it's ok with no loss packets.
the result of docker ps --all is :
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9f67c82da7c7 username/repo:tag "python app.py" 2 minutes ago Exited (2) 2 minutes ago getstartedlab_web.2.zcnr0ld9bioy0dffsxmn8zss5
f59e413b0780 username/repo:tag "python app.py" 4 minutes ago Exited (2) 4 minutes ago getstartedlab_web.5.ymxgnsf7n8306yr8963xyyljv
9ab631c4057c username/repo:tag "python app.py" 10 minutes ago Exited (2) 10 minutes ago getstartedlab_web.5.zr3gsvgbyxs8c51fiko5h9jxp
bee5816ce1f2 dockersamples/visualizer:stable "npm start" 15 minutes ago Up 15 minutes 8080/tcp getstartedlab_visualizer.1.oyiqwb5esq6zakcdtiw4txh8a
cadca594f8cd username/repo:tag "python app.py" 24 hours ago Exited (2) 24 hours ago getstartedlab_web.1.zehutsl9cefrccqrj86dz4ap7
576b1a6db0b0 username/repo:tag "python app.py" 24 hours ago Exited (2) 24 hours ago getstartedlab_web.5.za9xvxpo5yvl20kha9sjcimmz
2804ebc4fc0c username/repo:tag "python app.py" 24 hours ago Exited (2) 24 hours ago getstartedlab_web.1.zlk42chspvym3jxkzs2nc8k2d
03efb2b04489 dockersamples/visualizer:stable "npm start" 24 hours ago Exited (255) 16 minutes ago 8080/tcp getstartedlab_visualizer.1.qyp4egtu9vcd31kf2jxtzxko3
b85fd1600955 username/repo:tag "python app.py" 2 days ago Exited (2) 2 days ago getstartedlab_web.5.kzrj3m5c3jgkuox0ulpszizee
and the
docker logs 9f67c82da7c7
python: can't open file 'app.py': [Errno 2] No such file or directory
exuse me for representation of the results of those commands because when i copy past the results the lines breaks, how to copy past and preserve the same displaying ?
can anyone have a fix to this problem ? thanks.