airflow gunircorn [CRITICAL] WORKER TIMEOUT - docker

Question

I am trying to run apache airflow in ECS using the v1-10-stable branch of apache/airflow using my fork airflow. I am using env variables to set executor, Postgres and Redis info to the webserver.
AIRFLOW__CORE__SQL_ALCHEMY_CONN="postgresql+psycopg2://airflow_user:airflow_password#postgres:5432/airflow_db"
AIRFLOW__CELERY__RESULT_BACKEND="db+postgresql://airflow_user:airflow_password#postgres:5432/airflow_db"
AIRFLOW__CELERY__BROKER_URL="redis://redis_queue:6379/1"
AIRFLOW__CORE__EXECUTOR=CeleryExecutor
FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
AIRFLOW__CORE__LOAD_EXAMPLES=False
I am using CMD-SHELL [ -f /home/airflow/airflow/airflow-webserver.pid ] as health check for the ECS container. I can make a connection to the Postgres and Redis from the docker container so there is no issue of security groups as well.
With docker ps I can see that the container is healthy and container port mapping with ec2 instance 0.0.0.0:32794->8080/tcp
But when I try to open the webserver UI, it's not opening. Even with curl its not working. I have tried curl localhost:32794 from the ec2-instance and curl localhost:8080 from the container, but none of them are working. telnet is working in both cases.
In the container logs, I can see that the gunicorn workers are constantly getting a timeout
[2019-11-25 05:30:39,236] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-11-25 05:30:39 +0000] [11] [CRITICAL] WORKER TIMEOUT (pid:17337)
[2019-11-25 05:30:39 +0000] [17337] [INFO] Worker exiting (pid: 17337)
[2019-11-25 05:30:39,430] {dagbag.py:90} INFO - Filling up the DagBag from /home/airflow/airflow/dags
[2019-11-25 05:30:39,472] {dagbag.py:90} INFO - Filling up the DagBag from /home/airflow/airflow/dags
[2019-11-25 05:30:39,479] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-11-25 05:30:39,447] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-11-25 05:30:39,524] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-11-25 05:30:39,719] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-11-25 05:30:39,930] {dagbag.py:90} INFO - Filling up the DagBag from /home/airflow/airflow/dags
[2019-11-25 05:30:40,139] {dagbag.py:90} INFO - Filling up the DagBag from /home/airflow/airflow/dags
[2019-11-25 05:30:40,244] {dagbag.py:90} INFO - Filling up the DagBag from /home/airflow/airflow/dags
[2019-11-25 05:30:40 +0000] [11] [CRITICAL] WORKER TIMEOUT (pid:17338)
[2019-11-25 05:30:40 +0000] [11] [CRITICAL] WORKER TIMEOUT (pid:17339)
[2019-11-25 05:30:40 +0000] [17393] [INFO] Booting worker with pid: 17393
[2019-11-25 05:30:40,412] {dagbag.py:90} INFO - Filling up the DagBag from /home/airflow/airflow/dags
ec2-instance is using Amazon Linux 2 and I can these logs constantly in /var/log/messages
Nov 25 05:57:15 ip-172-31-67-43 ec2net: [rewrite_aliases] Rewriting aliases of eth0
Nov 25 05:58:16 ip-172-31-67-43 dhclient[2724]: XMT: Solicit on eth0, interval 131000ms.
Nov 25 06:00:27 ip-172-31-67-43 dhclient[2724]: XMT: Solicit on eth0, interval 127900ms.
Nov 25 06:01:01 ip-172-31-67-43 systemd: Created slice User Slice of root.
Nov 25 06:01:01 ip-172-31-67-43 systemd: Starting User Slice of root.
Nov 25 06:01:01 ip-172-31-67-43 systemd: Started Session 77 of user root.
Nov 25 06:01:01 ip-172-31-67-43 systemd: Starting Session 77 of user root.
Nov 25 06:01:01 ip-172-31-67-43 systemd: Removed slice User Slice of root.
Nov 25 06:01:01 ip-172-31-67-43 systemd: Stopping User Slice of root.
Nov 25 06:02:35 ip-172-31-67-43 dhclient[2724]: XMT: Solicit on eth0, interval 131620ms.
Nov 25 06:04:36 ip-172-31-67-43 systemd: Started Session 78 of user ec2-user.
Nov 25 06:04:36 ip-172-31-67-43 systemd-logind: New session 78 of user ec2-user.
Nov 25 06:04:36 ip-172-31-67-43 systemd: Starting Session 78 of user ec2-user.
Nov 25 06:04:46 ip-172-31-67-43 dhclient[2724]: XMT: Solicit on eth0, interval 125300ms.
Nov 25 06:06:52 ip-172-31-67-43 dhclient[2724]: XMT: Solicit on eth0, interval 115230ms.
Nov 25 06:08:47 ip-172-31-67-43 dhclient[2724]: XMT: Solicit on eth0, interval 108100ms.

Answer 1

Regarding these timeout errors:
[CRITICAL] WORKER TIMEOUT
you can set Gunicorn timeouts via these two Airflow environment variables:
AIRFLOW__WEBSERVER__WEB_SERVER_MASTER_TIMEOUT
Number of seconds the webserver waits before killing gunicorn master that doesn’t respond.
AIRFLOW__WEBSERVER__WEB_SERVER_WORKER_TIMEOUT
Number of seconds the gunicorn webserver waits before timing out on a worker.
See the Airflow documentation for more info.
I had to solve this error for my Airflow install. I set both of these timeouts to 300 seconds and that allowed me to get the Airflow web UI loading so that I could then debug the underlying cause of the slow page load.

Answer 2

I was getting this error when deploying Airflow with AIRFLOW_HOME set to my EFS mount. Setting it to ~/airflow fixed the issue.