error starting container in Kubernetes but not with docker-compose - docker

Question

My container runs fine using docker-compose. But once I apply my deployment on Kubernetes it fails every time on the same error. No matter what I try. I'm stuck and would love some input/help.
I don't know how to debug this. A bit new at Kubernetes. So If someone can guide me, I can try to debug it.
Error stack trace is the following:
Running migration with alembic
INFO [src.core.config] value for BACKEND_CORS_ORIGINS= ['http://localhost:3000', 'http://localhost:8001']
Traceback (most recent call last):
File "pydantic/env_settings.py", line 197, in pydantic.env_settings.EnvSettingsSource.__call__
File "pydantic/env_settings.py", line 131, in pydantic.env_settings.Config.parse_env_var
File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/alembic", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 590, in main
CommandLine(prog=prog).main(argv=argv)
File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 584, in main
self.run_cmd(cfg, options)
File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 561, in run_cmd
fn(
File "/usr/local/lib/python3.9/site-packages/alembic/command.py", line 322, in upgrade
script.run_env()
File "/usr/local/lib/python3.9/site-packages/alembic/script/base.py", line 569, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "alembic/env.py", line 22, in <module>
from src.core.config import settings
File "/app/src/core/config.py", line 72, in <module>
settings = Settings()
File "pydantic/env_settings.py", line 40, in pydantic.env_settings.BaseSettings.__init__
File "pydantic/env_settings.py", line 75, in pydantic.env_settings.BaseSettings._build_values
File "pydantic/env_settings.py", line 200, in pydantic.env_settings.EnvSettingsSource.__call__
pydantic.env_settings.SettingsError: error parsing env var "BACKEND_CORS_ORIGINS"
Alembic migration done
[2022-11-15 21:25:36 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2022-11-15 21:25:36 +0000] [1] [INFO] Listening at: http://0.0.0.0:8001 (1)
[2022-11-15 21:25:36 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2022-11-15 21:25:36 +0000] [23] [INFO] Booting worker with pid: 23
[2022-11-15 21:25:37 +0000] [24] [INFO] Booting worker with pid: 24
[2022-11-15 21:25:37 +0000] [25] [INFO] Booting worker with pid: 25
[2022-11-15 21:25:40 +0000] [23] [ERROR] Exception in worker process
Now how it is set up.
My docker-compose file builds using a Dockerfile. The end of the Dockerfile is as follow. There is an ENTRYPOINT before the CMD instruction
ENTRYPOINT ["./docker-entrypoint.sh"]
CMD ["./run.sh"]
docker-entrypoint.sh
if [ -n "$DB_SERVER" ]; then
./wait-for-it.sh "$DB_SERVER:${DB_PORT:-3306}"
fi
# Run the main container command.
exec "$#"
This works because I can see the logs from the 'wait-for-it' script
run.sh (where the error comes from)
export APP_MODULE=${APP_MODULE-src.main:app}
export HOST=${HOST:-0.0.0.0}
export PORT=${PORT:-8001}
export BACKEND_CORS_ORIGINS=${BACKEND_CORS_ORIGINS:-'http://localhost:8001'}
echo "Running migration with alembic"
# Run migrations
alembic upgrade head
echo "Alembic migration done"
exec gunicorn -b $HOST:$PORT "$APP_MODULE" --reload --workers=3 --timeout 0 -k uvicorn.workers.UvicornWorker
I've set a default value to BACKEND_CORS_ORIGINS just in case.
In the stack trace above, at the 2nd line you can see a log. This is inside the pydantic validator for BACKEND_CORS_ORIGINS.
Here is part of my config.py file:
class Settings(BaseSettings):
BACKEND_CORS_ORIGINS: List[AnyHttpUrl] = ["http://localhost:3000", "http://localhost:8001"]
#validator("BACKEND_CORS_ORIGINS", pre=True, allow_reuse=True)
def assemble_cors_origins(cls, value: Union[str, List[str]]) -> Union[List[str], str]:
logger.info(f"value for BACKEND_CORS_ORIGINS= {value}, type of BACKEND_CORS_ORIGINS= {type(value)}")
backend_cors_origins = None
if isinstance(value, str) and not value.startswith("["):
backend_cors_origins = [i.strip() for i in value.split(",")]
elif isinstance(value, (list, str)):
backend_cors_origins = value
logger.info(f"value for BACKEND_CORS_ORIGINS= {backend_cors_origins}, type of BACKEND_CORS_ORIGINS= {type(backend_cors_origins)}")
if backend_cors_origins:
return backend_cors_origins
raise ValueError(value)
For some reasons, I am not seeing the last part of the log where I want to see the type.
And to finish, here is my Deployment.yml file for Kubernetes
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: api-name
spec:
replicas: 1
selector:
matchLabels:
app: api-name
template:
metadata:
labels:
app: api-name
spec:
imagePullSecrets:
- name: secret_name
containers:
- name: api-name
image: [private_registry]/api
env:
- name: MARIADB_USER
value: 'db_user'
- name: MARIADB_PASSWORD
value: 'password'
- name: MARIADB_SERVER
value: 'IP_DB_SERVER'
- name: MARIADB_DATABASE
value: 'db_name'
- name: MARIADB_PORT
value: '1234'
- name: BACKEND_CORS_ORIGINS
value: 'http://localhost:8001'
resources:
limits:
memory: "512Mi"
cpu: "500m"
ports:
- containerPort: 8001
---
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: svc-api
spec:
type: NodePort
selector:
app: api-name
ports:
- port: 8001
targetPort: 8001
Can someone help or point my in the right direction? Been trying everything I know to resolve this without any luck.
Maybe I need more config files for Kubernetes?
If I remove the alembic command in the run.sh script, I'm still getting the same error, but without the log inside the #validator

Answer 1

The issue seems to come from this line of export
BACKEND_CORS_ORIGINS=${BACKEND_CORS_ORIGINS:-'http://localhost:8001'}
It seems to expect a list of URLs so maybe you can change the code
- name: BACKEND_CORS_ORIGINS
value:
- 'localhost:8001'
in Deployment.yml