Why e2e database tests failing within CI but not locally? - docker

Question

I've got pipelines for dev, staging and production.
The staging pipeline is where I've got the issue. The pipeline builds just fine on dev (on and off the CI runner) but staging code builds only locally and on live server but will fail
in the CI runner. I indicated suspecting code with <--.
I've checked whether the database container is running at the time of testing and it is up and running. Logs show nothing unusual.
Cypress tests fail on tests where interaction with the database is being tested:
test-ci.sh:
#!/bin/bash
env=$1
fails=""
inspect() {
if [ $1 -ne 0 ]; then
fails="${fails} $2"
fi
}
# run server-side tests
dev() {
docker-compose up -d --build
docker-compose exec -T users python manage.py recreate_db
docker-compose exec -T users python manage.py test
inspect $? users
docker-compose exec -T client npm test -- --coverage --watchAll --watchAll=false
inspect $? client
docker-compose down
}
# run e2e tests
e2e() {
if [ "${env}" = "staging" ]; then
docker-compose -f docker-compose-stage.yml up -d --build
docker-compose -f docker-compose-stage.yml exec -T users python manage.py recreate_db # <--
docker run -e REACT_APP_USERS_SERVICE_URL=$REACT_APP_USERS_SERVICE_URL -v $PWD:/e2e -w /e2e -e CYPRESS_VIDEO=$CYPRESS_VIDEO --network flaskondocker_default cypress/included:6.0.0 --config baseUrl=http://nginx
inspect $? e2e
docker-compose -f docker-compose-stage.yml down
else
docker-compose -f docker-compose-prod.yml up -d --build
docker-compose -f docker-compose-prod.yml exec -T users python manage.py recreate_db
docker run -e REACT_APP_USERS_SERVICE_URL=$REACT_APP_USERS_SERVICE_URL -v $PWD:/e2e -w /e2e -e CYPRESS_VIDEO=$CYPRESS_VIDEO --network flaskondocker_default cypress/included:6.0.0 --config baseUrl=http://nginx
inspect $? e2e
docker-compose -f docker-compose-prod.yml down
fi
}
# run specific tests
if [ "${env}" = "staging" ]; then
echo "****************************************"
echo "Running e2e tests ..."
echo "****************************************"
e2e
elif [ "${env}" = "production" ]; then
echo "****************************************"
echo "Running e2e tests ..."
echo "****************************************"
e2e
else
echo "****************************************"
echo "Running client and server-side tests ..."
echo "****************************************"
dev
fi
if [ -n "${fails}" ]; then
echo "Test failed: ${fails}"
exit 1
else
echo "Tests passed!"
exit 0
fi
The tests are behaving like docker-compose -f docker-compose-stage.yml exec -T users python manage.py recreate_db failed or hasn't been executed but logs show no errors.
gitlab-ci.yml file:
image: docker:stable
services:
- docker:19.03.12-dind
variables:
COMMIT: ${CI_COMMIT_SHORT_SHA}
MAIN_REPO: https://gitlab.com/coding_hedgehog/flaskondocker.git
USERS: training-users
USERS_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/users
USERS_DB: training-users-db
USERS_DB_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/users-db
CLIENT: training-client
CLIENT_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/client
SWAGGER: training-swagger
SWAGGER_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/swagger
stages:
- build
- push
before_script:
- export REACT_APP_USERS_SERVICE_URL=http://127.0.0.1
- export CYPRESS_VIDEO=false
- export SECRET_KEY=pythonrocks
- export AWS_ACCOUNT_ID=nada
- export AWS_ACCESS_KEY_ID=nada
- export AWS_SECRET_ACCESS_KEY=nada
- apk add --no-cache py-pip python2-dev python3-dev libffi-dev openssl-dev gcc libc-dev make npm
- pip install docker-compose
- npm install
compile:
stage: build
script:
- docker pull cypress/included:6.0.0
- sh test-ci.sh $CI_COMMIT_BRANCH
deployment:
stage: push
script:
- sh ./docker-push.sh
when: on_success
Let me just emphasize that the tests are passing locally on my computer as well as on live server. The database-related e2e tests fail when ran headlessly within CI.
What debugging steps I can take knowing that no containers are crashing, logs show no errors, same code builds locally and runs OK live but fails in the CI ?

Answer 1

We have had some issues where database checks worked locally, but not in headless CI. We found out that it was because of datetime fields. The markup response in CI was different than locally. Thus, all assertions that checked dates failed. We fixed this by writing MySQL queries that format the datetime result. Then adjust the assertions in Cypress accordingly. Maybe your problem has to do with this issue.
SELECT DATE_FORMAT(columnname, "%d-%c-%Y") as columnname FROM table
So for further debugging, do you have any simple tests that run correctly in CI? Or does nothing work?