Gitlab CI: old Docker images? - docker

Question

Our project uses a multi-stage CI setup where the first stage checks for modification of files like package-lock.json and Gemfile.lock, compiles all these dependencies and then pushes them to the Gitlab container registry.
Using --cache-from in Docker build based on the current mainline branch, this is quite fast and the Docker layering mechanism helps to prevent repetition of steps.
Subsequent stages and jobs then use the Docker image pushed in the first stage as their image:.
Abbreviated configuration for readability:
stages:
- create_builder_image
- test
Create Builder Image:
stage: create_builder_image
script:
- export DOCKER_BRANCH_TAG=$CI_COMMIT_REF_SLUG
# do stuff to build the image, using cache to speed it up
- docker push $GITLAB_IMAGE/builder:$DOCKER_BRANCH_TAG
Run Tests:
image: $GITLAB_IMAGE/builder:$CI_COMMIT_REF_SLUG
stage: test
script:
# do stuff in the context of the image build in the first stage
Unfortunately, when working on longer-running feature branches, we now have a situation where it looks like the image in the second step is sometimes outdated and not pulling the latest version from the registry before starting the job, which makes subsequent jobs complain about missing dependencies.
Is there anything I can do to force it to always pull the latest image for each job?

Answer 1

As already written in the comments, i would not use the $CI_COMMIT_REF_SLUG for tagging. Simply because it is not guaranteed that all pipelines will run in the same order, and this alone can create issues. The same one you are currently experiencing.
I recommend to use $CI_COMMIT_SHA as it is bound to the pipeline. I would also rely on previous builds for caching and i will shortly outline my approach here
stages:
- create_builder_image
- test
- deploy
Create Builder Image:
stage: create_builder_image
script:
- (docker pull $GITLAB_IMAGE/builder:$CI_COMMIT_REF_SLUG && export DOCKER_CACHE_TAG=$CI_COMMIT_REF_SLUG) || (docker pull $GITLAB_IMAGE/builder:latest && export DOCKER_CACHE_TAG=latest) || true
- docker build --cache-from $GITLAB_IMAGE/builder:$DOCKER_CACHE_TAG ...
# do stuff to build the image, using cache to speed it up
- docker push $GITLAB_IMAGE/builder:$CI_COMMIT_SHA
Run Tests:
image: $GITLAB_IMAGE/builder:$CI_COMMIT_SHA
stage: test
script:
# do stuff in the context of the image build in the first stage
Push image: # pushing the image for the current branch ref, as i know it is a working image and it can than be used for caching by others.
image: docker:20
stage: deploy
variables:
GIT_STRATEGY: none
stage: push
script:
- docker pull $GITLAB_IMAGE/builder:$CI_COMMIT_SHA
- docker tag $GITLAB_IMAGE/builder:$CI_COMMIT_SHA $GITLAB_IMAGE/builder:$CI_COMMIT_REF_SLUG
- docker push $GITLAB_IMAGE/builder:$CI_COMMIT_REF_SLUG
i know it might generate additional build steps, but this way, you can ensure that you will always have the image which belong to the pipeline. You still can use caching and layering from docker, and beneficiary, the image will not be pushed currently if the tests are failing.
Furthermore you can also create an step before creating the build image, where you can figure out, if you do need a new image at all.