where is the docker image stored in gitlab ci? - docker

Question

I have build a docker image successfully and tag it as testdock:latest ($CI_REGISTRY_IMAGE:latest) the $CI_REGISTRY variable is kept in GitLab project variable.
I have another stage , to start scanning the testdock image by using Trivy:
the process is just stuck without progress. I am guessing is that the image cannot be found or something wrong with the docker environment in GitLab.
Where is the `docker image (testdock)` stored?
this is the command that I used for Trivy to scan the testdock image:
$ TRIVY_INSECURE=true trivy --skip-update --output "$CI_PROJECT_DIR/scanning-report.json" $CI_REGISTRY_IMAGE:latest
the yml:
build:
stage: build
image: $CI_REGISTRY/devops/docker:latest
services:
- $CI_REGISTRY/devops/docker:dind-nx1.0
#tags:
# - docker
variables:
# No need to clone the repo, we exclusively work on artifacts. See
# https://docs.gitlab.com/ee/ci/runners/README.html#git-strategy
TRIVY_USERNAME: "$CI_REGISTRY_USER"
TRIVY_PASSWORD: "$CI_REGISTRY_PASSWORD"
TRIVY_AUTH_URL: "$CI_REGISTRY"
FULL_IMAGE_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
# Tell docker CLI how to talk to Docker daemon.
DOCKER_HOST: tcp://localhost:2375/
# Use the overlayfs driver for improved performance.
DOCKER_DRIVER: overlay2
# Disable TLS since we're running inside local network.
DOCKER_TLS_CERTDIR: ""
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- docker build -t $FULL_IMAGE_NAME .
# - docker push $CI_REGISTRY_IMAGE:latest
security_scan:
stage: test
image:
name: $CI_REGISTRY/devops/trivy/trivy:0.20.1
entrypoint: [""]
services:
- $CI_REGISTRY/devops/docker:dind-nx1.0
#tags:
# - docker
variables:
# No need to clone the repo, we exclusively work on artifacts. See
# https://docs.gitlab.com/ee/ci/runners/README.html#git-strategy
# GIT_STRATEGY: none
TRIVY_USERNAME: "$CI_REGISTRY_USER"
TRIVY_PASSWORD: "$CI_REGISTRY_PASSWORD"
TRIVY_AUTH_URL: "$CI_REGISTRY"
FULL_IMAGE_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
# Tell docker CLI how to talk to Docker daemon.
DOCKER_HOST: tcp://localhost:2375/
# Use the overlayfs driver for improved performance.
DOCKER_DRIVER: overlay2
# Disable TLS since we're running inside local network.
DOCKER_TLS_CERTDIR: ""
before_script:
- git config --global http.sslVerify false
- git clone $CI_REPOSITORY_URL
- echo "the project directory is - $CI_PROJECT_DIR"
- echo "the CI_REGISTRY_IMAGE variable is - $CI_REGISTRY_IMAGE"
- echo "the full image name is - $FULL_IMAGE_NAME"
- ls -la
- trivy -h | grep cache
- mkdir -p /root/.cache/trivy/db
- ls -la
- cp "eval-trivy-2/trivy-offline.db.tgz" "/root/.cache/trivy/db"
- cd /root/.cache/trivy/db
- tar xvf trivy-offline.db.tgz
- ls -la
script:
- trivy --version
- time trivy image --clear-cache
# running 1 hr and stopped.
#- TRIVY_INSECURE=true trivy --skip-update $CI_REGISTRY_IMAGE:latest
#- TRIVY_INSECURE=true trivy --skip-update -f json -o scanning-report.json $CI_REGISTRY/devops/aquasec/trivy:0.16.0
- TRIVY_INSECURE=true trivy --skip-update -o "$CI_PROJECT_DIR/scanning-report.json" $FULL_IMAGE_NAME
#keep loading by using testdock:latest
#- TRIVY_INSECURE=true trivy --skip-update -o "$CI_PROJECT_DIR/scanning-report.json" testdock:latest
# - TRIVY_INSECURE=true trivy --skip-update --exit-code 1 --severity CRITICAL $CI_REGISTRY/devops/aquasec/trivy:0.16.0
artifacts:
when: always
reports:
container_scanning: scanning-report.json

Answer 1

All jobs are running isolated. Therefore jobA normally does not know what jobB produced as long as you do not tell the job specifically to pass things on to the next job with the artifacts directive.
In your case you build your image in your job, but if you did not push it - it will be just like any throw away data and lost at the next stage. The easiest way is to push it to a docker registry and use it from there. eg. a common practice is to tag it with the commit SHA instead of latest. This way you can ensure you are always hitting the right image.

Answer 2

final gitlan-ci.yml which works well now:
variables:
# Tell docker CLI how to talk to Docker daemon.
DOCKER_HOST: tcp://localhost:2375/
# Use the overlayfs driver for improved performance.
DOCKER_DRIVER: overlay2
# Disable TLS since we're running inside local network.
DOCKER_TLS_CERTDIR: ""
services:
- $CI_REGISTRY/devops/docker:dind-nx1.0
stages:
- build
- test
#include:
# Trivy integration with GitLab Container Scanning
# - remote: "https://github.com/aquasecurity/trivy/raw/master/contrib/Trivy.gitlab-ci.yml"
build:
image: $CI_REGISTRY/devops/docker:latest
stage: build
variables:
IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
script:
- docker info
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
- docker build -t $IMAGE .
- docker tag $IMAGE $CI_REGISTRY/$IMAGE
- docker push $CI_REGISTRY/$IMAGE
Trivy_container_scanning:
stage: test
image:
name: $CI_REGISTRY/devops/trivy/trivy:0.20.1
variables:
# Override the GIT_STRATEGY variable in your `.gitlab-ci.yml` file and set it to `fetch` if you want to provide a `clair-whitelist.yml`
# file. See https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html#overriding-the-container-scanning-template
# for details
GIT_STRATEGY: none
IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
allow_failure: true
before_script:
- trivy image --reset
- git config --global http.sslVerify false
- git clone $CI_REPOSITORY_URL
- echo "the project directory is - $CI_PROJECT_DIR"
- echo "the registry image is - $CI_REGISTRY_IMAGE"
- ls -la
- trivy -h | grep cache
- mkdir -p /root/.cache/trivy/db
- ls -la
- cp "eval-trivy-4/trivy-offline.db.tgz" "/root/.cache/trivy/db"
- cd /root/.cache/trivy/db
- tar xvf trivy-offline.db.tgz
- ls -la
#- export TRIVY_VERSION=${TRIVY_VERSION:-v0.19.2}
#- apk add --no-cache curl docker-cli
#- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
#- curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin ${TRIVY_VERSION}
#- curl -sSL -o /tmp/trivy-gitlab.tpl https://github.com/aquasecurity/trivy/raw/${TRIVY_VERSION}/contrib/gitlab.tpl
script:
- TRIVY_INSECURE=true trivy image --skip-update -f json -o "$CI_PROJECT_DIR/gl-container-scanning-report.json" $CI_REGISTRY/$IMAGE
#unable to write results: failed to initialize template writer: error retrieving template from path: open /tmp/trivy-gitlab.tpl: no such file or directory
# - TRIVY_INSECURE=true trivy image --skip-update --format template --template "#/tmp/trivy-gitlab.tpl" -o gl-container-scanning-report.json $CI_REGISTRY/$IMAGE
#scan error
#- trivy --skip-update --format template --template "#/tmp/trivy-gitlab.tpl" -o gl-container-scanning-report.json $CI_REGISTRY/$IMAGE
#- trivy --exit-code 0 --cache-dir .trivycache/ --no-progress --format template --template "#/tmp/trivy-gitlab.tpl" -o gl-container-scanning-report.json $IMAGE
# cache:
# paths:
# - .trivycache/
artifacts:
reports:
container_scanning: gl-container-scanning-report.json
reference and modified for my env
https://gitlab.com/aquasecurity/trivy-ci-test/-/blob/master/.gitlab-ci.yml