"Unable to locate package git" when running GitLab CI/CD pipeline - docker

Question

I am trying to set up a GitLab CI/CD pipeline with this following .gitlab-ci.yml file:
stages:
- test
image: "ros:foxy-ros-base-focal"
before_script:
- apt-get -y update && apt-get install -y \
git wget qt5-default \
python3-osrf-pycommon \
python3-catkin-tools \
python3-rosdep \
python3-vcstool \
python3-pip \
python3-colcon-common-extensions \
apt-utils
- rm -rf /var/lib/apt/lists/*
- /bin/bash -c "source /opt/ros/${ROS_DISTRO}/setup.bash; colcon build"
- echo "source /root/dev_ws/install/setup.bash" >> /opt/ros/${ROS_DISTRO}/setup.bash
test_a:
stage: test
script:
- pip3 install pytest
- python -m pytest test_utils.py -vv -s
I am starting from the ROS2 Foxy Base Focal Docker image. However, despite running apt-get update, I still can't find many of the packages, including git as well as several other ROS2 packages. Full log here:
Running with gitlab-runner 13.8.0 (775dd39d)
on docker-auto-scale fa6cab46
Preparing the "docker+machine" executor
00:27
Using Docker executor with image ros:foxy-ros-base-focal ...
Pulling docker image ros:foxy-ros-base-focal ...
Using docker image sha256:59cf2af10ce4181bf4effbc683375f5e201bfe072c808c75fb3ee903b98265b9 for ros:foxy-ros-base-focal with digest ros#sha256:4f924ff4fdee6b7c999ad6bc013741bdf8430466c7a27842ac6255255ce9ae66 ...
Preparing environment
00:02
Running on runner-fa6cab46-project-23977848-concurrent-0 via runner-fa6cab46-srm-1611897044-53c1946d...
Getting source from Git repository
00:02
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/imda_dsl/vama-2/scene-understanding/scene-understanding-manager/.git/
Created fresh repository.
Checking out fc385931 as dev...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:05
$ apt-get -y update && apt-get install -y \ git wget qt5-default \ python3-osrf-pycommon \ python3-catkin-tools \ python3-rosdep \ python3-vcstool \ python3-pip \ python3-colcon-common-extensions \ apt-utils
Get:1 http://packages.ros.org/ros2/ubuntu focal InRelease [4670 B]
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB]
Get:4 http://packages.ros.org/ros2/ubuntu focal/main amd64 Packages [451 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [161 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [598 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [659 kB]
Get:12 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [13.3 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1003 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [21.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [194 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [920 kB]
Get:19 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [4301 B]
Fetched 17.4 MB in 2s (7001 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package git
E: Unable to locate package python3-osrf-pycommon
E: Unable to locate package python3-catkin-tools
E: Unable to locate package python3-rosdep
E: Unable to locate package python3-vcstool
E: Unable to locate package python3-pip
E: Unable to locate package python3-colcon-common-extensions
E: Unable to locate package apt-utils
Cleaning up file based variables
00:01
ERROR: Job failed: exit code 1

Answer 1

Installing packages or even running update commands in your .gitlab-ci.yml is generally against best practices for a CI/CD container because each and every job that runs will have to do the same thing, costing a lot of time as you run more pipelines. If you can't find an existing Docker image that has the packages you need (so as an example, python3 and git), you can create your own images. If you need to extend the image from your job, ros:foxy-ros-base-focal, create a Dockerfile file with the following contents:
FROM ros:foxy-ros-base-focal
MAINTAINER your name "your email"
RUN apt-get update -yqq
RUN apt-get install -yqq git
You can install/configure whatever else you need to do in there too, then when you're done build the image:
docker build /path/to/dir-with-dockerfile -t tagname:version
Once the build is done you can verify the tag is correct with docker images, then you can push it to a registry (docker hub, gitlab's registry if enabled, private registry, etc) with:
docker login my.hub.example.com
docker push my.hub.example.com/tagname:version
Then in your .gitlab-ci.yml file, you can use the tagname:version image in your jobs:
stages:
- test
image: "tagname:version"
...
If you have to auth to use your registry, you'll have to review the docs here https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#define-an-image-from-a-private-container-registry, and the general docs for using docker images in your pipelines is here: https://docs.gitlab.com/ee/ci/docker/using_docker_images.html