Could not resolve host when trying to access service in gitlab - docker

Question

I'm trying to access a gitlab-service from a container started with docker run, but it doesn't seem to work.
They actually have a nice section on gitlab about this: https://docs.gitlab.com/ee/ci/services/#using-services-with-docker-run-docker-in-docker-side-by-side
However, even after a 1:1 copy of their code:
access-service:
stage: build
image: docker:19.03.1
before_script:
- echo "Overriding default before_script"
services:
- docker:dind # necessary for docker run
- tutum/wordpress:latest
variables:
FF_NETWORK_PER_BUILD: "true" # activate container-to-container networking
script: |
docker run --rm --name curl \
--volume "$(pwd)":"$(pwd)" \
--workdir "$(pwd)" \
--network=host \
curlimages/curl:7.74.0 curl "http://tutum-wordpress"
I get an error:
Running with gitlab-runner 14.3.4 (77516d85)
on gitlab-aws-autoscaler 7ee750d2
feature flags: FF_NETWORK_PER_BUILD:true, FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR:true
Preparing the "docker+machine" executor 02:34
Using Docker executor with image docker:19.03.1 ...
WARNING: Container based cache volumes creation is disabled. Will not create volume for "/cache"
Starting service docker:dind ...
Authenticating with credentials from $DOCKER_AUTH_CONFIG
Pulling docker image docker:dind ...
Using docker image sha256:1a42336ff683d7dadd320ea6fe9d93a5b101474346302d23f96c9b4546cb414d for docker:dind with digest docker#sha256:6f2ae4a5fd85ccf85cdd829057a34ace894d25d544e5e4d9f2e7109297fedf8d ...
Starting service tutum/wordpress:latest ...
Authenticating with credentials from $DOCKER_AUTH_CONFIG
Pulling docker image tutum/wordpress:latest ...
Using docker image sha256:7e7f97a602ff0c3a30afaaac1e681c72003b4c8a76f8a90696f03e785bf36b90 for tutum/wordpress:latest with digest tutum/wordpress#sha256:2aa05fd3e8543b615fc07a628da066b48e6bf41cceeeb8f4b81e189de6eeda77 ...
Waiting for services to be up and running...
*** WARNING: Service runner-7ee750d2-project-2-concurrent-0-483783518ce3e922-docker-0 probably didn't start properly.
Health check error:
service "runner-7ee750d2-project-2-concurrent-0-483783518ce3e922-docker-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2022-02-22T20:44:10.523612305Z Generating RSA private key, 4096 bit long modulus (2 primes)
2022-02-22T20:44:11.037778878Z ...................................................................................++++
2022-02-22T20:44:11.319540033Z ..................................++++
2022-02-22T20:44:11.320611978Z e is 65537 (0x010001)
2022-02-22T20:44:11.341349948Z Generating RSA private key, 4096 bit long modulus (2 primes)
2022-02-22T20:44:11.360835661Z .++++
2022-02-22T20:44:11.678902603Z ...................................................++++
2022-02-22T20:44:11.679451336Z e is 65537 (0x010001)
2022-02-22T20:44:11.719133216Z Signature ok
2022-02-22T20:44:11.719148571Z subject=CN = docker:dind server
2022-02-22T20:44:11.719151811Z Getting CA Private Key
2022-02-22T20:44:11.734914635Z /certs/server/cert.pem: OK
2022-02-22T20:44:11.738748856Z Generating RSA private key, 4096 bit long modulus (2 primes)
2022-02-22T20:44:11.993700065Z .........................................++++
2022-02-22T20:44:12.036121070Z .....++++
2022-02-22T20:44:12.036364885Z e is 65537 (0x010001)
2022-02-22T20:44:12.067743203Z Signature ok
2022-02-22T20:44:12.067755273Z subject=CN = docker:dind client
2022-02-22T20:44:12.067758449Z Getting CA Private Key
2022-02-22T20:44:12.081823033Z /certs/client/cert.pem: OK
2022-02-22T20:44:12.174949567Z time="2022-02-22T20:44:12.174783104Z" level=info msg="Starting up"
2022-02-22T20:44:12.177055953Z time="2022-02-22T20:44:12.176931675Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
2022-02-22T20:44:12.177086275Z failed to load listeners: can't create unix socket /var/run/docker.sock: device or resource busy
*********
Authenticating with credentials from $DOCKER_AUTH_CONFIG
Pulling docker image docker:19.03.1 ...
Using docker image sha256:0cecfefe921f22fc898f7a0055358380c8870ab6f05b01999367911714fe9d00 for docker:19.03.1 with digest docker#sha256:2dcf87c9893b05ab815880e3d223cd6976c388a6f6697de10e90523255259ca4 ...
Not using umask - FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR is set!
...
$ docker run --rm --name curl \ # collapsed multi-line command
Unable to find image 'curlimages/curl:7.74.0' locally
7.74.0: Pulling from curlimages/curl
aad63a933944: Pulling fs layer
...
3d4876cbff99: Pull complete
110e7f874674: Pull complete
Digest: sha256:a3e534fced74aeea171c4b59082f265d66914d09a71062739e5c871ed108a46e
Status: Downloaded newer image for curlimages/curl:7.74.0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: tutum-wordpress
Can anyone give me a pointer why this is not working? Does this have to do with the fact that this is the executer docker+machine and not docker?
Here's our config.toml:
[[runners]]
name = "gitlab-aws-autoscaler"
url = "https://code.example.com"
token = "${TOKEN}"
executor = "docker+machine"
limit = ${LIMIT_MEDIUM_RUNNERS}
[runners.docker]
image = "example/gitlabrunner:2.10"
privileged = true
disable_cache = true
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache", "/builds:/builds"]
wait_for_services_timeout = 120
[runners.cache]
Type = "s3"
ServerAddress = "s3.amazonaws.com"
AccessKey = "${KEY}"
SecretKey = "${SECRET}"
BucketName = "example-gitlab-runner-cache-virginia"
BucketLocation = "us-east-1"
Shared = true
[runners.machine]
IdleCount = 0
IdleTime = 1800
MaxBuilds = 100
MachineDriver = "amazonec2"
MachineName = "gitlab-docker-machine-%s"
MachineOptions = [
"amazonec2-instance-type=t2.medium",
"amazonec2-access-key=${KEY}",
"amazonec2-secret-key=${SECRET}",
"amazonec2-root-size=100", # GB
"amazonec2-region=us-east-1",
"amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true",
"amazonec2-security-group=EC2-X-ci-runner",
"amazonec2-vpc-id=vpc-XXX",
"amazonec2-subnet-id=subnet-XXX",
"amazonec2-zone=b",
"amazonec2-use-private-address=true",
"amazonec2-private-address-only=true"
]
Edit:
When trying to set the DOCKER_HOST variable as suggested in one answer, I get the following errors:
Running with gitlab-runner 14.3.4 (77516d85)
on gitlab-aws-autoscaler 7ee750d2
feature flags: FF_NETWORK_PER_BUILD:true, FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR:true
Preparing the "docker+machine" executor 02:42
Using Docker executor with image docker:19.03.1 ...
WARNING: Container based cache volumes creation is disabled. Will not create volume for "/cache"
Starting service docker:dind ...
Authenticating with credentials from $DOCKER_AUTH_CONFIG
Pulling docker image docker:dind ...
Using docker image sha256:1a42336ff683d7dadd320ea6fe9d93a5b101474346302d23f96c9b4546cb414d for docker:dind with digest docker#sha256:6f2ae4a5fd85ccf85cdd829057a34ace894d25d544e5e4d9f2e7109297fedf8d ...
Starting service tutum/wordpress:latest ...
Authenticating with credentials from $DOCKER_AUTH_CONFIG
Pulling docker image tutum/wordpress:latest ...
Using docker image sha256:7e7f97a602ff0c3a30afaaac1e681c72003b4c8a76f8a90696f03e785bf36b90 for tutum/wordpress:latest with digest tutum/wordpress#sha256:2aa05fd3e8543b615fc07a628da066b48e6bf41cceeeb8f4b81e189de6eeda77 ...
Waiting for services to be up and running...
*** WARNING: Service runner-7ee750d2-project-2-concurrent-0-a0ec4dc562ad3891-docker-0 probably didn't start properly.
Health check error:
service "runner-7ee750d2-project-2-concurrent-0-a0ec4dc562ad3891-docker-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2022-02-24T16:21:42.803216350Z time="2022-02-24T16:21:42.803077740Z" level=info msg="Starting up"
2022-02-24T16:21:42.804161387Z time="2022-02-24T16:21:42.804107933Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
2022-02-24T16:21:42.804233443Z failed to load listeners: can't create unix socket /var/run/docker.sock: device or resource busy
*********
Authenticating with credentials from $DOCKER_AUTH_CONFIG
Pulling docker image docker:19.03.1 ...
Using docker image sha256:0cecfefe921f22fc898f7a0055358380c8870ab6f05b01999367911714fe9d00 for docker:19.03.1 with digest docker#sha256:2dcf87c9893b05ab815880e3d223cd6976c388a6f6697de10e90523255259ca4 ...

Answer 1

The issue here is that your job is not utilizing the docker:dind service. While you have your job configured mostly correct, your docker GitLab runner defines the following volumes configuration:
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache", "/builds:/builds"]
When bind-mounting /var/run/docker.sock, and not providing the DOCKER_HOST environment variable, your jobs will default to using the bind-mounted docker socket and connect to the daemon on the "metal" host directly, instead of connecting to the docker:dind container, which is required for this services: setup to work correctly.
You can run docker info in your job to confirm this.
You should be able to fix this by setting the DOCKER_HOST environment variable in your job (normally, this is set for you when using gitlab.com runners, which is why it is omitted in their documentation).
access-service:
variables:
DOCKER_HOST: "tcp://docker:2375"
DOCKER_TLS_CERTDIR: ""
# ...
Note: DOCKER_TLS_CERTDIR is also unset here to disable TLS to ensure port 2375 is used. Using TLS is an available option and should be considered more secure.