Sporadically my docker container gets 'orphaned' - docker

My docker container builds a software product which takes more or less a couple of hours.
Most times, it runs fine. However, sometimes it gets 'orphaned' towards the end of execution.
By 'orphaned,' I mean:
1. 'docker ps' reports the container properly.
2. 'docker inspect' reports normally too.
3. however, 'docker exec' returns an error, saying "connect: connection refused": unknown"
[jenkins#aga-slave-jenkins-lnx1 ~]$ docker exec -it 6f667c2ca550 bash
connection error: desc = "transport: dial unix /var/run/docker/containerd/docker-containerd.sock: connect: connection refused": unknown
Once orphaned, it wouldn't go to the next step or exit; it's just hanging there forever.
So my only option is to restart the docker daemon in order to end this misery.
Here is my dockerfile after non-crucial parts omitted for brevity.
RUN svn update -q --no-auth-cache --username $SVN_USER --password $SVN_PASSWORD $WORKSPACE/_Build && \
svn update -q --no-auth-cache --username $SVN_USER --password $SVN_PASSWORD $IVY_REPOSITORY && \
ant -f $WORKSPACE/_Build/_Checkout.xml checkoutLibraries $ANT_ARGUMENTS -Daga.component=ui && \
ant -f $WORKSPACE/_Build/_BuildAll.xml retrieveAll && \
ant -f $WORKSPACE/_Build/_BuildAll.xml $ANT_ARGUMENTS -Daga.component=ui -Drun.tests=false -Dgenerate.javadoc=false -Drun.findbugs=false -Drun.checkstyle=false -Drun.pmd=false && \
ant -f $WORKSPACE/_Build/_BuildAll.xml gather
FROM ${AGA_REPO}base_aga${AGA_VERSION}
Once the 'orphan' problem happens, the docker container hangs between the two tasks.
The dockerd logs contain an interesting line which is the last one in the following snippet.
[jenkins#aga-slave-jenkins-lnx1 ~]$ journalctl -u docker.service |grep 'Jan 13' |tail
Jan 13 23:03:30 aga-slave-jenkins-lnx1.aga.net dockerd[11352]: time="2018-01-13T23:03:27-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." module=containerd type=io.containerd.grpc.v1
Jan 13 23:03:30 aga-slave-jenkins-lnx1.aga.net dockerd[11352]: time="2018-01-13T23:03:27-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." module=containerd type=io.containerd.grpc.v1
Jan 13 23:03:30 aga-slave-jenkins-lnx1.aga.net dockerd[11352]: time="2018-01-13T23:03:27-05:00" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." module=containerd type=io.containerd.monitor.v1
Jan 13 23:03:30 aga-slave-jenkins-lnx1.aga.net dockerd[11352]: time="2018-01-13T23:03:27-05:00" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." module=containerd type=io.containerd.runtime.v1
Jan 13 23:03:30 aga-slave-jenkins-lnx1.aga.net dockerd[11352]: time="2018-01-13T23:03:27-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." module=containerd type=io.containerd.grpc.v1
Jan 13 23:03:30 aga-slave-jenkins-lnx1.aga.net dockerd[11352]: time="2018-01-13T23:03:27-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." module=containerd type=io.containerd.grpc.v1
Jan 13 23:03:31 aga-slave-jenkins-lnx1.aga.net dockerd[11352]: time="2018-01-13T23:03:27-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." module=containerd type=io.containerd.grpc.v1
Jan 13 23:03:31 aga-slave-jenkins-lnx1.aga.net dockerd[11352]: time="2018-01-13T23:03:27-05:00" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
Jan 13 23:03:31 aga-slave-jenkins-lnx1.aga.net dockerd[11352]: time="2018-01-13T23:03:27-05:00" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
Jan 13 23:03:31 aga-slave-jenkins-lnx1.aga.net dockerd[11352]: time="2018-01-13T23:03:27-05:00" level=info msg="containerd successfully booted in 0.274601s" module=containerd
I am a beginner in docker and wondering the issue might have to do with the booted containerd.
Thanks for your help!

Thanks for looking.
I came to believe too tight memory for the VM was causing or at least giving too much stress to my docker container.
It's running fine free of trouble with a doubled memory.
Not sure what exactly was going wrong with the small memory.

Related

Unable to fix error Cannot connect to the Docker daemon at tcp://localhost:2375/. Is the docker daemon running? for remote GitLab runner

I am struggling to resolve the issue
Cannot connect to the Docker daemon at tcp://localhost:2375/. Is the docker daemon running?
I am using our companies GitLab EE instance, which comes with a bunch of shared group runners. However I would like to be able to use my own runners especially since I will be able to employ the GPU for some machine learning tasks. I have the following .gitlab-ci.yml:
run_tests:
image: python:3.9-slim-buster
before_script:
- apt-get update
- apt-get install make
script:
- python --version
- pip --version
- make test
build_image:
image: docker:20.10.23
services:
- docker:20.10.23-dind
variables:
DOCKER_TLS_CRETDIR: "/certs"
DOCKER_HOST: tcp://localhost:2375/
before_script:
- echo "User $REGISTRY_USER"
- echo "Token $ACCESS_TOKEN"
- echo "Host $REGISTRY_HOST_ALL"
- echo "$ACCESS_TOKEN" | docker login --username $REGISTRY_USER --password-stdin $REGISTRY_HOST_ALL
script:
- docker build --tag $REGISTRY_HOST_ALL/<PATH_TO_USER>/python-demoapp .
- docker push $REGISTRY_HOST_ALL/<PATH_TO_USER>/python-demoapp
The application is currently a demo and it's used in the following tutorial. Note that <PATH_TO_USER> in the above URLs is just a placeholder (I cannot reveal the original one since it contains internal information) and points at my account space, where the project python-demoapp is located. With untagged jobs enabled, I am hoping to have the following workflow:
Push application code change
GitLab pipeline triggered
2.1 Execute tests
2.2 Build image
2.3 Push image to container repository
Re-use image with application inside (e.g. run locally)
I have setup the variables accordingly to contain my username, an access token (generated in GitLab) and the registry host. All of these are correct and I am able to execute everything up to the docker build ... section.
Now as for the runner I followed the instructions provided in GitLab to set it up. I chose to create a VM (QEMU+KVM+libvirt) with a standard minimal installation of Debian 11 with everything set to default (including NAT network, which appears to be working since I can access the Internet through it), where the runner currently resides. I am doing this in order to save the setup and later on transfer it onto a server and run multiple instances of the VM with slight modification (e.g. GPU passthrough for Nvidia CUDA Docker/Podman setup).
Beside the runner (binary was downloaded from our GitLab instance), I installed Docker CE (in the future will be replaced with Podman due to licensing and pricing) following the official instructions. The Docker executor is ran as a systemd service (docker.service, docker.socket), that is I need sudo to interact with it. The runner has its own user (also part of the sudo group) as the official documentation is telling me to do.
The GitLab runner's configuration file gitlab-runner-config.toml contains the following information:
concurrent = 1
check_interval = 0
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "Test runner (Debian 11 VM, Docker CE, personal computer)"
url = "<COMPANY_GITLAB_INSTANCE_URL>"
id = <RUNNER_ID>
token = "<ACCESS_TOKEN>"
token_obtained_at = 2023-01-24T09:18:33Z
token_expeires_at = 2023-02-01T00:00:00Z
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "python:3.9-slim-buster"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
cache_dir = "/cache"
volumes = ["/cache", "/certs/client", "/var/run/docker.sock"]
shm_size = 0
The configuration file was generated by running
sudo gitlab-runner register --url <COMPANY_GITLAB_INSTANCE_URL> --registration-token <ACCESS_TOKEN>
I added the extra cache volumes beside /cache, the cache_dir and changed priveleged to true` (based on my research). All for this based on various posts (including Docker's own issue tracking system) from people having the same issue.
I have made sure that dockerd is listening on the respective port (see comment below for the original poster):
$ sudo ss -nltp
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=601,fd=3))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=601,fd=4))
LISTEN 0 4096 *:2375 *:* users:(("dockerd",pid=618,fd=9))
In addition I have added export DOCKER_HOST=tcp://0.0.0.0:2375 to the .bashrc of ever user out there (except root - perhaps that's the problem?) including the gitlab-runner user.
The Dockerfile within the repository contains the following:
FROM python:3.9-slim-buster
RUN apt-get update && apt-get install make
The log file from the CICD pipeline for this job is (trimmed down) as follows:
Running with gitlab-runner 15.8.0 (12335144)
on Test runner (Debian 11 VM, Docker CE, personal computer) <IDENTIFIER>, system ID: <SYSTEM_ID>
Preparing the "docker" executor 02:34
Using Docker executor with image docker:20.10.23 ...
Starting service docker:20.10.23-dind ...
Pulling docker image docker:20.10.23-dind ...
Using docker image sha256:70ae571e74c1d711d3d5bf6f47eaaf6a51dd260fe0036c7d6894c008e7d24297 for docker:20.10.23-dind with digest docker#sha256:85a1b877d0f59fd6c7eebaff67436e26f460347a79229cf054dbbe8d5ae9f936 ...
Waiting for services to be up and running (timeout 30 seconds)...
*** WARNING: Service runner-dbms-tss-project-42787-concurrent-0-b0bbcfd1a821fc06-docker-0 probably didn't start properly.
Health check error:
service "runner-dbms-tss-project-42787-concurrent-0-b0bbcfd1a821fc06-docker-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2023-01-26T10:09:30.933962365Z Certificate request self-signature ok
2023-01-26T10:09:30.933981575Z subject=CN = docker:dind server
2023-01-26T10:09:30.943472545Z /certs/server/cert.pem: OK
2023-01-26T10:09:32.607191653Z Certificate request self-signature ok
2023-01-26T10:09:32.607205915Z subject=CN = docker:dind client
2023-01-26T10:09:32.616426179Z /certs/client/cert.pem: OK
2023-01-26T10:09:32.705354066Z time="2023-01-26T10:09:32.705227099Z" level=info msg="Starting up"
2023-01-26T10:09:32.706355355Z time="2023-01-26T10:09:32.706298649Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
2023-01-26T10:09:32.707357671Z time="2023-01-26T10:09:32.707318325Z" level=info msg="libcontainerd: started new containerd process" pid=72
2023-01-26T10:09:32.707460567Z time="2023-01-26T10:09:32.707425103Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2023-01-26T10:09:32.707466043Z time="2023-01-26T10:09:32.707433214Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2023-01-26T10:09:32.707468621Z time="2023-01-26T10:09:32.707445818Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2023-01-26T10:09:32.707491420Z time="2023-01-26T10:09:32.707459517Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2023-01-26T10:09:32.768123834Z time="2023-01-26T10:09:32Z" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
2023-01-26T10:09:32.768761837Z time="2023-01-26T10:09:32.768714616Z" level=info msg="starting containerd" revision=5b842e528e99d4d4c1686467debf2bd4b88ecd86 version=v1.6.15
2023-01-26T10:09:32.775684382Z time="2023-01-26T10:09:32.775633270Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
2023-01-26T10:09:32.775764839Z time="2023-01-26T10:09:32.775729470Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
2023-01-26T10:09:32.779824244Z time="2023-01-26T10:09:32.779733556Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"ip: can't find device 'aufs'\\nmodprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1
2023-01-26T10:09:32.779836825Z time="2023-01-26T10:09:32.779790644Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
2023-01-26T10:09:32.779932891Z time="2023-01-26T10:09:32.779904447Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
2023-01-26T10:09:32.779944348Z time="2023-01-26T10:09:32.779929392Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
2023-01-26T10:09:32.779958443Z time="2023-01-26T10:09:32.779940747Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
2023-01-26T10:09:32.779963141Z time="2023-01-26T10:09:32.779951447Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
2023-01-26T10:09:32.780022382Z time="2023-01-26T10:09:32.780000266Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
2023-01-26T10:09:32.780134525Z time="2023-01-26T10:09:32.780107812Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
2023-01-26T10:09:32.780499276Z time="2023-01-26T10:09:32.780466045Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
2023-01-26T10:09:32.780507315Z time="2023-01-26T10:09:32.780489797Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
2023-01-26T10:09:32.780548237Z time="2023-01-26T10:09:32.780529316Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
2023-01-26T10:09:32.780552144Z time="2023-01-26T10:09:32.780544232Z" level=info msg="metadata content store policy set" policy=shared
2023-01-26T10:09:32.795982271Z time="2023-01-26T10:09:32.795854170Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
2023-01-26T10:09:32.795991535Z time="2023-01-26T10:09:32.795882407Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
2023-01-26T10:09:32.795993243Z time="2023-01-26T10:09:32.795894367Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
2023-01-26T10:09:32.795994639Z time="2023-01-26T10:09:32.795932065Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
2023-01-26T10:09:32.795996061Z time="2023-01-26T10:09:32.795949931Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
2023-01-26T10:09:32.795997456Z time="2023-01-26T10:09:32.795963627Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
2023-01-26T10:09:32.796001074Z time="2023-01-26T10:09:32.795983562Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
2023-01-26T10:09:32.796219139Z time="2023-01-26T10:09:32.796194319Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
2023-01-26T10:09:32.796231068Z time="2023-01-26T10:09:32.796216520Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
2023-01-26T10:09:32.796240878Z time="2023-01-26T10:09:32.796228403Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
2023-01-26T10:09:32.796254974Z time="2023-01-26T10:09:32.796239993Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
2023-01-26T10:09:32.796261567Z time="2023-01-26T10:09:32.796252251Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
2023-01-26T10:09:32.796385360Z time="2023-01-26T10:09:32.796360610Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
2023-01-26T10:09:32.796451372Z time="2023-01-26T10:09:32.796435082Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
2023-01-26T10:09:32.797042788Z time="2023-01-26T10:09:32.796984264Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
2023-01-26T10:09:32.797093357Z time="2023-01-26T10:09:32.797073997Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797100437Z time="2023-01-26T10:09:32.797091084Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
2023-01-26T10:09:32.797148696Z time="2023-01-26T10:09:32.797138286Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797164876Z time="2023-01-26T10:09:32.797153186Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797176732Z time="2023-01-26T10:09:32.797165488Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797187328Z time="2023-01-26T10:09:32.797176464Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797208889Z time="2023-01-26T10:09:32.797196407Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797220812Z time="2023-01-26T10:09:32.797209290Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797232031Z time="2023-01-26T10:09:32.797221051Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797242686Z time="2023-01-26T10:09:32.797231676Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797254415Z time="2023-01-26T10:09:32.797243815Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
2023-01-26T10:09:32.797484534Z time="2023-01-26T10:09:32.797456547Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797500729Z time="2023-01-26T10:09:32.797487444Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797524336Z time="2023-01-26T10:09:32.797502098Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
2023-01-26T10:09:32.797535447Z time="2023-01-26T10:09:32.797526933Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
2023-01-26T10:09:32.797562995Z time="2023-01-26T10:09:32.797539848Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
2023-01-26T10:09:32.797570791Z time="2023-01-26T10:09:32.797558864Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
2023-01-26T10:09:32.797589770Z time="2023-01-26T10:09:32.797579849Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
2023-01-26T10:09:32.797766243Z time="2023-01-26T10:09:32.797741256Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
2023-01-26T10:09:32.797805542Z time="2023-01-26T10:09:32.797792483Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
2023-01-26T10:09:32.797836935Z time="2023-01-26T10:09:32.797820296Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
2023-01-26T10:09:32.797854712Z time="2023-01-26T10:09:32.797842891Z" level=info msg="containerd successfully booted in 0.029983s"
2023-01-26T10:09:32.802286356Z time="2023-01-26T10:09:32.802232926Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2023-01-26T10:09:32.802291484Z time="2023-01-26T10:09:32.802269035Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2023-01-26T10:09:32.802322916Z time="2023-01-26T10:09:32.802306355Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2023-01-26T10:09:32.802369464Z time="2023-01-26T10:09:32.802323232Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2023-01-26T10:09:32.803417318Z time="2023-01-26T10:09:32.803366010Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2023-01-26T10:09:32.803424723Z time="2023-01-26T10:09:32.803376046Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2023-01-26T10:09:32.803426453Z time="2023-01-26T10:09:32.803384392Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2023-01-26T10:09:32.803428210Z time="2023-01-26T10:09:32.803389450Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2023-01-26T10:09:32.837720263Z time="2023-01-26T10:09:32.837658881Z" level=info msg="Loading containers: start."
2023-01-26T10:09:32.886897024Z time="2023-01-26T10:09:32.886828923Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address"
2023-01-26T10:09:32.920867085Z time="2023-01-26T10:09:32.920800006Z" level=info msg="Loading containers: done."
2023-01-26T10:09:32.944768798Z time="2023-01-26T10:09:32.944696558Z" level=info msg="Docker daemon" commit=6051f14 graphdriver(s)=overlay2 version=20.10.23
2023-01-26T10:09:32.944804324Z time="2023-01-26T10:09:32.944774928Z" level=info msg="Daemon has completed initialization"
2023-01-26T10:09:32.973804146Z time="2023-01-26T10:09:32.973688991Z" level=info msg="API listen on /var/run/docker.sock"
2023-01-26T10:09:32.976059008Z time="2023-01-26T10:09:32.975992051Z" level=info msg="API listen on [::]:2376"
*********
Pulling docker image docker:20.10.23 ...
Using docker image sha256:25deb61ef2709b05249ad4e66f949fd572fb43d67805d5ea66fe3f86766b5cef for docker:20.10.23 with digest docker#sha256:2655039c6abfc8a1d75978c5258fccd5c5cedf880b6cfc72077f076d0672c70a ...
Preparing environment 00:00
Running on runner-dbms-tss-project-42787-concurrent-0 via debian...
Getting source from Git repository 00:02
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/<PATH_TO_USER>/python-demoapp/.git/
Checking out 93e494ea as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:01
Using docker image sha256:25deb61ef2709b05249ad4e66f949fd572fb43d67805d5ea66fe3f86766b5cef for docker:20.10.23 with digest docker#sha256:2655039c6abfc8a1d75978c5258fccd5c5cedf880b6cfc72077f076d0672c70a ...
$ echo "User $REGISTRY_USER"
User [MASKED]
$ echo "Token $ACCESS_TOKEN"
Token [MASKED]
$ echo "Host $REGISTRY_HOST_ALL"
Host ..............
$ echo "$ACCESS_TOKEN" | docker login --username $REGISTRY_USER --password-stdin $REGISTRY_HOST_ALL
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker build --tag $REGISTRY_HOST_ALL/<PATH_TO_USER>/python-demoapp .
Cannot connect to the Docker daemon at tcp://localhost:2375/. Is the docker daemon running?
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1
From my understanding I need two images here:
The python-capable one - here the official Python image from Docker Hub, which is used to run the tests as well as for the image that is added to the container registry
The Docker DinD one - this is the Docker in Docker setup, which allows building a Docker image inside a running Docker container.
The second one is way above my head and it's the (for me) obvious culprit for my headaches.
Perhaps important additional information: my computer is outside our company's network. The GitLab instance is accessible externally through user authentification (username + password for the WebUI, access tokens and SSH keys otherwise).
Do I need two separate runners? I have seen a lot of examples, where people are using a single runner to do multiple jobs including testing and image building (even packaging) so I don't believe I do. I am not really a Docker expert as you can probably tell. :D If more information is required, please let me know in the comments below, especially if I am overdoing it and there is a much easier way to accomplish what I am trying to.
DISCUSSION
Health check error regarding Docker volume
I can see the following error in the log posted above:
Health check error:
service "runner-dbms-tss-project-42787-concurrent-0-b0bbcfd1a821fc06-docker-0-wait-for-service" timeout
The footprint looked familiar so I went back to check some old commands I executed and apparently this is a Docker volume. However on my host
$ docker volume ls
DRIVER
local runner-...415a70
local runner-...66cea8
neither volumes have that name. So I am guessing that this is a volume that is created by Docker in Docker.
Adding hosts to JSON configuration file for Docker daemon
I added the following configuration at /etc/systemd/system/docker.service.d/90-docker.conf:
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --config-file /etc/docker/daemon.json
with daemon.json containing the following:
{
"hosts": [
"tcp://0.0.0.0:2375",
"unix:///var/run/docker.sock"
]
}
Now I am noticing an additional error in the job's log:
failed to load listeners: can't create unix socket /var/run/docker.sock: is a directory
On my host I checked and the path is an actual socket file (information retrieved by executing file command on the path). This means that the issues is again inside the Docker container, that is part of the DinD. I have read online that apparently Docker would automatically create the path and it will be a directory for some reason.
In addition the above mentioned error in the original question has now changed to
unable to resolve docker endpoint: Invalid bind address format: http://localhost:2375/
even though I cannot find any http://localhost:2375 entry on my host, leading again to the conclusion that something with the DinD setup went wrong.

docker systemD stuck at activating (start) on kali

Service stuck on activating (start) and never starts
Jan 02 08:35:49 kail dockerd[1337]: time="2021-01-02T08:35:49.254338338Z" level=info msg="Loading containers: start."
Jan 02 08:35:59 kail dockerd[1337]: time="2021-01-02T08:35:59.327638653Z" level=info msg="Container f34db773beee2371244dbcf3d75ded7fed932b2eed8d038b0f8113ab6a19e0ea failed to exit within 10 seconds of signal 15 - using the force"
How to resolve this ?
resolved by deleting the container from the system
rm -rf /var/lib/docker/containers/<container>

Docker image pull fails and makes docker daemon inaccessible until machine restart

I am facing a strange issue with the docker image pull command where the command fails with this error;
[desai#brilp0017 ~]$ docker image pull nginx:latest
latest: Pulling from library/nginx
d121f8d1c412: Extracting [==================================================>]
27.09MB/27.09MB
ebd81fc8c071: Download complete
655316c160af: Download complete
d15953c0e0f8: Download complete
2ee525c5c3cc: Download complete
failed to register layer: Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root534731447: device or resource busy
After this error the docker daemon is no longer accessible and all docker commands return following error;
[desai#brilp0017 ~]$ docker info
Client:
Debug Mode: false
Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
The command systemctl status docker however shows it as running;
[desai#brilp0017 ~]$ systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2020-09-11 14:25:53 BST; 14min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 2380 (dockerd)
Tasks: 14
Memory: 249.5M
CGroup: /system.slice/docker.service
└─2380 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
The journalctl log has the same error line as encountered in the pull command;
Sep 11 14:25:52 brilp0017 systemd[1]: Starting Docker Application Container Engine...
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.345006155+01:00" level=info msg="Starting up"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348597478+01:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348667479+01:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348733420+01:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348765306+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353865701+01:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353908904+01:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353944835+01:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353988191+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.497701794+01:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.816295801+01:00" level=warning msg="Your kernel does not support cgroup rt period"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.816318357+01:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.816442165+01:00" level=info msg="Loading containers: start."
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.101411528+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.125378601+01:00" level=info msg="Loading containers: done."
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.291896277+01:00" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" storage-driver=overlay2
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.292711063+01:00" level=info msg="Docker daemon" commit=48a66213fe graphdriver(s)=overlay2 version=19.03.12-ce
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.293190069+01:00" level=info msg="Daemon has completed initialization"
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.340381428+01:00" level=info msg="API listen on /run/docker.sock"
Sep 11 14:25:53 brilp0017 systemd[1]: Started Docker Application Container Engine.
Sep 11 14:32:38 brilp0017 dockerd[2380]: time="2020-09-11T14:32:38.011501405+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 11 14:33:11 brilp0017 dockerd[2380]: time="2020-09-11T14:33:11.592234770+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 11 14:34:52 brilp0017 dockerd[2380]: time="2020-09-11T14:34:52.864254519+01:00" level=info msg="Attempting next endpoint for pull after error: failed to register layer: Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root534731447: device or resource busy"
After this the error remains the same even after stopping and starting docker service multiple times with systemctl. After restarting the laptop fully and starting the docker service it starts work as expected until the next time docker pull command is used.
I have searched for solution on the internet but majority of them point to the user not being in docker group but that is not the case for me;
[desai#brilp0017 ~]$ groups
sys network power vboxusers wireshark sambashare docker lp wheel desai
Here is the output of docker version before it crashes for version details;
[desai#brilp0017 ~]$ docker version
Client:
Version: 19.03.12-ce
API version: 1.40
Go version: go1.14.5
Git commit: 48a66213fe
Built: Sat Jul 18 01:33:21 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.12-ce
API version: 1.40 (minimum version 1.12)
Go version: go1.14.5
Git commit: 48a66213fe
Built: Sat Jul 18 01:32:59 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.4.0.m
GitCommit: 09814d48d50816305a8e6c1a4ae3e2bcc4ba725a.m
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.18.0
GitCommit: fec3683
I am using Manjaro linux OS;
Operating System: Manjaro Linux
KDE Plasma Version: 5.19.4
KDE Frameworks Version: 5.73.0
Qt Version: 5.15.0
Kernel Version: 4.19.141-2-MANJARO
OS Type: 64-bit
Processors: 8 × Intel® Core™ i7-8550U CPU # 1.80GHz
Memory: 31.2 GiB of RAM
Graphics Processor: Mesa Intel® UHD Graphics 620
Any help on this would be appreciated.
This issue was resolved by updating the kernel to 5.4 version.

GitLab CI pipeline failing when deploying via docker compose

I have a locally hosted GitLab CI setup, that I am running via Docker Compose. I am trying to push a basic app through the pipeline, but keep getting the error below in the runner. My gitlab-ci.yml file is in a repo with a Kotlin project. When I run the same gitlab-ci.yml file in a blank repo (i.e., no Kotlin project, just the gitlab-ci.yml file) it works. Any idea why I'm getting this error and the pipeline is failing?
GitLab CI File
image: alpine
stages:
- build
- test
build:
stage: build
script:
- mkdir build
- touch build/info.txt
artifacts:
paths:
- ./build/libs/
test:
stage: test
script:
- echo "Testing"
- test -f "build/info.txt"
Docker Compose File
version: "3.7"
services:
web:
image: 'gitlab/gitlab-ce:latest'
restart: always
hostname: 'XXX'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://XXX'
ports:
- '80:80'
- '443:443'
- '223:22'
volumes:
- '/Users/XXX/dockvol/srv/gitlab/config:/etc/gitlab'
- '/Users/XXX/dockvol/srv/gitlab/logs:/var/log/gitlab'
- '/Users/XXX/dockvol/srv/gitlab/data:/var/opt/gitlab'
runner:
image: 'gitlab/gitlab-runner:latest'
restart: always
user: root
privileged: true
volumes:
- '/Users/XXX/dockvol/srv/gitlab-runner/config:/etc/gitlab-runner'
- '/var/run/docker.sock:/var/run/docker.sock'
GitLab Runner Logs
[0KRunning with gitlab-runner 12.10.1 (ce065b93)
[0;m[0K on first u7d9d-Gt
[0;msection_start:1589194575:prepare_executor
[0K[0K[36;1mPreparing the "docker" executor[0;m
[0;m[0KUsing Docker executor with image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v0.2.2 ...
[0;m[0KStarting service docker:19.03.8-dind ...
[0;m[0KPulling docker image docker:19.03.8-dind ...
[0;m[0KUsing docker image sha256:c814ba3a41a3de0a9a23b7d0bb36f64257b12aef5103b4ce1d5f1bfc3033aad3 for docker:19.03.8-dind ...
[0;m[0KWaiting for services to be up and running...
[0;m
[0;33m*** WARNING:[0;m Service runner-u7d9d-gt-project-2-concurrent-0-2742755dfb40c120-docker-0 probably didn't start properly.
Health check error:
service "runner-u7d9d-gt-project-2-concurrent-0-2742755dfb40c120-docker-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2020-05-11T10:56:16.757478300Z time="2020-05-11T10:56:16.753561500Z" level=info msg="Starting up"
2020-05-11T10:56:16.757519200Z time="2020-05-11T10:56:16.754810900Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
2020-05-11T10:56:16.757539400Z time="2020-05-11T10:56:16.754999600Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
2020-05-11T10:56:16.759713500Z time="2020-05-11T10:56:16.759610700Z" level=info msg="libcontainerd: started new containerd process" pid=24
2020-05-11T10:56:16.759987700Z time="2020-05-11T10:56:16.759877800Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2020-05-11T10:56:16.760232100Z time="2020-05-11T10:56:16.760052300Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2020-05-11T10:56:16.760440300Z time="2020-05-11T10:56:16.760323100Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
2020-05-11T10:56:16.760697900Z time="2020-05-11T10:56:16.760562700Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2020-05-11T10:56:16.802604300Z time="2020-05-11T10:56:16.802375600Z" level=info msg="starting containerd" revision=7ad184331fa3e55e52b890ea95e65ba581ae3429 version=v1.2.13
2020-05-11T10:56:16.802887300Z time="2020-05-11T10:56:16.802666400Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
2020-05-11T10:56:16.802911600Z time="2020-05-11T10:56:16.802756700Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
2020-05-11T10:56:16.803104600Z time="2020-05-11T10:56:16.802954000Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
2020-05-11T10:56:16.803127900Z time="2020-05-11T10:56:16.802996000Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
2020-05-11T10:56:16.808895200Z time="2020-05-11T10:56:16.808690300Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "ip: can't find device 'aufs'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n": exit status 1"
2020-05-11T10:56:16.808920800Z time="2020-05-11T10:56:16.808735700Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
2020-05-11T10:56:16.808938400Z time="2020-05-11T10:56:16.808831800Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
2020-05-11T10:56:16.809111500Z time="2020-05-11T10:56:16.808985800Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
2020-05-11T10:56:16.809360200Z time="2020-05-11T10:56:16.809185500Z" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
2020-05-11T10:56:16.809517400Z time="2020-05-11T10:56:16.809286000Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
2020-05-11T10:56:16.809541700Z time="2020-05-11T10:56:16.809360200Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin"
2020-05-11T10:56:16.809561500Z time="2020-05-11T10:56:16.809381000Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
2020-05-11T10:56:16.809576500Z time="2020-05-11T10:56:16.809405200Z" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "ip: can't find device 'aufs'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n": exit status 1"
2020-05-11T10:56:16.815691100Z time="2020-05-11T10:56:16.815570700Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
2020-05-11T10:56:16.815717500Z time="2020-05-11T10:56:16.815635400Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
2020-05-11T10:56:16.815792400Z time="2020-05-11T10:56:16.815691100Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
2020-05-11T10:56:16.815812800Z time="2020-05-11T10:56:16.815711600Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
2020-05-11T10:56:16.815832200Z time="2020-05-11T10:56:16.815731400Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
2020-05-11T10:56:16.815959900Z time="2020-05-11T10:56:16.815758300Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
2020-05-11T10:56:16.815979600Z time="2020-05-11T10:56:16.815786300Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
2020-05-11T10:56:16.816031600Z time="2020-05-11T10:56:16.815812800Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
2020-05-11T10:56:16.816050500Z time="2020-05-11T10:56:16.815832200Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
2020-05-11T10:56:16.816069200Z time="2020-05-11T10:56:16.815852500Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
2020-05-11T10:56:16.816256700Z time="2020-05-11T10:56:16.816012200Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
2020-05-11T10:56:16.816295100Z time="2020-05-11T10:56:16.816107400Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
2020-05-11T10:56:16.816670700Z time="2020-05-11T10:56:16.816517200Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
2020-05-11T10:56:16.816689200Z time="2020-05-11T10:56:16.816565100Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
2020-05-11T10:56:16.816905500Z time="2020-05-11T10:56:16.816601200Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.816927300Z time="2020-05-11T10:56:16.816644400Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.816946500Z time="2020-05-11T10:56:16.816664600Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.816970000Z time="2020-05-11T10:56:16.816683100Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.816988200Z time="2020-05-11T10:56:16.816706000Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.817007500Z time="2020-05-11T10:56:16.816725600Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.817027300Z time="2020-05-11T10:56:16.816748100Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.817051600Z time="2020-05-11T10:56:16.816770600Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.817069300Z time="2020-05-11T10:56:16.816826200Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
2020-05-11T10:56:16.817164600Z time="2020-05-11T10:56:16.817013400Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.817184800Z time="2020-05-11T10:56:16.817051600Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.817204200Z time="2020-05-11T10:56:16.817069300Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.817512500Z time="2020-05-11T10:56:16.817088000Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
2020-05-11T10:56:16.817535100Z time="2020-05-11T10:56:16.817246500Z" level=info msg=serving... address="/var/run/docker/containerd/containerd-debug.sock"
2020-05-11T10:56:16.817554300Z time="2020-05-11T10:56:16.817388600Z" level=info msg=serving... address="/var/run/docker/containerd/containerd.sock"
2020-05-11T10:56:16.817887700Z time="2020-05-11T10:56:16.817745100Z" level=info msg="containerd successfully booted in 0.015996s"
2020-05-11T10:56:16.832721600Z time="2020-05-11T10:56:16.831736800Z" level=info msg="Setting the storage driver from the $DOCKER_DRIVER environment variable (overlay2)"
2020-05-11T10:56:16.832749800Z time="2020-05-11T10:56:16.831998200Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2020-05-11T10:56:16.832767100Z time="2020-05-11T10:56:16.832027100Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2020-05-11T10:56:16.832787000Z time="2020-05-11T10:56:16.832051500Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
2020-05-11T10:56:16.832814000Z time="2020-05-11T10:56:16.832071300Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2020-05-11T10:56:16.835365700Z time="2020-05-11T10:56:16.834371800Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2020-05-11T10:56:16.835384000Z time="2020-05-11T10:56:16.834434500Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2020-05-11T10:56:16.835404400Z time="2020-05-11T10:56:16.834464500Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
2020-05-11T10:56:16.835460300Z time="2020-05-11T10:56:16.834487500Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2020-05-11T10:56:16.872802700Z time="2020-05-11T10:56:16.870967500Z" level=info msg="Loading containers: start."
2020-05-11T10:56:16.892366800Z time="2020-05-11T10:56:16.891473000Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'\nip: can't find device 'br_netfilter'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1"
2020-05-11T10:56:17.032576600Z time="2020-05-11T10:56:17.032377200Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address"
2020-05-11T10:56:17.091567300Z time="2020-05-11T10:56:17.091375400Z" level=info msg="Loading containers: done."
2020-05-11T10:56:17.113255800Z time="2020-05-11T10:56:17.113013400Z" level=info msg="Docker daemon" commit=afacb8b7f0 graphdriver(s)=overlay2 version=19.03.8
2020-05-11T10:56:17.113701300Z time="2020-05-11T10:56:17.113556300Z" level=info msg="Daemon has completed initialization"
2020-05-11T10:56:17.179131600Z time="2020-05-11T10:56:17.178944800Z" level=info msg="API listen on [::]:2375"
2020-05-11T10:56:17.179529600Z time="2020-05-11T10:56:17.179155300Z" level=info msg="API listen on /var/run/docker.sock"
[0;33m*********[0;m
[0KPulling docker image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v0.2.2 ...
[0;m[0KUsing docker image sha256:a9a470e7a925ecfd27cfbb60e98c0915f02a3eb8a81f15fb6b11af1baca21e63 for registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v0.2.2 ...
[0;msection_end:1589194608:prepare_executor
[0Ksection_start:1589194608:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on runner-u7d9d-gt-project-2-concurrent-0 via 242fc900f561...
section_end:1589194609:prepare_script
[0Ksection_start:1589194609:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /builds/XXX/starter-project-kotlin/.git/
From http://XXX/XXX/starter-project-kotlin
* [new ref] refs/pipelines/9 -> refs/pipelines/9
fa35e89..260c063 master -> origin/master
[32;1mChecking out 260c0632 as master...[0;m
Removing Dockerfile
[32;1mSkipping Git submodules setup[0;m
section_end:1589194611:get_sources
[0Ksection_start:1589194611:restore_cache
[0K[0K[36;1mRestoring cache[0;m
[0;msection_end:1589194612:restore_cache
[0Ksection_start:1589194612:download_artifacts
[0K[0K[36;1mDownloading artifacts[0;m
[0;msection_end:1589194613:download_artifacts
[0Ksection_start:1589194613:build_script
[0K[0K[36;1mRunning before_script and script[0;m
[0;m[32;1m$ if [[ -z "$CI_COMMIT_TAG" ]]; then # collapsed multi-line command[0;m
[32;1m$ /build/build.sh[0;m
Building Heroku-based application using gliderlabs/herokuish docker image...
invalid reference format
invalid reference format
invalid argument "/master:260c0632aca32f789a54acdb976cde17e0113f62" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
section_end:1589194615:build_script
[0Ksection_start:1589194615:after_script
[0K[0K[36;1mRunning after_script[0;m
[0;msection_end:1589194616:after_script
[0Ksection_start:1589194616:upload_artifacts_on_failure
[0K[0K[36;1mUploading artifacts for failed job[0;m
[0;msection_end:1589194617:upload_artifacts_on_failure
[0K[31;1mERROR: Job failed: exit code 1
[0;m
Seems like build tag "/master:260c0632aca32f789a54acdb976cde17e0113f62" is in wrong format for Docker.
A tag name must be valid ASCII and may contain lowercase and uppercase letters, digits, underscores, periods and dashes. A tag name may not start with a period or a dash and may contain a maximum of 128 characters. Ref.
Is it identical for "$CI_COMMIT_TAG" ENV content on GitLab CI? There are some problems with that build script build.sh.
Some related issues:
https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64530
Docker build failed: tag invalid reference format (Gitlab CI)
Whereas the problem is potentially solved as:
If anyone is having this issue in combination with Heroku-based applications (e.g. in Gitlab AutoDevOps) you might need to activate the GitLab container registry on your GitLab installation and in your project.

Unable to run docker-compose

Following this article on Jhipster, I build the project. I, however, can't run docker-compose. So, I try to figure out the problem. I walk step by step with this Docker-Compose article without luck.
$ sudo service docker status
docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2018-12-27 15:40:22 PST; 2 days ago
Docs: https://docs.docker.com
Main PID: 1960 (dockerd)
Tasks: 20 (limit: 4440)
Memory: 68.2M
CGroup: /system.slice/docker.service
├─1960 /usr/bin/dockerd -H fd://
└─2093 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --s
Dec 27 15:40:10 tk-PC dockerd[1960]: time="2018-12-27T15:40:10.493340278-08:00" level=warning msg="Your kernel does not support swap memory li
Dec 27 15:40:10 tk-PC dockerd[1960]: time="2018-12-27T15:40:10.493612101-08:00" level=warning msg="Your kernel does not support cgroup rt peri
Dec 27 15:40:10 tk-PC dockerd[1960]: time="2018-12-27T15:40:10.493681034-08:00" level=warning msg="Your kernel does not support cgroup rt runt
Dec 27 15:40:10 tk-PC dockerd[1960]: time="2018-12-27T15:40:10.496381656-08:00" level=info msg="Loading containers: start."
Dec 27 15:40:17 tk-PC dockerd[1960]: time="2018-12-27T15:40:17.498415923-08:00" level=info msg="Default bridge (docker0) is assigned with an I
Dec 27 15:40:19 tk-PC dockerd[1960]: time="2018-12-27T15:40:19.646853084-08:00" level=info msg="Loading containers: done."
Dec 27 15:40:22 tk-PC dockerd[1960]: time="2018-12-27T15:40:22.512083092-08:00" level=info msg="Daemon has completed initialization"
Dec 27 15:40:22 tk-PC dockerd[1960]: time="2018-12-27T15:40:22.512266914-08:00" level=info msg="Docker daemon" commit=89658be graphdriver=aufs
Dec 27 15:40:22 tk-PC dockerd[1960]: time="2018-12-27T15:40:22.553322342-08:00" level=info msg="API listen on /var/run/docker.sock"
Dec 27 15:40:22 tk-PC systemd[1]: Started Docker Application Container Engine.
$ sudo ls -la /var/run/docker.sock
srw-rw---- 1 root docker 0 Dec 27 15:39 /var/run/docker.sock
$ sudo usermod -aG docker ${USER}
$ docker-compose -f docker-compose.yml build --build-arg UID=$(id -u)
ERROR: Couldn't connect to Docker daemon at http+docker://localhost - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
The result of the last step is the same as "docker-compose up -d". According to the article, it would be a permission problem if the problem still exists at this point. How can I find what permission issue?
There are multiple ways you can solve this problem. Firstly try to export environment variable of docker host with command:
export DOCKER_HOST=/var/run/docker.sock
If it works you can add the same line to your bashrc config to save this export permanently.
If it doesn't work you can try to modify docker daemon config. It located in
/etc/docker/daemon.json
You'll need to append the localhost to your hosts like that:
"hosts": ["old_hosts_not_modified_only_append_new_one", "tcp://localhost:2376"],
and restart docker daemon using command:
service docker restart
Hope it gonna help ya

Resources