I am unable to use docker commands in google/cloud-sdk image - docker

I am running a yml script in gitlab where I am using gitlab-ci.yaml to create a docker image and push it to Google Registry. Where as I am unable to run the commands. Here is my gitlab code.
image: docker:latest
services:
- docker:dind
variables:
SPRING_PROFILES_ACTIVE: gitlab-ci
stages:
- build
- imagecreation
- deploy
maven-build:
image: maven:3-jdk-8
stage: build
script: "mvn package -B"
artifacts:
paths:
- target/*.war
docker-build:
image: google/cloud-sdk
stage: imagecreation
script:
- docker build -t gcr.io/project-test-to/counter .
- gcloud docker -- push gcr.io/project-test-to/counter
k8s-deploy:
image: google/cloud-sdk
stage: deploy
script:
- echo "$GOOGLE_KEY" > key.json
- gcloud auth activate-service-account --key-file key.json
- gcloud config unset container/use_client_certificate
- gcloud container clusters get-credentials gitlab --zone us-central1-a --project project-test-to
- kubectl apply -f deployment.yaml
Her is the error that I am getting.Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
ERROR: Job failed: exit code 1
How can I run the docker commands in this image?(google/cloud-sdk)

You need to add Docker in Docker service then bind your runner on this service with DOCKER_HOST . Here my configuration
job:publish:api:
image: google/cloud-sdk:latest
stage: publish
when: on_success
services:
- docker:dind
before_script:
- echo $GCLOUD_SERVICE_KEY > ${HOME}/gcloud-service-key.json
- gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json
- gcloud auth configure-docker
script:
- docker build --compress -t ${GCLOUD_IMAGE_CI_FULLNAME} .
- docker push ${GCLOUD_IMAGE_CI_FULLNAME}
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""

Since the gcloud docker command simply passes commands to docker, and the docker image google/cloud-sdk doesn't seem to ship with docker installed, you might need to mount your host socket into the container like this:
docker-build:
image: google/cloud-sdk
stage: imagecreation
script:
- docker build -t gcr.io/project-test-to/counter .
- gcloud docker -- push gcr.io/project-test-to/counter
volume:
- "/var/run/docker.sock:/var/run/docker.sock"
Please also keep in mind the command gcloud docker is deprecated.

Following up on the original answer, the "gcloud docker" command is now deprecated as mentioned. To push an image to Container Registry, you just need to run the following:
$docker push
I would also suggest setting up the docker command so that you can run it as non root by following the steps mentioned in this documentation.

If this is CircleCI you are using to build the image you can use setup_remote_docker as described in:
https://circleci.com/docs/2.0/configuration-reference/#setup_remote_docker
https://circleci.com/blog/docker-what-you-should-know/
Then the config.yml looks something like:
create_app_docker:
docker:
- image: google/cloud-sdk:latest
steps:
- checkout
- setup_remote_docker
- run:
name: Build docker with application
command: |
docker --version
docker build -t my-app ./
docker images

Related

Gitlab CI/CD: deploy docker image to existing kubernetes cluster

I have node.js application that I need to deploy to exising kubernetes cluster.
The cluster is setup using kops on AWS.
I have created .gitlab-ci.yml file for building docker images.
So, whenever a change is pushed to either master or develop branch. It will build the docker image.
I have already followed steps defined here to add an existing cluster.
Now, I have to roll update to exisitng kubernetes cluster..
# This file is a template, and might need editing before it works on your project.
docker-build-master:
# Official docker image.
image: docker:latest
stage: build
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:prod" .
- docker push "$CI_REGISTRY_IMAGE:prod"
only:
- master
docker-build-dev:
# Official docker image.
image: docker:latest
stage: build
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:dev" .
- docker push "$CI_REGISTRY_IMAGE:dev"
only:
- develop
For now, I am using shared runner.
How can I integrate kubernetes deployment after image is built with gitlab ci/cd to deploy on aws (cluster is created with kops)?
For registry I am using gitlab's container registry not docker hub.
Update
I changed configuration and doing below,
stages:
- docker-build
- deploy
docker-build-master:
image: docker:latest
stage: docker-build
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:prod" .
- docker push "$CI_REGISTRY_IMAGE:prod"
only:
- master
deploy-prod:
stage: deploy
image: roffe/kubectl
script:
- kubectl apply -f scheduler-deployment.yaml
only:
- master
docker-build-dev:
image: docker:latest
stage: docker-build
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:dev" .
- docker push "$CI_REGISTRY_IMAGE:dev"
only:
- develop
But now I am getting below error.
roffe/kubectl with digest roffe/kubectl#sha256:ba13f8ffc55c83a7ca98a6e1337689fad8a5df418cb160fa1a741c80f42979bf ...
$ kubectl apply -f scheduler-deployment.yaml
error: the path "scheduler-deployment.yaml" does not exist
Cleaning up file based variables
00:01
ERROR: Job failed: exit code 1
File scheduler-deployment.yaml does exist in the root directory.

gitlab-ci: Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

I'm installing a personal runner for my projects.
sudo docker run --rm -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register \ --non-interactive \ --executor "docker" \ --docker-image docker:stable \ --url "https://gitlab.com" \ --registration-token "my-token" \
--description "docker-runner privileged" \ --tag-list "docker,aws, gara" \ --run-untagged="true" \ --locked="false" \
--access-level="not_protected" \ --docker-privileged
with this runner i can successfully the step test, build and analysis of my maven projet.
For step docker build, I always have:
Cannot connect to the Docker daemon at tcp://docker:2375. Is the
docker daemon running? ERROR: Job failed: exit code 1
Full log:
Running with gitlab-runner 13.1.1 (6fbc7474) on docker-runner
privileged GYDNagVx Preparing the "docker" executor 00:21 Using Docker
executor with image docker:latest ... Starting service docker:dind ...
Pulling docker image docker:dind ... Using docker image
sha256:b3893e48cf281b58a763fb90904fd5d63595f9e5ae5736ee2d892a2dea6a371a
for docker:dind ... Waiting for services to be up and running...
Pulling docker image docker:latest ... Using docker image
sha256:809cc4dba987efb4641d5b6fef38f47abcde1bc5c5e9850616f0ed3552737e8c
for docker:latest ... Preparing environment 00:04 Running on
runner-gydnagvx-project-15477556-concurrent-0 via e59663817b22...
Getting source from Git repository 00:07 Fetching changes with git
depth set to 50... Reinitialized existing Git repository in
/builds/gara-project/back-end-micro-services/msearch/.git/ Checking
out 80fcdd6e as develop... Removing target/ Skipping Git submodules
setup Downloading artifacts 00:18 Downloading artifacts for
maven-build (788252737)... Downloading artifacts from coordinator...
ok id=788252737 responseStatus=200 OK token=jjUorX2D Executing
"step_script" stage of the job script 00:27 $ docker login -u
"${registry_user}" -p "${registry_pass}" "${registry_url}" WARNING!
Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in
/root/.docker/config.json. Configure a credential helper to remove
this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded $ docker build -t
"${registry_url}/${image_name}:${image_tag}" . Cannot connect to the
Docker daemon at tcp://docker:2375. Is the docker daemon running?
ERROR: Job failed: exit code 1
It seems to be a common error but none of solutions found over internet works for me.
My gitla-ci.yml file:
include:
- local: '/templates/.gitlab-ci-common-template.yml'
variables:
SPRING_PROFILES_ACTIVE: test
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
DOCKER_TLS_CERTDIR: ""
default:
image: maven:3.6.3-jdk-8-slim
cache:
key: "$CI_COMMIT_REF_NAME"
paths:
- .m2/repository/
- target/
stages:
- test
- build
- analysis
- docker
- scan
- deploy
- delivery
maven-test:
stage: test
extends:
- .ci_only
- .skip_build
script:
- mvn test
artifacts:
reports:
junit:
- target/surefire-reports/TEST-*.xml
maven-build:
stage: build
extends:
- .ci_only
- .skip_build
script:
- "./mvnw -Pint clean install -DskipTests=true"
artifacts:
paths:
- target/*
expire_in: 1 week
docker-build:
image: docker:latest
services:
- docker:19.03.0-dind
cache: {}
before_script:
- docker login -u "${registry_user}" -p "${registry_pass}" "${registry_url}"
stage: docker
dependencies:
- maven-build
extends:
- .skip_build
- .ci_only
script:
- docker build -t "${registry_url}/${image_name}:${image_tag}" .
- docker run -d --name ${image_tag} "${registry_url}/${image_name}:${image_tag}"
- sleep 60 && docker ps && docker logs ${image_tag}
- if [ $(docker inspect -f '{{.State.Running}}' ${image_tag}) = "true" ]; then echo "Application well started"; else exit 1; fi
- docker rm -f ${image_tag}
- docker push "${registry_url}/${image_name}:${image_tag}"
docker_release_image:
image: docker:latest
stage: delivery
before_script:
- docker login -u "${registry_user}" -p "${registry_pass}" "${registry_url}"
# - docker login -u "${registry_prod_user}" -p "${registry_prod_pass}" "${registry_prod_url}"
services:
- docker:19.03.0-dind
script:
- docker pull "${registry_url}/${image_name}:${image_tag}"
- docker tag "${registry_url}/${image_name}:${image_tag}" "${registry_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker tag "${registry_url}/${image_name}:${image_tag}" "${registry_url}/${image_name}:latest"
# - docker tag "${registry_url}/${image_name}:${image_tag}" "${registry_prod_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker push "${registry_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker push "${registry_url}/${image_name}:latest"
# - docker push "${registry_prod_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker rmi "${registry_url}/${image_name}:${image_tag}"
when: manual
extends:
- .only_tag_on_master
Please how could I correct it ?
Thanks

How to deploy multi container application to heroku via travis ci

Hey I am using travis ci and i dont understand what i have to write inside the travis yml to deploy multi container into heroku
this is my travis.yml
sudo: required
services:
- docker
before_install:
- docker build -t 307949230/client-test -f ./client/Dockerfile.dev ./client
script:
- docker run 307949230/client-test npm test -- --coverage
after_success:
- docker build -t 307949230/mern-client ./client
- docker build -t 307949230/mern-nginx ./nginx
- docker build -t 307949230/mern-server ./server
# Log in to the docker CLI
- echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_ID" --password-stdin
# Take those images and push them to docker hub
- docker push 307949230/mern-client
- docker push 307949230/mern-nginx
- docker push 307949230/mern-server

Docker + GitLab CI - Tag image as "latest" after successful test

I'm trying to tag auto-built Docker images on my private Registry within GitLab-CI, but the 'release' job fails with:
Error response from daemon: No such image: dev.skibapro.de:5050/dransfeld/dockerci-test:v0.4
This is my .gitlab-ci.yml, the build and test jobs run without errors and docerci-test:v0.4 is present in my Registry after the pipeline has run.
image: docker:stable
variables:
DOCKER_DRIVER: overlay2
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
services:
- docker:dind
stages:
- build
- test
- release
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
build:
only:
- tags
stage: build
script:
- docker build -t $IMAGE_TAG -f docker/Dockerfile .
- docker push $IMAGE_TAG
test:
only:
- tags
stage: test
script:
- docker run $IMAGE_TAG /usr/local/bin/test.sh
release:
only:
- tags
stage: release
script:
- docker tag $IMAGE_TAG "$CI_REGISTRY_IMAGE:latest"
This is the error I'm getting in the job log:
$ docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
$ docker tag $IMAGE_TAG "$CI_REGISTRY_IMAGE:latest"
Error response from daemon: No such image: dev.skibapro.de:5050/dransfeld/dockerci-test:v0.4
ERROR: Job failed: exit code 1
I don't know it the image just isn't present yet when the 'release' stage runs, or if I'm asking docker to do something it can't... I want the latest tag to only be applied after the test stage finished successfully.
Altough Docker seems to support tagging images in remote registries (Add remote tag to a docker image), GitLab needs to pull the image from the remote registry first. From GitLab's Blog (https://about.gitlab.com/2016/05/23/gitlab-container-registry/)
release-image:
stage: release
script:
- docker pull $CONTAINER_TEST_IMAGE
- docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
- docker push $CONTAINER_RELEASE_IMAGE

Docker image with PHP, Composer and Docker installed

I use Gitlab.com for my CI using their shared docker runners. I have a project which requires PHP and composer installed, while it also needs docker to build a docker image of the project.
I've tried for hours to build a docker image which has PHP, composer and docker installed, but I can't seem to figure it out.
For reference, my gitlab-ci.yml file looks like this;
image: docker:latest
variables:
DOCKER_DRIVER: overlay2
services:
- docker:dind
composer:install:
stage: build
artifacts:
paths:
- /
expire_in: 1 week
script:
- docker run --rm --interactive --tty --volume $PWD:/app composer install
build:image:
stage: build
dependencies:
- composer:install
script:
- docker login registry.gitlab.com -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD
- docker build -t registry.gitlab.com/accountname/projectname/develop .
- docker push registry.gitlab.com/accountname/projectname/develop
Using the sample build script provided by Stefan below, I put together the following build file which appears to work perfectly. It builds the project using the project Dockerfile, and pushed the resulting image to my Gitlab repository.
image: docker:latest
variables:
DOCKER_DRIVER: overlay2
services:
- docker:dind
before_script:
- docker run --rm --volume $PWD:/app composer install
build:image:
stage: build
script:
- docker login registry.gitlab.com -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD
- docker build -t registry.gitlab.com/accountname/projectname/develop .
- docker push registry.gitlab.com/accountname/projectname/develop
You should use a 2 job CI and use the artifact feature provided by Gitlab to pass the composer install result as dependencies between jobs.
Your first job (composer:install) could use something like https://hub.docker.com/r/library/composer/ in your script section to install all composer packages, then pass it to the build:image job that builds your Docker image.
This would roughly look like this:
image: docker:latest
variables:
DOCKER_DRIVER: overlay2
services:
- docker:dind
composer:install:
stage: composer
artifacts:
paths:
- /
expire_in: 1 week
script:
- docker run --rm --interactive --tty --volume $PWD:/app composer install
build:image:
stage: build
dependencies:
- composer:install
script:
- docker build -t myimage:latest .
Where your Dockerfile could be something like this (based on this):
FROM php:7.0-cli
COPY . /usr/src/myapp
WORKDIR /usr/src/myapp
CMD [ "php", "./your-script.php" ]

Resources