I have been breaking my head over this problem the past few hours. My setup is as follows:
Running Ubuntu 19.10 hosting docker
Running this docker-compose.yml file:
version: '3.7'
services:
jenkins:
command: --httpPort=80 --httpsPort=443 --httpsKeyStore=/var/jenkins_ssl/jenkins.jks --httpsKeyStorePassword=very-secret-password
build: .
privileged: true
user: root
ports:
- 80:8080
- 50000:50000
- 443:443
container_name: jenkins
volumes:
- ~/jenkins:/var/jenkins_home
- ~/jenkins-ssl:/var/jenkins_ssl
- ~/app_config/:/var/app_config/
- ~/app_logs/:/var/app_logs/
- /var/run/docker.sock:/var/run/docker.sock
- /usr/local/bin/docker:/usr/local/bin/docker
The host has the group docker with id 998. My Dockerfile for the jenkins container is as follows:
USER root
ARG DOCKER_GID=998
RUN groupadd -g ${DOCKER_GID} docker
RUN usermod -aG docker jenkins
However, when I try to build something in a Jenkins job, the following error pops up:
/var/jenkins_home/workspace/cicd-test-backend#tmp/durable-79b7d7d6/script.sh: 1: /var/jenkins_home/workspace/cicd-test-backend#tmp/durable-79b7d7d6/script.sh: docker: Permission denied
script returned exit code 127
The build stage is:
steps {
script {
docker.withRegistry('localhost:5000') {
def image = docker.build("url-shortener-backend:${env.BUILD_ID}")
image.push()
}
}
}
Can anyone spot what is going wrong here? Thanks in advance :)
Related
I have a Dockerfile as below:
FROM jenkins/jenkins:latest
USER root
RUN whoami
USER jenkins
RUN whoami
and this docker-compose file
version: '2'
services:
test:
build:
context: .
dockerfile: Dockerfile
container_name: test
hostname: test
ports:
- '8080:8080'
user: root
I am wondering
What is the difference between the user that is defined in the docker-compose and the user that is defined in the dockerfile
How to see the logs of the build stage? When I RUN whoami, how and where I can see the result?
What if question is if I change the user in docker-compose to other
version: '2'
services:
test:
build:
context: .
dockerfile: Dockerfile
container_name: test
hostname: test
ports:
- '8080:8080'
user: other
Why isn't it working
And if I change the dockerfile to
FROM jenkins/jenkins:latest
USER root
RUN whoami
RUN groupadd -g 999 docker && \
usermod -aG staff,docker jenkins
USER jenkins
RUN whoami
and change the docker-compose to
version: '2'
services:
test:
build:
context: .
dockerfile: Dockerfile
container_name: test
hostname: test
ports:
- '8080:8080'
user: jenkins
Still not working.
What is the problem
Another question is when I do docker exec -it container_name bash
It get access to the container as a root user. How to change that
The user config in the docker-compose file will overwrite the user used to execute the command in your container. If you omit it, it will use the last USER set in your Dockerfile. If you have neither, it should default to root.
Note that your whoami commands are run at build-time (not at run-time) so they should not be impacted by what you specified in your docker-compose file which is overriding the user at run-time only.
To see the entire build log, you can use docker compose build --progress flat.
I have just started to learn Docker.
I have tried to run jenkins in my docker.
I have tried the commands:
docker run jenkins ,
docker run jenkins:latest
But showing the error in the docker interactive shell:
C:\Program Files\Docker Toolbox\docker.exe: Error response from daemon: manifest for jenkins:latest not found: manifest unknown: manifest unknown.
You can run the container by using the command
docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins:lts
The documentation page is pretty good.
I would use a docker-compose file to
mount a volume for home to make it persistent
(in order to look into the build workspace you need to attach another container to it)
control the version programmatically
add docker client or other utilities installed later
add 'fixed' agents
docker compose file:
version: '3.5'
services:
jenkins-server:
build: ./JenkinsServer
container_name: jenkins
restart: always
environment:
JAVA_OPTS: "-Xmx1024m"
ports:
- "50000:50000"
- "8080:8080"
networks:
jenkins:
aliases:
- jenkins
volumes:
- jenkins-data:/var/jenkins_home
networks:
jenkins:
external: true
volumes:
jenkins-data:
external: true
dockerfile for server:
FROM jenkins/jenkins:2.263.2-lts
USER root
I am attempting to run e2e tests in the gitlab ci that use a React frontend, Java Spring backend and PostgreSQL.
The relevant pieces of the .gitlab-ci -config are as follows:
variables:
IMAGE_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
FF_NETWORK_PER_BUILD: 1
docker-backend-build:
image: docker:latest
services:
- docker:dind
stage: package
dependencies:
- backend-build
script:
- docker build -t registry.gitlab.com/repo-name .
- docker tag registry.gitlab.com/repo-name $IMAGE_NAME
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com
- docker push $IMAGE_NAME
end-to-end-test:
stage: integration-test
image: node:latest
services:
- name: postgres:9.6
- name: $IMAGE_NAME
alias: backend
variables:
DB_USERNAME: postgres
DB_PASSWORD: postgres
JDBC_CONNECTION_STRING: 'jdbc:postgresql://postgres:5432/database?stringtype=unspecified'
dependencies:
- frontend-build
script:
- cd frontend
- yarn start:ci & ./node_modules/wait-on/bin/wait-on http://backend:9070/api/health http://localhost:3000
- yarn run cy:run
artifacts:
when: always
paths:
- frontend/cypress/videos/*.mp4
- frontend/cypress/screenshots/**/*.png
expire_in: 1 day
The Dockerfile for the backend is as follows:
FROM tomcat:latest
ADD backend/target/server.war /usr/local/tomcat/webapps/
RUN sed -i 's/port="8080"/port="9070"/' /usr/local/tomcat/conf/server.xml
EXPOSE 9070
CMD ["catalina.sh", "run"]
The server.war is created on an earlier stage in the CI-pipeline.
The server.war is set to listen to port 9070, and the Dockerfile succesfully changes the Tomcat port to 9070 as well. The Tomcat instance is able to connect to the postgres instance via postgres:5432 because of the FF_NETWORK_PER_BUILD -flag, but for some reason this script hangs on the wait-on http://backend:9070/api/health command forever. It can not connect to backend:9070 even though the server is up and running. (and the health-endpoint exists). The server doesn't receive any indication that it is trying to be connected to.
What could I be doing wrong? I also tried to connect to http://localhost:9070/api/health but that didn't work either.
The answer for me was simply changing the Dockerfile as follows:
- ADD backend/target/server.war /usr/local/tomcat/webapps/
+ ADD backend/target/server.war /usr/local/tomcat/webapps/ROOT.war
because without that, the server was actually listening in http://backend:9070/api/health/server. Silly me.
I am trying to make sure my docker work or not in my Jenkins,
I am running Jenkins in docker and it was running but when I check in Jenkins Pipeline, it said docker: not found
here is my docker-compose.yml
version: '3.7'
services:
jenkins:
image: jenkinsci/blueocean:latest
user: root
privileged: true
restart: always
ports:
- 8080:8080
volumes:
- ./jenkins_home:/var/jenkins_home
- /var/run/docker.sock:/var/run/docker.sock
- /usr/bin/docker:/usr/bin/docker
registry:
image: registry
container_name: registry
restart: always
ports:
- 5000:5000
then I run sudo docker-compose up -d
then the Jenkins is running,
can I know why the docker not found ? is my docker-compose wrong ?
You do not need to bind - /usr/bin/docker:/usr/bin/docker, as - /var/run/docker.sock:/var/run/docker.sock is engough to interact with host docker. you should not bind executable with docker container
remove this from the compose file and it should work.
- /usr/bin/docker:/usr/bin/docker
The problem
I have made a project with docker compose. It works well on localhost. I want to use this base to test or analyze code with Gitlab Runner. I solved a lot of problems, like install docker compose, run and build selected containers and run commands in container. The first job ran and success (!!!), but the following jobs failed before "before_script":
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
...
Error response from daemon: Conflict.
...
Error response from daemon: Conflict.
I don't understand why. What do I do wrong? I repeat: the first job of the pipeline runs well with "success" message! Each other jobs of the pipeline fail.
Full output:
Running with gitlab-ci-multi-runner 9.4.0 (ef0b1a6)
on XXX Runner (fdc0d656)
Using Docker executor with image docker:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image docker:dind ID=sha256:5096e5a0cba00693905879b09e24a487dc244b56e8e15349fd5b71b432c6ec9ffor docker service...
ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Will be retried in 3s ...
Using Docker executor with image docker:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image docker:dind ID=sha256:5096e5a0cba00693905879b09e24a487dc244b56e8e15349fd5b71b432c6ec9f for docker service...
ERROR: Preparation failed: Error response from daemon: Conflict. The container name "/runner-fdc0d656-project-35-concurrent-0-docker" is already in use by container "80918876ffe53e33ce1f069e6e545f03a15469af6596852457f11dbc7a6c5b58". You have to remove (or rename) that container to be able to reuse that name.
Will be retried in 3s ...
Using Docker executor with image docker:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image docker:dind ID=sha256:5096e5a0cba00693905879b09e24a487dc244b56e8e15349fd5b71b432c6ec9f for docker service...
ERROR: Preparation failed: Error response from daemon: Conflict. The container name "/runner-fdc0d656-project-35-concurrent-0-docker" is already in use by container "80918876ffe53e33ce1f069e6e545f03a15469af6596852457f11dbc7a6c5b58". You have to remove (or rename) that container to be able to reuse that name.
Will be retried in 3s ...
ERROR: Job failed (system failure): Error response from daemon: Conflict. The container name "/runner-fdc0d656-project-35-concurrent-0-docker" is already in use by container "80918876ffe53e33ce1f069e6e545f03a15469af6596852457f11dbc7a6c5b58". You have to remove (or rename) that container to be able to reuse that name.
Files
.gitlab-ci.yml
# Select image from https://hub.docker.com/r/_/php/
image: docker:latest
# Services
services:
- docker:dind
stages:
- build
- test
- deploy
cache:
key: ${CI_BUILD_REF_NAME}
untracked: true
paths:
- vendor
- var
variables:
DOCKER_CMD: docker exec --user user bin
COMPOSE_HTTP_TIMEOUT: 300
before_script:
- apk add --no-cache py-pip bash
- pip install docker-compose
- touch ~/.gitignore
- bin/docker-init.sh
- cp app/config/parameters.gitlab-ci.yml app/config/parameters.yml
- cp app/config/nodejs_parameters.yml.dist app/config/nodejs_paramteres.yml
- chmod -R 777 app/cache app/logs var
# Load only binary and mysql
- docker-compose up -d binary mysql
build:
stage: build
script:
- ${DOCKER_CMD} composer install -n
- ${DOCKER_CMD} php app/console doctrine:database:create --env=test --if-not-exists
- ${DOCKER_CMD} php app/console doctrine:migrations:migrate --env=test
codeSniffer:
stage: test
script:
- ${DOCKER_CMD} bin/php-cs-fixer fix --dry-run --config-file=.php_cs
database:
stage: test
script:
- ${DOCKER_CMD} php app/console doctrine:mapping:info --env=test
- ${DOCKER_CMD} php app/console doctrine:schema:validate --env=test
- ${DOCKER_CMD} php app/console doctrine:fixtures:load --env=test
unittest:
stage: test
script:
- ${DOCKER_CMD} bin/phpunit -c app --debug
deploy_demo:
stage: deploy
script:
- echo "Deploy to staging server"
environment:
name: staging
url: https://staging.example.com
only:
- develop
deploy_prod:
stage: deploy
script:
- echo "Deploy to production server"
environment:
name: production
url: https://example.com
when: manual
only:
- master
docker-compose.yml
version: "2"
services:
web:
image: nginx:latest
ports:
- "${HTTP_PORT}:80"
depends_on:
- mysql
- elasticsearch
- binary
links:
- binary:php
volumes:
- ".:/var/www"
- "./app/config/docker/vhost.conf:/etc/nginx/conf.d/site.conf"
- "${BASE_LOG_DIR}/nginx:/var/log/nginx"
mysql:
image: mysql:5.6
environment:
MYSQL_USER: test
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: test
ports:
- "${MYSQL_PORT}:3306"
volumes:
- "${BASE_LOG_DIR}/mysql:/var/log/mysql"
- "${BASE_MYSQL_DATA_DIR}:/var/lib/mysql"
- "./app/config/docker/mysql.cnf:/etc/mysql/conf.d/mysql.cnf"
elasticsearch:
image: elasticsearch:1.7.6
ports:
- "${ELASTICSEARCH_PORT}:9200"
volumes:
- "${BASE_ELASTICSEARCH_DATA_DIR}:/usr/share/elasticsearch/data"
binary:
image: fchris82/kunstmaan-test
container_name: bin
volumes:
- ".:/var/www"
- "${BASE_LOG_DIR}/php:/var/log/php"
- "~/.ssh:/home/user/.ssh"
tty: true
environment:
LOCAL_USER_ID: ${LOCAL_USER_ID}
config.toml
[[runners]]
name = "XXX Runner"
url = "https://gitlab.xxx.xx/"
token = "xxxxxxxxxxx"
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = true
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
[runners.cache]
OK, I found the problem. I spoilt the configuration. If you use dind service in .gitlab-ci.yml then don't use /var/run/docker.sock volume in config.toml file OR vica versa if you use "socket" method, don't use the dind service.
More informations: https://docs.gitlab.com/ce/ci/docker/using_docker_build.html