I want to package my Maven/Java app in a Docker Gitlab CI runner.
I'm behind a corporation proxy. This is my .gitlab-ci.yml:
image: maven:3-jdk-7
build:
script: "mvn clean package -B"
When a build is triggered, I get this error (in the Gitlab build console):
Unknown host repo.maven.apache.org: Name or service not known -> [Help 1]
Then, I have added
variables:
http_proxy: http://user:pass#corp.proxy.ip:port
to the .gitlab-ci.yml. But I get another error:
fatal: unable to access
'http://gitlab-ci-token:xxxxxx#170.20.20.20:8080/myapp.git/':
The requested URL returned error: 504
When I registered the Docker runner, Docker image selected was maven:3-jdk-7.
I have just tried adding no_proxy variable with 172.20.20.20 as value (Gitlab IP) but I get the same error (the first one.)
How can I solve it? Is there a way to force the Docker runner (container) to use --net=host?
What I did was I had to open up the mvnw document. Inside of it I found this line
MAVEN_OPTS="$(concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config") $MAVEN_OPTS"
In between the ')' and the $MAVEN_OPTS I placed the
-Dhttps.proxyHost=yourHost -Dhttps.proxyPort=yourPort
arguments. This worked for me. Hope this helps. I didn't need the "variables" section you described above.
Related
I'm attempting to POC BitBucket Pipelines for some terraform work. I've got a self-hosted runner, running locally in my Docker environment, which is registered to my repository. This was set up following the generic instructions in the BitBucket UI.
My bitbucket-pipelines.yml file looks like this:
pipelines:
branches:
master:
- step:
runs-on: self.hosted
image: hashicorp/terraform:latest
name: 'Terraform Version'
script:
- terraform -v
Extremely basic, just run a terraform -v command on the hashicorp/terraform image.
The pipeline succeeds, and I can see the image is pulled, however there is absolutely no output in BitBucket from the container. All I see in the step log is:
Runner matching labels:
- linux
- self.hosted
Runner name: my-runner
Runner labels: self.hosted, linux
Runner version:
current: 1.252
latest: 1.252
Images used:
build: hashicorp/terraform#sha256:984ac701744995019b1309b542de03535a63097444e72b8f248d0a0d95520443
Even a simple echo "string" script does not get to the logs as output. I find that really strange, and I must be missing something fundamental. I've scoured the docs and can't find anything.
Does anyone know how to get the output from a custom image into the Bitbucket logs?
Do you use Docker Desktop on Windows?
You won't see any logs from containers if you use DockerDesktop (tested on 4.3.2) on Windows with WSL integration. That's due to container logs have another location and they're not available to bitbucket runner container.
-- Update --
There's a feature request to add local runners WSL full compatibility now. Pls vote if you need it too.
https://jira.atlassian.com/browse/BCLOUD-21611
I had a similar issue where I was getting no logs in my Pipeline output UI, though the ultimate status was reflected correctly (i.e. pass or fail).
I was using the command provided by Bitbucket to create a Linux Docker runner, and I noticed it contains this volume definition:
... -v /var/lib/docker/containers:/var/lib/docker/containers:ro ...
However, I am using a custom data-root for docker (see this blog for details), so the path /var/lib/docker/containers doesn't exist on my host machine. So, I modified this volume to point at my data-root setting, and then the logs showed up as expected.
We want to use Paketo.io / CloudNativeBuildpacks (CNB) GitLab CI in the most simple way. Our GitLab setup uses an AWS EKS cluster with unprivileged GitLab CI Runners leveraging the Kubernetes executor. We also don't want to introduce security risks by using Docker in our builds. So we don't have our host’s /var/run/docker.sock exposed nor want to use docker:dind.
We found some guides on how to use Paketo with GitLab CI like this https://tanzu.vmware.com/developer/guides/gitlab-ci-cd-cnb/ . But as described beneath the headline Use Cloud Native Buildpacks with GitLab in GitLab Build Job WITHOUT Using the GitLab Build Template, the approach relies on Docker and pack CLI. We tried to resemble this in our .gitlab-ci.yml which looks like this:
image: docker:20.10.9
stages:
- build
before_script:
- |
echo "install pack CLI (see https://buildpacks.io/docs/tools/pack/)"
apk add --no-cache curl
(curl -sSL "https://github.com/buildpacks/pack/releases/download/v0.21.1/pack-v0.21.1-linux.tgz" | tar -C /usr/local/bin/ --no-same-owner -xzv pack)
build-image:
stage: build
script:
- pack --version
- >
pack build $REGISTRY_GROUP_PROJECT/$CI_PROJECT_NAME:latest
--builder paketobuildpacks/builder:base
--path .
But as outlined our setup does not support docker and we end up with the following error inside our logs:
...
$ echo "install pack CLI (see https://buildpacks.io/docs/tools/pack/)" # collapsed multi-line command
install pack CLI (see https://buildpacks.io/docs/tools/pack/)
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
(1/4) Installing brotli-libs (1.0.9-r5)
(2/4) Installing nghttp2-libs (1.43.0-r0)
(3/4) Installing libcurl (7.79.1-r0)
(4/4) Installing curl (7.79.1-r0)
Executing busybox-1.33.1-r3.trigger
OK: 12 MiB in 26 packages
pack
$ pack --version
0.21.1+git-e09e397.build-2823
$ pack build $REGISTRY_GROUP_PROJECT/$CI_PROJECT_NAME:latest --builder paketobuildpacks/builder:base --path .
ERROR: failed to build: failed to fetch builder image 'index.docker.io/paketobuildpacks/builder:base': Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: command terminated with exit code 1
Any idea on how to use Paketo Buildpacks with GitLab CI without having Docker present inside our GitLab Kubernetes runners (which seems to be kind of a best practice)? We also don't want our setup to become to complex - e.g. by adding kpack.
TLDR;
Use the Buildpack's lifecycle directly inside your .gitlab-ci.yml here's a fully working example):
image: paketobuildpacks/builder
stages:
- build
# We somehow need to access GitLab Container Registry with the Paketo lifecycle
# So we simply create ~/.docker/config.json as stated in https://stackoverflow.com/a/41710291/4964553
before_script:
- mkdir ~/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" >> ~/.docker/config.json
build-image:
stage: build
script:
- /cnb/lifecycle/creator -app=. $CI_REGISTRY_IMAGE:latest
The details: "using the lifecycle directly"
There are ongoing discussions about this topic. Especially have a look into https://github.com/buildpacks/pack/issues/564 and https://github.com/buildpacks/pack/issues/413#issuecomment-565165832. As stated there:
If you're looking to build images in CI (not locally), I'd encourage
you to use the lifecycle directly for that, so that you don't need
Docker. Here's an example:
The link to the example is broken, but it refers to the Tekton implementation on how to use buildpacks in a Kubernetes environment. Here we can get a first glue about what Stephen Levine referred to as "to use the lifecycle directly". Inside it the crucial point is the usage of command: ["/cnb/lifecycle/creator"]. So this is the lifecycle everyone is talking about! And there's good documentaion about this command that could be found in this CNB RFC.
Choosing a good image: paketobuildpacks/builder:base
So how to develop a working .gitlab-ci.yml? Let's start simple. Digging into the Tekton implementation you'll see that the lifecycle command is executed inside an environment defined in BUILDER_IMAGE, which itself is documented as The image on which builds will run (must include lifecycle and compatible buildpacks). That sound's familiar! Can't we simply pick the builder image paketobuildpacks/builder:base from our pack CLI command? Let's try this locally on our workstation before commiting to much noise into our GitLab. Choose a project you want to build (I created a example Spring Boot app if you'd like at gitlab.com/jonashackt/microservice-api-spring-boot you can clone) and run:
docker run --rm -it -v "$PWD":/usr/src/app -w /usr/src/app paketobuildpacks/builder bash
Now inside the paketobuildpacks/builder image powered container try to run the Paketo lifecycle directly with:
/cnb/lifecycle/creator -app=. microservice-api-spring-boot:latest
I only used the -app parameter of the many possible parameters for the creator command, since most of them have quite good defaults. But as the default app directory path is not the default /workspace - but the current directory, I configured it. Also we need to define an <image-name> at the end, which will simply be used as the resulting container image name.
The first .gitlab-ci.yml
Both commands did work at my local workstation, so let's finally create a .gitlab-ci.yml using this approach (here's a fully working example .gitlab-ci.yml):
image: paketobuildpacks/builder
stages:
- build
build-image:
stage: build
script:
- /cnb/lifecycle/creator -app=. $CI_REGISTRY_IMAGE:latest
docker login without docker
As we don't have docker available inside our Kubernetes Runners, we can't login into GitLab Container Registry as described in the docs. So the following error occured to me using this first approach:
===> ANALYZING
ERROR: failed to get previous image: connect to repo store "gitlab.yourcompanyhere.cloud:4567/yourgroup/microservice-api-spring-boot:latest": GET https://gitlab.yourcompanyhere.cloud/jwt/auth?scope=repository%3Ayourgroup%2Fmicroservice-api-spring-boot%3Apull&service=container_registry: DENIED: access forbidden
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: command terminated with exit code 1
Using the approach described in this so answer fixed the problem. We need to create a ~/.docker/config.json containing the GitLab Container Registry login information - and then the Paketo build will pick them up, as stated in the docs:
If CNB_REGISTRY_AUTH is unset and a docker config.json file is
present, the lifecycle SHOULD use the contents of this file to
authenticate with any matching registry.
Inside our .gitlab-ci.yml this could look like:
# We somehow need to access GitLab Container Registry with the Paketo lifecycle
# So we simply create ~/.docker/config.json as stated in https://stackoverflow.com/a/41710291/4964553
before_script:
- mkdir ~/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" >> ~/.docker/config.json
Our final .gitlab-ci.yml
As we're using the image: paketobuildpacks/builder at the top of our .gitlab-ci.yml, we can now leverage the lifecycle directly. Which is what we wanted to do in the first place. Only remember to use the correct GitLab CI variables to describe your <image-name> like this:
/cnb/lifecycle/creator -app=. $CI_REGISTRY_IMAGE:latest
Otherwise the Buildpack process analyser step will break and it finally won't get pushed to the GitLab Container Registry. So finally our .gitlab-ci.yml looks like this (here's the fully working example):
image: paketobuildpacks/builder
stages:
- build
# We somehow need to access GitLab Container Registry with the Paketo lifecycle
# So we simply create ~/.docker/config.json as stated in https://stackoverflow.com/a/41710291/4964553
before_script:
- mkdir ~/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" >> ~/.docker/config.json
build-image:
stage: build
script:
- /cnb/lifecycle/creator -app=. $CI_REGISTRY_IMAGE:latest
Our builds should now run successfully using Paketo/Buildpacks without pack CLI and Docker:
See the full log of the example project here.
I made a JHipster application, and I want to add CI/CD with a private Gitlab runner to deploy on a private Docker registry. I get this failure:
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.0.0:build (default-cli) on project powerfront: Invalid image reference :master-35274d52bd71e28f08a0428832001cc67e9c446d, perhaps you should check that the reference is formatted correctly according to https://docs.docker.com/engine/reference/commandline/tag/#extended-description
[ERROR] For example, slash-separated name components cannot have uppercase letters: Invalid image reference: :master-35274d52bd71e28f08a0428832001cc67e9c446d
This is the relevant part of my .gitlab-ci.yml
# Uncomment the following line to use gitlabs container registry. You need to adapt the REGISTRY_URL in case you are not using gitlab.com
docker-push:
stage: release
variables:
REGISTRY_URL: 10.1.10.58
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHA
dependencies:
- maven-package
script:
- ./mvnw -ntp compile jib:build -Pprod -Djib.to.image=$IMAGE_TAG -Djib.to.auth.username=gitlab-ci-token -Djib.to.auth.password=$CI_BUILD_TOKEN -Dmaven.repo.local=$MAVEN_USER_HOME
EDIT: There was an unconfigured variable. Now I get
[ERROR] I/O error for image [10.1.10.58:5000/powerfront]:
[ERROR] javax.net.ssl.SSLException
[ERROR] Unrecognized SSL message, plaintext connection?
How do I tell the runner to accept unsecure (plaintext) connections?
To publish to a private (or another public) registry, your image name must start by the host name of the registry : private.registry.io/group/image:version so that Docker daemon knows that it doesn't push to Docker Hub (which the default) but to private.registry.io
Also U can use Kaniko to publish your image, as it doesn't require dind or privileged mode on Docker daemon.
I'm not sure that is a Gitlab CI problem. but with JHipster.
What is the value for CI_REGISTRY_IMAGE? We don't see the value in the error message.
I'm just getting started with docker and continuous integration with Gitlab. I've added the following gitlab-ci.yml file to the root of my repository:
# Official docker image
image: docker:latest
services:
- docker:dind
build-dev:
stage: build
script:
- docker build -t obikerui/project -f app/Dockerfile.dev ./app
test:
stage: test
script:
- docker run obikerui/project npm run test -- --coverage
The build-dev stage runs and passes but the test stage fails with the following error message:
$ docker run obikerui/project npm run test -- --coverage
Unable to find image 'obikerui/project:latest' locally
docker: Error response from daemon: pull access denied for obikerui/project, repository does not exist or may require 'docker login'.
See 'docker run --help'.
ERROR: Job failed: exit code 125
Can anyone explain what's going wrong and suggest a fix? The repository is private, so do I need to provide some extra configuration to accommodate this?
Each job runs in a different container. You build and you tag your image correctly but that stays in that container.
For the test job a new container starts and that one does not have the image build by the previous job.
You should push your image to a registry (after you tag it accordingly) and then the test job should use the image from the repository.
You can use a public registry like the one offered by Docker or you can run a local container based on the image registry:2 provided by docker. In this case you have to make sure that the domain name pointing to the registry is available on your network (it can be an nginx with reverse proxy)
I recently used the docker image gitlab/gitlab-runner:9.1.0 in conjunction with a gitlab container to have some CI.
An error occurs and similar support requests recommended using a different version, so I tried with :latest and some :1.11 too.
Unfortunately it keeps telling me this error:
Running with gitlab-ci-multi-runner 1.11.4 (5e7ba4a)
on foo (02cdacdc)
Using Docker executor with image pretzlaw/php:7.1-apache ...
Starting service mariadb:latest ...
Pulling docker image mariadb:latest ...
Waiting for services to be up and running...
Pulling docker image pretzlaw/php:7.1-apache ...
Running on runner-02cdacdc-project-7-concurrent-0 via 9d1d33dc9212...
Fetching changes...
HEAD is now at 7580815 QA: CI Lint
From http://idgaf.example.org/foo/bar
7580815..affeede develop -> origin/develop
Checking out affeede as develop...
Skipping Git submodules setup
[: 1: [: Syntax error: end of file unexpected
[: 1: [: Syntax error: end of file unexpected
ERROR: Job failed: exit code 2
Neither do I know how to debug this nor do I see any problem in my container or test script. This is the .gitlab-ci.yml:
before_script:
- composer install
test_7_1:
image: pretzlaw/php:7.1-apache
script: ls
It could be a problem of the container somewhere but I don't get it. Doing this manually (with the recent failed docker container) everything works fine:
docker container exec 68c7b5448a56 ls
bin
builds
...
How do I trace back the problem?
What is it all about?
It's for GitLab 9.1.1-ce.0.
As pointed out by #1550, the problem seems to be coming from the shell detection done in the bash.go file between lines 16-31 which it is injected without newlines and thus creating the syntax error you are yourself experiencing.
Since you have a custom entrypoint in your Dockerfile and it looks like you have not passed quotes to the arguments of the exec here, that's what I think is failing and should be modified.
Change it to
exec "${*}"
As a workaround to the bug mentioned by Balthazar, try overriding the entrypoint in .gitlab-ci.yml as illustrated in the GitLab docs:
For Docker 17.06 and later:
image:
name: super/sql:experimental
entrypoint: [""]
For Docker 17.03 and earlier:
image:
name: super/sql:experimental
entrypoint: ["/bin/sh", "-c"]