GITLAB CI dependency:get does not find pom file cause of TIMESTAMPLE - docker

I deploy maven packages on gitlab artifact. Maven called them xxx-api-model-2.0.2-20221223.095542-17.pom
My mvn dependency:get from docker file in another folder search for xxx-api-model-2.0.2.pom
Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.2.0:get (default-cli) on project xxx-action-batch: Couldn't download artifact: org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact xxx:xxx-api-model:jar:2.0.2 in gitlab-maven
Here is settings file:
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
https://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server> <!-- serveur Gitlab :
<id>gitlab-maven</id>
<username>${GITLAB_REPO_USER}</username>
<password>${GITLAB_REPO_USER}</password>
<configuration>
<httpHeaders>
<property>
<name>Job-Token</name>
<value>${env.CI_JOB_TOKEN}</value>
</property>
</httpHeaders>
</configuration>
</server>
</servers>
</settings>
pom file :
<repositories>
<repository>
<id>gitlab-maven</id>
<url>${GITLAB_REPO_URL}</url>
</repository>
</repositories>
<distributionManagement>
<repository>
<id>gitlab-maven</id>
<url>${GITLAB_REPO_URL}</url>
</repository>
<snapshotRepository>
<id>gitlab-maven</id>
<url>${GITLAB_REPO_URL}</url>
</snapshotRepository>
</distributionManagement>
GITLAB_REPO_URL = https://example.com/api/v4/projects/2/packages/maven
dockerfile :
FROM eclipse-temurin:11-jre-focal as base
FROM maven:3.6.2-jdk-11 as build
WORKDIR /app
COPY pom.xml .
COPY src src
COPY .m2 .m2
ARG GITLAB_REPO_URL
ARG GITLAB_REPO_USER
ARG GITLAB_REPO_PASS
ENV GITLAB_REPO_USER=$GITLAB_REPO_USER
ENV GITLAB_REPO_PASS=$GITLAB_REPO_PASS
RUN bash -c "mvn dependency:get -Dartifact=xxx:xxx-api-model:2.0.2 -DremoteRepositories=gitlab-maven::::$GITLAB_REPO_URL -DGITLAB_REPO_USER=$GITLAB_REPO_USER -DGITLAB_REPO_PASS=$GITLAB_REPO_PASS"
RUN bash -c "mvn -B -f pom.xml -s .m2/settings.xml install -DskipTests -DGITLAB_REPO_URL=$GITLAB_REPO_URL -DGITLAB_REPO_USER=$GITLAB_REPO_USER -DGITLAB_REPO_PASS=$GITLAB_REPO_PASS"
WORKDIR /workspace/app/target/
ENTRYPOINT exec java $JAVA_OPTS -jar xxx-action-batch-1.0.0.jar
ci :
stages:
- action-batch-build&push
build&push:action-batch:
stage: action-batch-build&push
image: docker:stable
services:
- docker:19.03-dind
script:
- >
docker build
--pull
--build-arg VCS_REF=$CI_COMMIT_SHA
--build-arg VCS_URL=$CI_PROJECT_URL
--build-arg GITLAB_REPO_URL=${GITLAB_REPO_URL}
--build-arg GITLAB_REPO_USER=${GITLAB_REPO_USER}
--build-arg GITLAB_REPO_PASS=${GITLAB_REPO_PASS}
--tag xxx-action-batch:$CI_COMMIT_SHA
action-batch/
allow_failure: false
I tried to put some
-DuniqueVersion=false
to solve the problem with names of pom and jar file put nothing change

Related

Openshift Docker Strategy - not able to pull maven artifacts from private repo

In OpenShift i am using s2i( docker strategy) .
I have configured two repo's public and private in pom.xml .
maven is able to pull the artifacts from the public repo , But it could not pull the application specific artifact from the private repo.
Is there a way to update the maven setting.xml inside the openshift by providing maven repo credentials. how to provide credentials to pull the artifacts from private maven repo configured.
Error Below
Downloading from central: https://artifactory.mycomp.com/artifactory/java-snapshot-local/com/test/common-security/1.0.0/common-security-1.0.0.pom
Failed to execute goal on project demo-app: Could not resolve dependencies for project com.example:demo-app:jar:0.0.1-SNAPSHOT: Failed to collect dependencies at com.mycomp:common-security:jar:1.0.0: Failed to read artifact descriptor for
com.mycomp:common-security:jar:1.0.0: Could not transfer artifact com.mycomp:common-security:pom:1.0.0 from/to central (https://artifactory.mycomp.com/artifactory/java-snapshot-local): authentication failed for https://artifactory.mycomp.com/artifactory/java-snapshot-local/com/test/common-security/1.0.0/common-security-1.0.0.pom,
status: 401 Unauthorized -> [Help 1]
[ERROR]
The below one worked for me but not sure this one is ideal. Add the settings.xml to the root of the source:
<settings>
<servers>
<server>
<id>${repo.id}</id>
<username>${repo.login}</username>
<password>${repo.pwd}</password>
</server>
</servers>
</settings>
Inside the docker, use can use the settings.xml as an argument during build and supply the credentials.
# Build the application first using Maven
FROM maven:3.8-openjdk-11 as build
WORKDIR /app
COPY . .
COPY settings.xml /usr/share/
RUN mvn --version
RUN mvn --settings /usr/share/settings.xml -Drepo.id=central -Drepo.login=Myname -Drepo.pwd=MyPass clean install -Dmaven.test.skip=true
# Inject the JAR file into a new container to keep the file small
FROM openjdk:8-jre-alpine
WORKDIR /app
COPY --from=build /app/target/demo-app*.jar /app/app.jar
EXPOSE 8080
ENTRYPOINT ["sh", "-c"]
CMD ["java -jar app.jar"]

Gitlab: How to pass CI_JOB_TOKEN to Docker to be used with maven?

I have a ci_settings.xml. With it im building artifacts and deploy them to the
gitlab package repository:
<settings xmlns="http://maven.apache.org/SETTINGS/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
<servers>
<server>
<id>gitlab-maven</id>
<configuration>
<httpHeaders>
<property>
<name>Job-Token</name>
<value>${env.CI_JOB_TOKEN}</value>
</property>
</httpHeaders>
</configuration>
</server>
</servers>
</settings>
But if I also want to be able do this in a docker container I need to pass the CI_JOB_TOKEN
to the docker container. Yet I have not been able to find out how to do that.
My dockerfile:
FROM maven:3.8.1-jdk-11
COPY pom.xml /
COPY ci_settings.xml /build/
COPY lombok.config /build/
COPY app/pom.xml /build/
COPY app/src /build/src
WORKDIR /build/
ARG CI_JOB_TOKEN
ENV CI_JOB_TOKEN=$CI_JOB_TOKEN
RUN mvn package -B -s ci_settings.xml
RUN cp /build/target/*.jar /app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]
My gitlab-ci.yml:
stages:
- build
variables:
IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG # TODO semantic versioning
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
build:
stage: build
# variables:
# CI_JOB_TOKEN: "${CI_JOB_TOKEN}" # does not work
script:
# - echo "CI_JOB_TOKEN=$CI_JOB_TOKEN" >> .env # does not work
- docker info
# setting it as build-arg also does not work
- docker build --build-arg CI_JOB_TOKEN=${CI_JOB_TOKEN} -f app/Dockerfile --pull -t $IMAGE .
- docker push $IMAGE
What do I need to do to pass the CI_JOB_TOKEN to the docker container so that
${env.CI_JOB_TOKEN} inside the ci_settings.xml which I copy to the container
is getting resolved correctly?
Use something like
RUN sed -i "s/CI_JOB_TOKEN/$CI_JOB_TOKEN/" ci_settings.xml
after the file is copied, and replace ${env.CI_JOB_TOKEN} by CI_JOB_TOKEN in your xml file to simplify the substitution.

Getting an error "COPY failed: stat /home/app/target/Service.jar : file does not exist" in multistage docker build

My Docker File is,
FROM maven:3.6.0-jdk-11-slim AS build
COPY src /home/app/src
COPY pom.xml /home/app
RUN mvn -f /home/app/pom.xml clean package
FROM openjdk:11-jre-slim
COPY --from=build /home/app/target/Service.jar /usr/local/lib/Service.jar
EXPOSE 8081
ENTRYPOINT ["java","-jar","/usr/local/lib/service.jar"]
When executing COPY --from line, getting an error "COPY failed: stat /home/app/target/Service.jar : file does not exist".
Note: when we execute "RUN ls /home/app" command, its listing Files and target directory .
Could someone help on this.
Do you have a finalName defined in your pom?
<build>
<finalName>${project.artifactId}</finalName>
<plugins>
...
if not the resulting artifact will have versioning in the name and will not be called Service.jar but something like Service-1.0-SNAPSHOT.jar
FROM maven:3.6.0-jdk-11-slim AS build
COPY src /home/app/src
COPY pom.xml /home/app/pom.xml
WORKDIR /home/app
RUN mvn clean package
FROM openjdk:11-jre-slim
COPY --from=build /home/app/target/Service_AND_VERSION_STUFF.jar /usr/local/lib/Service.jar
EXPOSE 8081
ENTRYPOINT ["java","-jar","/usr/local/lib/Service.jar"]
don't forget to always use the same casing (Sevice != service)

Can't connect to the docker daemon while running Maven jib plugin in Gitlab CI

I'm trying to build a docker image with my application in Gitlab continuous integration using the jib Maven plugin, but I can't get the plugin connect to the docker daemon using the dind (docker-in-docker) service. Currently I'm using this configuration in my gitlab-ci.yml file:
deploy:mvn:
image: maven:3.6.3-jdk-8-openj9
stage: deploy
services:
- docker:dind
script:
- mvn compile jib:dockerBuild
This is the error I get:
[ERROR] Failed to execute goal
com.google.cloud.tools:jib-maven-plugin:0.9.11:dockerBuild
(default-cli) on project my-application: Build to Docker daemon failed,
perhaps you should make sure Docker is installed and you have correct
privileges to run it -> [Help 1]
UPDATE
I updated to 2.2.0 and it's running locally. I added the docker images command before and the plugin seems not to be able to find the docker command in Gitlab CI:
$ docker images && mvn compile jib:dockerBuild /bin/bash: line 97:
docker: command not found
This is the configuration for the jib plugin:
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<version>2.2.0</version>
<configuration>
<from>
<image>adoptopenjdk/openjdk11:alpine-jre</image>
</from>
<to>
<image>my-application:latest</image>
</to>
<container>
<entrypoint>
<shell>sh</shell>
<option>-c</option>
<arg>chmod +x /entrypoint.sh && sync && /entrypoint.sh</arg>
</entrypoint>
<ports>
<port>8080</port>
</ports>
<environment>
<SPRING_OUTPUT_ANSI_ENABLED>ALWAYS</SPRING_OUTPUT_ANSI_ENABLED>
<JHIPSTER_SLEEP>0</JHIPSTER_SLEEP>
</environment>
<creationTime>USE_CURRENT_TIMESTAMP</creationTime>
</container>
</configuration>
</plugin>
the plugin seems not to be able to find the docker command in Gitlab CI:
No, it is not Jib but /bin/bash that cannot find the docker command. Even before using Jib, you don't have docker available. Take a closer look at the error message.
$ docker images && mvn compile jib:dockerBuild /bin/bash: line 97: docker: command not found
For example, on my Linux, if I try a command foo that doesn't exist in a shell script, it outputs the same message.
$ /bin/bash -c "foo && mvn -v"
/bin/bash: line 1: foo: command not found
Therefore, the following command without mvn will fail with the same error.
script:
- docker images
This proves that either docker doesn't exist in your GitLab runtime environment or is not on the PATH environment variable.
UPDATE
Updating my answer, since you answered that you now use jib:build instead of jib:dockerBuild.
If you use jib:build, you don't even need Docker. Jib doesn't require Docker when building and pushing an image to a remote registry with jib:build. Therefore, you can completely forget about setting up Docker and remove docker:dind and export DOCKER_HOST:
mvn compile jib:build -Djib.to.auth.username=$DOCKER_REGISTRY_USER -Djib.to.auth.password=$DOCKER_REGISTRY_PWD
Finally this is the configuration I got it working with:
services:
- docker:dind
deploy:mvn:
image: maven:3.6.3-jdk-8-openj9
stage: deploy
script:
- export DOCKER_HOST=tcp://docker:2375
- mvn compile jib:build -Djib.to.auth.username=$DOCKER_REGISTRY_USER -Djib.to.auth.password=$DOCKER_REGISTRY_PWD
only:
- tags
Apart from using the Docker in Docker service I needed to establish the DOCKER_HOST environment variable and also pass the credentials to my mvn jib:build command. I stored the credentials in CI settings, as environment variables. Thanks #Chanseok Oh for your help.
See also:
Using docker build
Jib docs

Azure DevOps private nuget repo Unauthorized when building with docker

I am trying to create a docker image for my project that has a private Azure DevOps nuget repo. I am able to connect to the repo in visual studio with my credentials but when I try to build a docker image I get an error.
Unable to load the service index for source
https://{my_url}/nuget/v3/index.json Response status code does not
indicate success: 401 (Unauthorized)
I copied my Nuget.config file to the root of my project
nuget.config
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<disabledPackageSources>
<add key="Microsoft and .NET" value="true" />
</disabledPackageSources>
<activePackageSource>
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
</activePackageSource>
<packageSources>
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
<add key="libraries" value="https://{my_url}/nuget/v3/index.json" />
</packageSources>
<packageSourceCredentials>
<libraries>
<add key="Username" value="{my_username}" />
<add key="ClearTextPassword" value="{my_pass}" />
</libraries>
</packageSourceCredentials>
</configuration>
and this is my dockerfile
FROM mcr.microsoft.com/dotnet/core/aspnet:3.0.0-preview8-nanoserver-1903 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/core/sdk:3.0.100-preview8-nanoserver-1903 AS build
WORKDIR /src
COPY NuGet.config "MyProject/"
COPY ["MyProject/MyProject.csproj", "MyProject/"]
COPY . .
RUN dotnet restore "MyProject/MyProject.csproj"
WORKDIR "/src/MyProject"
RUN dotnet build "MyProject.csproj" -c Release -o /app
FROM build AS publish
RUN dotnet publish "MyProject.csproj" -c Release -o /app
FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "MyProject.dll"]
I am not sure what else to do for it to be able to connect to my repo
I stumbled across this post.
(ref: solution # the bottom)
Basically, change the PAT Organization to "All accessible organizations" AND scroll through the permission\security sections and set the following:
Build (Artifacts, definitions, requests, queue a build, and updated build properties): Read
Connected server (Access endpoints): Connected server
Packaging (Create, read, update, and delete feeds and packages): Read
(Not sure if all of these are required.)
Docker File:
FROM mcr.microsoft.com/dotnet/core/aspnet:2.2-stretch-slim AS base
WORKDIR /app
FROM mcr.microsoft.com/dotnet/core/sdk:2.2-stretch AS build
WORKDIR /src
# The Personal Access Token arg
ARG PAT
# Set environment variables
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS '{"endpointCredentials":[{"endpoint":"https://pkgs.dev.azure.com/ORG/_packaging/FEEDNAME/nuget/v3/index.json","username":"USERNAME","password":"'${PAT}'"}]}'
# Get and install the Artifact Credential provider
RUN wget -O - https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh | bash
COPY ["PROJECTNAME.csproj", "./"]
RUN dotnet restore -s "https://pkgs.dev.azure.com/ ORG /_packaging/ FEEDNAME /nuget/v3/index.json" -s "https://api.nuget.org/v3/index.json"
COPY . ./
WORKDIR /src
RUN dotnet build " PROJECTNAME.csproj" -c Release -o /app
FROM build AS publish
RUN dotnet publish --no-restore " PROJECTNAME.csproj" -c Release -o /app
FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", " PROJECTNAME.dll"]
docker command:
docker build -t BUILDNAME:local . --build-arg PAT=xxxxxxxxxxxxxxxxxxxxxxxxx
Previously we were using a nuget.config file with credentials\PAT and this was working. I have no idea what changed to cause it to stop. If anyone had an idea pls post.
you need to force dotnet restore to use your config file:
RUN dotnet restore "MyProject/MyProject.csproj" --configfile Nuget.Config
alternatively you can do something like this in your dockerfile:
# Add Environment Variables references for access private Azure Artifacts Repository
# Use --build-arg <ARG> to pass this in.
ARG TOKENPASS
ARG TOKENUSER
ARG NUGET_ENDPOINT
# Fetch Azure Artifacts "Magic" credentials providers
RUN wget -qO- https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh | bash
# Needs to be enabled to fetch the private repository credentials for NuGet restore.
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS="{\"endpointCredentials\": [{\"endpoint\":\"$NUGET_ENDPOINT\", \"username\":\"$TOKENUSER\", \"password\":\"$TOKENPASS\"}]}"
# Workaround of some kind to help Azure Artifact Private repository image collection.
ENV DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER 0
and use docker build with --build-arg TOKENPASS=$(TOKENPASS) --build-arg TOKENUSER=$(TOKENUSER) --build-arg NUGET_ENDPOINT=$(NUGET_ENDPOINT)

Resources