I have two projects, first is SOURCES (technical base) and second is an ant software.
I have create a gitlab-ci.yml on the second project but the build pipeline is always in error because sources libraries are not referenced.
In my build.xml, sources are referenced like this: ${SOURCES_DIRRECTORY}/folder/lib/library.jar
In the gitlab-ci.yml I can set SOURCES_DIRRECTORY but putting the url of the other project doesn't seem to work.
A prohibited solution is to add the sources in an local lib folder. (so as not to duplicate)
Is it possible to reference another gitlab project from my CI ?
Otherwise, how can I retrieve and use the artifact generated by the sources project to obtain the libraries on my software project?
Thank you for your answers
You can use the GitLab CI APIs to download the artifacts made in the other project.
I do something similar in one of my project and it work for me
curl -k --location --output artifacts.zip --header "JOB-TOKEN: $CI_JOB_TOKEN" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/artifacts/devel/download?job=$STAGE_NAME"
In this case i download the artifacts made by another branch in the same project, so I already have setted by default the variables and I can use JOB-TOKEN, instead you should use the PRIVATE-TOKEN that is an ACCESS TOKEN with the API permission grant. You should have something like that
curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/main/download?job=test"
Related
I have some common configurations files that are the same for many projets. It's very painful to manage because they are just duplicate and when I have a modification for one file I need to do it for all of them.
I want to find a solution with gitlab ci/cd to share common configuration files across different projects.
I saw in the Gitlab documentation that you can include other gitlab files here / cd here. But I don't want to share only the yaml configuration file, I want to share all the configuration files that are the same in my projects.
I found an idea but I don't know if it's a good one.
Imagine a project named "A" and a project named "Common Configurations Files".
The "Common Configurations Files" project will contain all the common configurations files in a "config" folder and a gitlab ci/cd file with a function that can copy the files to the root folder.
Project A will include files and will call the before_script step to copy the files to the root folder.
I don't know if it can work and I don't know how to do it.
I saw two other options but I don't know if it's suitable:
Gitlab artifact
git submodule
Both options, either working with a submodule or artifacts, should work for your case. Using submodules would add a little bit more complexity as you would need to keep the submodules updated in all your dependent repositories.
Another option that would be suitable here would be the use of Gitlab Generic Packages. If you have a Repository that includes all your configuration files, I would suggest creating a pipeline for this repository which versions/tags the repo upon changes. After versioning, I would zip the content up and push it to this projects package repository.
curl --header "PRIVATE-TOKEN: <your_access_token>" \
--upload-file path/to/file.txt \
"https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/0.0.1/dependencies.zip"
In all your dependent projects, in the before_script you could just download and extract the zip file.
curl --header "PRIVATE-TOKEN: <your_access_token>" \
"https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/0.0.1/dependencies.zip"
The example in the docs looks like this and might help you get started.
image: curlimages/curl:latest
stages:
- upload
- download
upload:
stage: upload
script:
- 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file path/to/file.txt "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my_package/0.0.1/file.txt"'
download:
stage: download
script:
- 'wget --header="JOB-TOKEN: $CI_JOB_TOKEN" ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my_package/0.0.1/file.txt'
Can some one please provide what is the recommended solution for authenticating inside Dockerfile during docker build phase for Azure Artifacts for Maven, NPM or Python?
While most of the solutions for the particular problem of accessing authenticated feed of Azure Artifacts while building from inside a Dockerfile has been mentioned on Stack Overflow or on the internet (in various articles such as below) have targeted the nuget ecosystem, my question is for Maven, NPM or Python Packages.
Example blogs that provide solutions for nuget & docker-
https://blog.ehn.nu/2019/05/accessing-azure-artifacts-feed-in-a-docker-build/
Appreciate the help here.
My question is for Maven, NPM or Python Packages.
These package types are from different systems, so we should use different ways to authenticate them.
For Maven:
Check set-up-your-feed
. You should configure your pom.xml file to define the url, and then define settings.xml file with PAT. You can manage these two files in your project, then use Dockerfile to copy the settings.xml to ${user.home}/.m2 folder.
For NPM:
You can follow Other topic here to configure one .npmrc which contains credentials. Also I think you can create a normal .npmrc file, and then use Npm Authenticate task to modify it. Here's one working sample.
For Python:
We can use the Pip Authenticate task to populates the PIP_EXTRA_INDEX_URL environment variable and pass it as build arguments. Something like this: arguments: --build-arg INDEX_URL=$(PIP_EXTRA_INDEX_URL)
You can check the two answers from this similar issue for more details.
I want to update Jenkins plugin via Artifactory.
Create a remote repo named Jenkins-update
Create a local repo named jenkins-update-center
Get the update-center.json from repo Jenkins-update to local and modify the URL from 'http://updates.jenkins-ci.org/' to my own URL 'https://artifacts.xxx.com/artifactory/Jenkins-update/' in update-center.json, then put update-center.json into local repo.
#!/bin/sh
curl -L -o /tmp/update-center.json http://localhost:8081/artifactory/Jenkins-update-cache/update-center.json
sed -i 's#http://updates.jenkins-ci.org/#https://artifacts.xxx.com/artifactory/Jenkins-update/#g' /tmp/update-center.json
curl -L -uuser:pass -T /tmp/update-center.json "http://localhost:8081/artifactory/jenkins-update-center/update-center.json"
Change the default update site from 'http://updates.jenkins-ci.org/' to 'https://artifacts.xxx.com/artifactory/jenkins-update-center/update-center.json' in Jenkins
There is an error 'SHA-512 digest mismatch: expected=49a22dc23f739a76623d10128b6803f79e0489de3ded0f1d01f3dfba4557136c7f318baaf4749a7713ec4b3f56633f2ac3afc4703e87d423ede029d68f84c74d in 'update site 'default''' when I click 'check now' button.
What should I do to make Jenkins update plugins from Artifactory?
Tkx
As soon as the content of update-center.json changed you need to re-generate "signature" section of this file.
For that you need to generate your key pair (see more details in How to create a local mirror of public Jenkins update site?)
Also you may use the following proposed approach :
there is probably a better way, by having a sandbox Jenkins on a system that has access to the internet. You update the server using the UI and then you can test that updated Jenkins thoroughly. When done, you just need to copy the war and hpi files over to your 'production' Jenkins. now you have even a nice process and QA in place.
Another way is to setup a transparent https proxy between your Jenkins and Artifactory server - in that case update-center.json will not change and signature verification should work fine.
With best regards,
Dmytro Gorbunov
As of 2023-01-10 there is a problem with making a mirror of the jenkins plugins on artifactory.
Artifactory documentation decribes only how to create a mirror: https://jfrog.com/knowledge-base/how-to-configure-artifactory-as-a-mirror-for-jenkins-plugins/
But this is not a complete solution. Because this leads to the situation when every plugin shall be manually updated. Having plugins with bunch of dependencies it is huge effort.
There is a need to generate a file: update-center.json
There is an internal jenkins tool to do this: https://github.com/jenkins-infra/update-center2, but documentation is poor and contains vague statements like:
With a few modifications it could easily be used to generate your corporate update center as well.
Without clear description, what shall be done.
I tried to follow steps and completely failed. Tool require some special environment variables, which are also not documented and so on.
So as of my experience mirroring jenkins plugins on artifactory is practically not possible. And honestly spoken, I would like to be wrong here.
I would like to know, how can I install a plugin to Jenkins, using the Jenkins Remote
access API?
I found a way to install using jenkins CLI. But I need to know how to do the same using API.
I tried using jenkins-python library. But I did not find any way to
install plugin there.
Send (HTTP POST) the following xml data (with your plugin-id#version) to Jenkins plugin manager. Check out my jenkins install plugin script on gist.
This HTTP POST request install jenkins git plugin 2.0.
curl -X POST -d '<jenkins><install plugin="git#2.0" /></jenkins>' --header 'Content-Type: text/xml' http://localhost:8080/pluginManager/installNecessaryPlugins
Some plugins are hard to update on the file system because others depend on it (credentials is one example). For such plugins it is only possible to update them using the web interface.
Jenkins frontend has a page under 'Manage Jenkins' -> 'Manage Plugins'. Under the 'Advanced' tab is a form to 'uploadPlugin'. It allows web automation with curl, you might need to add authentication.
curl -i -F file=#pluginfilename.hpi http://jenkinshost/jenkins/pluginManager/uploadPlugin
In addition to the methods already mentioned (I personally used the "curl uploadPlugin" one provided by #bbaassssiiee), you need to consider that if you use pluginManager Jenkins will try to load your plugin dinamically, but in case you need to restart Jenkins to initialize the plugin properly (this was my case), you should add:
curl -kX POST https://${JENKINS_URL}/safeRestart
In case you copy the plugin directly to jenkins/plugin, the restart is mandatory for the plugin to be loaded.
As suggested by malenkiy_scot, we can create a job and use the Jenkins CLI. Here is the secret way I do for my automation in installing plugins. Jenkins plugins are available in the Jenkins mirror here: http://updates.jenkins-ci.org/latest This link might not list anything but you can download the plugin if you know the name of the plugin. For example, if you want to download the skype-notifier plugin, you can download it from http://updates.jenkins-ci.org/latest/skype-notifier.hpi The generic URL is "http://updates.jenkins-ci.org/latest/.hpi"
After downloading that plugin, it should go to the "plugins" directory in Jenkins home on the server. For linux machine, it will most likely be in "/var/lib/jenkins/plugins". Simple example
wget http://updates.jenkins-ci.org/latest/skype-notifier.hpi
mv skype-notifier.hpi /var/lib/jenkins/plugins
There are two things to note here:
If the plugin has any dependencies, those will not be installed by default. If you know what other plugins are required, those can be installed the same way. A bit of manual process is required here. But if a same set of plugins are required, the dependency can be resolved just once and script can be written to download and move them to the Jenkins home.
Downloaded plugins cannot be used right away. A reload of Jenkins is required.
After a lot of blood sweat and tears my suggested solution is:
Download the hpi files (plugin and dependencies) using plugin-installation-manager-tool
(requires java) or install-plugins.sh (requires bash only, but is officially deprecated, though still working 09/2021)
Note: Both are also contained in official docker image (see also Offline Installations)
Then install all downloaded files via
curl -i -F file=#plugin.hpi http://${JENKINS_URL}/pluginManager/uploadPlugin
Why?
POSTing to /pluginManager/installNecessaryPlugins always installs latest version (known bug or feature?) and seems to only install the requested plugin without proper dependency handling.
Simple example
Requires install-plugins.sh and its dependency jenkins-support from jenkinsci/docker.
You have do adapt install-plugins.sh line 27 to point to your jenkins-support file, e.g.
. jenkins-support if you have everything in one folder and execute it from there.
pluginFolder=$(mktemp -d)
# Download plugins
JENKINS_UC=https://updates.jenkins.io REF="${pluginFolder}" \
install-plugins.sh \
docker-workflow:1.26 docker-plugin:1.2.2
# add more plugins in here, pass a bash array or load from file
# (see Real-life example bellow)
# Install all downloaded plugin files via HTTP
for pluginFile in "${pluginFolder}/plugins"/*; do
curl -i -F "file=#${pluginFile}" http://${JENKINS_URL}/pluginManager/uploadPlugin
done
Real-life example
Taken from cloudogu/gitops-playground.
download-plugins.sh - loads all plugins declared in plugins.txt using install-plugins.sh to a directory passed as parameter.
init-jenkins.sh calls download-plugins.sh, then installs the plugins using jenkins-REST-client.sh
I do not think this is possible. However, as a workaround you may consider creating a job that would install plugins via Jenkins CLI; you then can invoke that job via the API with appropriate parameters.
I would like to know, how can I install a plugin to Jenkins, using the Jenkins Remote
access API?
I found a way to install using jenkins CLI. But I need to know how to do the same using API.
I tried using jenkins-python library. But I did not find any way to
install plugin there.
Send (HTTP POST) the following xml data (with your plugin-id#version) to Jenkins plugin manager. Check out my jenkins install plugin script on gist.
This HTTP POST request install jenkins git plugin 2.0.
curl -X POST -d '<jenkins><install plugin="git#2.0" /></jenkins>' --header 'Content-Type: text/xml' http://localhost:8080/pluginManager/installNecessaryPlugins
Some plugins are hard to update on the file system because others depend on it (credentials is one example). For such plugins it is only possible to update them using the web interface.
Jenkins frontend has a page under 'Manage Jenkins' -> 'Manage Plugins'. Under the 'Advanced' tab is a form to 'uploadPlugin'. It allows web automation with curl, you might need to add authentication.
curl -i -F file=#pluginfilename.hpi http://jenkinshost/jenkins/pluginManager/uploadPlugin
In addition to the methods already mentioned (I personally used the "curl uploadPlugin" one provided by #bbaassssiiee), you need to consider that if you use pluginManager Jenkins will try to load your plugin dinamically, but in case you need to restart Jenkins to initialize the plugin properly (this was my case), you should add:
curl -kX POST https://${JENKINS_URL}/safeRestart
In case you copy the plugin directly to jenkins/plugin, the restart is mandatory for the plugin to be loaded.
As suggested by malenkiy_scot, we can create a job and use the Jenkins CLI. Here is the secret way I do for my automation in installing plugins. Jenkins plugins are available in the Jenkins mirror here: http://updates.jenkins-ci.org/latest This link might not list anything but you can download the plugin if you know the name of the plugin. For example, if you want to download the skype-notifier plugin, you can download it from http://updates.jenkins-ci.org/latest/skype-notifier.hpi The generic URL is "http://updates.jenkins-ci.org/latest/.hpi"
After downloading that plugin, it should go to the "plugins" directory in Jenkins home on the server. For linux machine, it will most likely be in "/var/lib/jenkins/plugins". Simple example
wget http://updates.jenkins-ci.org/latest/skype-notifier.hpi
mv skype-notifier.hpi /var/lib/jenkins/plugins
There are two things to note here:
If the plugin has any dependencies, those will not be installed by default. If you know what other plugins are required, those can be installed the same way. A bit of manual process is required here. But if a same set of plugins are required, the dependency can be resolved just once and script can be written to download and move them to the Jenkins home.
Downloaded plugins cannot be used right away. A reload of Jenkins is required.
After a lot of blood sweat and tears my suggested solution is:
Download the hpi files (plugin and dependencies) using plugin-installation-manager-tool
(requires java) or install-plugins.sh (requires bash only, but is officially deprecated, though still working 09/2021)
Note: Both are also contained in official docker image (see also Offline Installations)
Then install all downloaded files via
curl -i -F file=#plugin.hpi http://${JENKINS_URL}/pluginManager/uploadPlugin
Why?
POSTing to /pluginManager/installNecessaryPlugins always installs latest version (known bug or feature?) and seems to only install the requested plugin without proper dependency handling.
Simple example
Requires install-plugins.sh and its dependency jenkins-support from jenkinsci/docker.
You have do adapt install-plugins.sh line 27 to point to your jenkins-support file, e.g.
. jenkins-support if you have everything in one folder and execute it from there.
pluginFolder=$(mktemp -d)
# Download plugins
JENKINS_UC=https://updates.jenkins.io REF="${pluginFolder}" \
install-plugins.sh \
docker-workflow:1.26 docker-plugin:1.2.2
# add more plugins in here, pass a bash array or load from file
# (see Real-life example bellow)
# Install all downloaded plugin files via HTTP
for pluginFile in "${pluginFolder}/plugins"/*; do
curl -i -F "file=#${pluginFile}" http://${JENKINS_URL}/pluginManager/uploadPlugin
done
Real-life example
Taken from cloudogu/gitops-playground.
download-plugins.sh - loads all plugins declared in plugins.txt using install-plugins.sh to a directory passed as parameter.
init-jenkins.sh calls download-plugins.sh, then installs the plugins using jenkins-REST-client.sh
I do not think this is possible. However, as a workaround you may consider creating a job that would install plugins via Jenkins CLI; you then can invoke that job via the API with appropriate parameters.