I have a (private) repository at GitHub with my project and integrated GitHub-actions which is building a docker-image and pushing it directly to GHCR.
But I have a problem with storing and passing secrets to my build image. I have the following structure in my project:
.git (gitignored)
.env (gitignored)
config (gitignored) / config files (jsons)
src (git) / other folders and files
As you may see, I have .env file and config folder. Both of them store data or files, which are not in the repo but are required to be in the built environment.
So I'd like to ask, is there any option not to pass all these files to my main remote repo (even if it's private) but to link them during the build stage within the github-actions?
It's not a problem to publish env & configs somewhere else, privately, in another separate private remote-repo. The point is not to push these files to the main-private-repo, because RBAC logic doesn't allow me to restrict access to the selected files.
P.S. Any other advice of using GitLab CI or BitBucket, if you know how to solve the problem is also appreciated. Don't be shy to share it!
So it seems that this question is a bit hot, so I have found an answer for it.
Example that is shown above is based on node.js and nest.js app and pulling the private remote repo from GitHub.
In my case, this scenario was about pulling from separate private repo config files and other secrets. And we merge them with our project during container build. This option isn't about security of secrets inside container itself. But for making one part of a project (repo itself with business logic) available to other developers (they won't see credentionals and configs from separate private repo, in your development repo) and a secret-private repo with separate access permission.
You all need your personal access token (PAT), on github you can found it here:
As for GitLab, the flow is still the same. You'll need to pass token from somewhere in the settings. And also, just a good advice, create not just one, but two docker files, before testing it.
Why https instead of ssh? In that case you'll need also to pass ssh keys and also config the client correctly. It's a bit more complicated because of CRLF and LF formats, crypto-algos supported by ssh and so on.
# it could be Go, PHP, what-ever
FROM node:17
# you will need your GitHub token from settings
# we will pass it to build env via GitHub action
ARG CR_PAT
ENV CR_PAT=$CR_PAT
# update OS in build container
RUN apt-get update
RUN apt-get install -y git
# workdir app, it is a cd (directory)
WORKDIR /usr/src/app
# installing nest library
RUN npm install -g #nestjs/cli
# config git with credentials
# we will use https since it is much easier to config instead of ssh
RUN git config --global url."https://${github_username}:${CR_PAT}#github.com/".insteadOf "https://github.com/"
# cloning the repo to WORKDIR
RUN git clone https://github.com/${github_username}/${repo_name}.git
# we move all files from pulled repo to root of WORKDIR
# including files named with dot at the beginning (like .env)
RUN mv repo_folder/* repo_folder/.[^.]* . && rmdir repo_folder/
# node.js stuff
COPY package.json ./
RUN yarn install
COPY . .
RUN nest build app
CMD wait && ["node"]
As a result, you'll see a fully container with your code merged with files and code from other separate repo which we pull from.
Is there any way to capture the users email address who ran the build and put it into the jenkins yml config file? in other words, can I use a standard JENKINS ENV VAR in my yaml config file that is read into my pipeline?
I am using this plugin: https://engineering.salesforce.com/open-sourcing-the-jenkins-config-driven-pipelines-plugin-9c0becaa5f79
As they use
script_to_run: |-
pip install pipenv
export PATH=$(python -m site --user-base)/bin:$PATH
looks like it's running in bash. You may then try e.g. $JENKINS_HOME or $BRANCH_NAME and see if it works.
I have now searched the Internet for 2 whole days without any progress. There might be so, that the answer is just a click away, but I cannot find it. I'm blinded. I work on Windows 10 by the way...
The problem is that I want to create a simple Dockerfile, that can create a Docker image with a Jenkins master with everything I want it to contain. For instance:
general Jenkins configuration
users
plugins
jobs
slaves
etc...
Now I at least some parts working... I'm able to get rid of the initial "hello welcome to jenkins"-thing, and define my first user + install plugins that i wrote down in a text file.
BUT when it comes to the freakin jenkins jobs it seems to get a bit more complicated.
Since my Dockerfile takes a "base image" from jenkins/jenkins:lts-alpine, it creates a volume for /var/jenkins_home/ which seems to be where the jobs are stored at. And it seems to be a bit problematic to copy files to this folder. I tried add a COPY instruction in my Dockerfile to copy folder & files that are created when I manually create a job in Jenkins, but it seems like Jenkins does not read them for some reason, even after restarting/reloading from disk. The thing is, it is actually working when copying the jobs after "installation". Like docker cp jobs-on-my-machine container:/var/jenkins_home/jobs, but I dont want a lot of extra stuff outside my Dockerfile. I want to keep it as simple as possible. With this solution I will most likely not even be able to commit this change since the docker diff show this output:
C:\Dev>docker diff 3
C /tmp
A /tmp/hsperfdata_jenkins
A /tmp/hsperfdata_jenkins/6
A /tmp/jetty-0.0.0.0-8080-war-_-any-6459623464825334329.dir
A /tmp/jna--1712433994
A /tmp/winstone4948624124562796293.jar
you see... Nothing have changed in /var/jenkins_home/jobs/ directory... Nothing trackable at all :(
Take a look at my Dockerfile content below:
# Pull the latest Jenkins docker image from Docker Hub.
# Currently Alpine is used because the Internet says its safer :)
from jenkins/jenkins:lts-alpine
# Disable Jenkins setup wizard normally showing up during initial startup
ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false"
# Copy the groovy script creating the first user into the
# run-on-startup-directory into the docker image
COPY **security.groovy** /usr/share/jenkins/ref/init.groovy.d/security.groovy
# Copy the plugins text file into the docker image
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
# Run the Jenkins default install-plugins script to install
# plugins defined in the plugins text file
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt
# Add the predefined Jenkins jobs _inside_ folder 'jenkins-jobs' into Jenkins
COPY jenkins-jobs /var/jenkins_home/jobs/
security.groovy content:
#!groovy
import jenkins.model.*
import hudson.security.*
import jenkins.security.s2m.AdminWhitelistRule
def instance = Jenkins.getInstance()
def hudsonRealm = new HudsonPrivateSecurityRealm(false)
hudsonRealm.createAccount("admin", "admin")
instance.setSecurityRealm(hudsonRealm)
def strategy = new FullControlOnceLoggedInAuthorizationStrategy()
instance.setAuthorizationStrategy(strategy)
instance.save()
Jenkins.instance.getInjector().getInstance(AdminWhitelistRule.class).setMasterKillSwitch(false)
plugins.txt:
cloudbees-folder
bouncycastle-api
structs
script-security
workflow-step-api
scm-api
workflow-api
junit
antisamy-markup-formatter
workflow-support
workflow-job
token-macro
build-timeout
credentials
ssh-credentials
plain-credentials
credentials-binding
timestamper
durable-task
workflow-durable-task-step
matrix-project
resource-disposer
ws-cleanup
ant
gradle
pipeline-milestone-step
jquery-detached
jackson2-api
ace-editor
workflow-scm-step
workflow-cps
pipeline-input-step
pipeline-stage-step
pipeline-graph-analysis
pipeline-rest-api
handlebars
momentjs
pipeline-stage-view
pipeline-build-step
pipeline-model-api
pipeline-model-extensions
apache-httpcomponents-client-4-api
jsch
git-client
git-server
workflow-cps-global-lib
display-url-api
mailer
branch-api
workflow-multibranch
authentication-tokens
docker-commons
docker-workflow
pipeline-stage-tags-metadata
pipeline-model-declarative-agent
workflow-basic-steps
pipeline-model-definition
workflow-aggregator
github-api
git
github
github-branch-source
pipeline-github-lib
mapdb-api
subversion
ssh-slaves
matrix-auth
pam-auth
ldap
email-ext
mercurial
In case someone else find this question:
From the GitHub - jenkinsci/docker:
In such a derived image, you can customize your jenkins instance with hook scripts or additional plugins. For this purpose, use /usr/share/jenkins/ref as a place to define the default JENKINS_HOME content you wish the target installation to look like :
When jenkins container starts, it will check JENKINS_HOME has this reference content, and copy them there if required. It will not override such files, so if you upgraded some plugins from UI they won't be reverted on next start.
In case you do want to override, append '.override' to the name of the reference file. E.g. a file named /usr/share/jenkins/ref/config.xml.override will overwrite an existing config.xml file in JENKINS_HOME.
What I've done myself and the thing that is working for me is: copy the jobs directory found on your Jenkins instance to where you are able to run docker build. Place the jobs directory from the Jenkins instance in to another directory e.g jobs_jenkins. Meaning your will have $PWD/jobs_jenkins/jobs
Add the following line in your Dockerfile:
...
FROM jenkins/jenkins:lts-alpine
COPY jobs_jenkins /usr/share/jenkins/ref/
...
From my understanding I think the jenkins image itself run a "startup-script" utilizing /usr/share/jenkins/ref, and the stuff copied directly to /var/jenkins_home will not be present...
For more information read the README.md. (From the GitHub - jenkinsci/docker)
I want to install same plugins in to my local Jenkins which are already installed in other Jenkins.
Want to avoid installing all the 50-60 odds plugin manually
The official Jenkins documentation on installing plugins gives two ways of installing plugins:
Via the web interface
Save the downloaded *.hpi/*.jpi file into the $JENKINS_HOME/plugins directory.
So my answer to your question would be: copy the $JENKINS_HOME/plugins directory from server A to server B.
Don't forget to restart Jenkins afterwards!
There's another way, that's ideal if you're using Jenkins inside a docker container, first you need to extract a list of installed plugins by running curl against your jenkins domain in terminal:
export JENKINS_URL=http://<jenkins_domain>
curl -sSL "$JENKINS_URL/pluginManager/api/xml?depth=1&xpath=/*/*/shortName|/*/*/version&wrapper=plugins" | perl -pe 's/.*?<shortName>([\w-]+).*?<version>([^<]+)()(<\/\w+>)+/\1 \2\n/g'|sed 's/ /:/'
this you return you a list of installed plugins formatted like this:
aws-credentials:1.15
aws-beanstalk-publisher-plugin:1.6.0
aws-java-sdk:1.10.45.2
Then you can run this script against the list saved in a txt file to install all the exported plugins or add it to the end of your Dockerfile like this:
# copy script to container's bin
ADD ./plugin.sh /usr/local/bin/plugins.sh
# copy plugins list to inside the container
COPY plugins.txt /plugins.txt
# runs it
RUN /usr/local/bin/plugins.sh /plugins.txt
Just remember export your JENKINS_HOME variable before doing this.
Installing a plugin from the Update center results in:
Checking internet connectivity Failed to connect to
http://www.google.com/. Perhaps you need to configure HTTP proxy? Deploy Plugin Failure - Details hudson.util.IOException2: Failed to download from
http://updates.jenkins-ci.org/download/plugins/deploy/1.9/deploy.hpi
Is it possible to download the plugin and install it manually into Jenkins?
Yes, you can. Download the plugin (*.hpi file) and put it in the following directory:
<jenkinsHome>/plugins/
Afterwards you will need to restart Jenkins.
Download the plugin.
Inside Jenkins: Manage Jenkins → Manage Plugins → There is a tab called Advanced and on that page there is an option to upload a plugin (the extension of the file must be hpi).
Sometimes, when you download plugins you may get (.zip) files then just rename with (.hpi) and use the UI to install the plugin.
If you use Docker, you should read this file: https://github.com/cloudbees/jenkins-ci.org-docker/blob/master/plugins.sh
Example of a parent Dockerfile:
FROM jenkins
COPY plugins.txt /plugins.txt
RUN /usr/local/bin/plugins.sh /plugins.txt
plugins.txt
<name>:<version>
<name2>:<version2>
I have created a simple script that does the following:
Download one or more plugins to the plugin directory
Scan all plugins in that directory for missing dependencies
download this dependencies as well
loop until no open dependencies are left
The script requires no running jenkins - I use it to provision a docker box.
https://gist.github.com/micw/e80d739c6099078ce0f3
Sometimes when you download plugins you may get (.zip) files then just rename with (.hpi) and then extract all the plugins and move to <jenkinsHome>/plugins/ directory.
Update for Docker: use the install-plugins.sh script. It takes a list of plugin names minus the '-plugin' extension. See the description here.
install-plugins.sh replaces the deprecated plugins.sh which now warns :
WARN: plugins.sh is deprecated, please switch to install-plugins.sh
To use a plugins.txt as per plugins.sh see this issue and this workaround:
RUN /usr/local/bin/install-plugins.sh $(cat /usr/share/jenkins/plugins.txt | tr '\n' ' ')
Use https://updates.jenkins-ci.org/download/plugins/. Download it from this central update repository for Jenkins.
The accepted answer is accurate, but make sure that you also install all necessary dependencies as well. Installing using the CLI or web seems to take care of this, but my plugins were not showing up in the browser or using java -jar jenkins-cli.jar -s http://localhost:8080 list-plugins until I also installed the dependencies.
The answers given work, with added plugins.
If you want to replace/update a built-in plugin like the credentials plugin, that has dependencies, then you have to use the frontend. To automate I use:
curl -i -F file=#pluginfilename.hpi http://jenkinshost/jenkins/pluginManager/uploadPlugin
In my case, I needed to install a plugin to an offline build server that's running a Windows Server (version won't matter here). I already installed Jenkins on my laptop to test out changes in advance and it is running on localhost:8080 as a windows service.
So if you are willing to take the time to setup Jenkins on a machine with Internet connection and carry these changes to the offline server Jenkins (it works, confirmed by me!), these are steps you could follow:
Jenkins on my laptop: Open up Jenkins, http://localhost:8080
Navigator: Manage Jenkins | Download plugin without install option
Windows Explorer: Copy the downloaded plugin file that is located at "c:\program files (x86)\Jenkins\plugins" folder (i.e. role-strategy.jpi)
Paste it into a shared folder in the offline server
Stop the Jenkins Service (Offline Server Jenkins) through Component Services, Jenkins Service
Copy the plugin file (i.e. role-strategy.jpi) into "c:\program files (x86)\Jenkins\plugins" folder on the (Offline Jenkins) server
Restart Jenkins and voila! It should be installed.
This is a way to copy plugins from one Jenkins box to another.
Copy over the plugins directory:
scp -r jenkins-box.url.com:/var/lib/jenkins/plugins .
Compress the plugins:
tar cvfJ plugins.tar.xz plugins
Copy them over to the other Jenkins box:
scp plugins.tar.xz different-jenkins-box.url.com
ssh different-jenkins-box.url.com "tar xvfJ plugins.tar.xz -C /var/lib/jenkins"
Restart Jenkins.
use this link to download the lastest version of the plugins' hpi. https://updates.jenkins-ci.org/download/plugins/
Then upload the plugin through 'manage plugin' in Jenkins
To install plugin "git" with all its dependencies:
curl -XPOST http://localhost:8080/pluginManager/installNecessaryPlugins -d '<install plugin="git#current" />'
Here, the plugin installed is git ; the version, specified as #current is ignored by Jenkins. Jenkins is running on localhost port 8080, change this as needed. As far as I know, this is the simplest way to install a plugin with all its dependencies 'by hand'. Tested on Jenkins v1.644
RUN /usr/local/bin/install-plugins.sh amazon-ecs:1.37 configuration-as-code:1.47 workflow-aggregator:2.6 \
cloudbees-folder:6.15 antisamy-markup-formatter:2.1 build-timeout:1.20 credentials-binding:1.24
Cat out the plugins.txt and install in Dockerfile as above.