I would like to be able to perform linting on Jenkins pipelines and it seems that Groovy linting is not enough.
How can I do this?
HTTP without crumb.
If you want to use HTTP and don't want to use CRUMB. just add your username and password using the '-u' parameter. Replace <username> and <password> with the username and password of your user. Also Check that the url of the jenkins server is correct.
curl --user <username>:<password> -X POST -F "jenkinsfile=<Jenkinsfile" http://localhost:8080/pipeline-model-converter/validate
src
If for some reason you can't use Jenkins server linter, you can use npm-groovy-lint (works with Declarative or Scripted Jenkinsfile, and also groovy shared libraries)
https://github.com/nvuillam/npm-groovy-lint
npm install -g npm-groovy-lint
npm-groovy-lint // in the root directory of the Jenkinsfile
Looks like there are two options for linting pipeline scripts, one via the cli on the leader or an http POST call:
Linting via the CLI with SSH
# ssh (Jenkins CLI)
# JENKINS_SSHD_PORT=[sshd port on master]
# JENKINS_HOSTNAME=[Jenkins master hostname]
ssh -p $JENKINS_SSHD_PORT $JENKINS_HOSTNAME declarative-linter < Jenkinsfile
Linting via HTTP POST using curl
# curl (REST API)
# Assuming "anonymous read access" has been enabled on your Jenkins instance.
# JENKINS_URL=[root URL of Jenkins master]
# JENKINS_CRUMB is needed if your Jenkins master has CRSF protection enabled as it should
JENKINS_CRUMB=`curl "$JENKINS_URL/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,\":\",//crumb)"`
curl -X POST -H $JENKINS_CRUMB -F "jenkinsfile=<Jenkinsfile" $JENKINS_URL/pipeline-model-converter/validate
https://jenkins.io/doc/book/pipeline/development/#linter
In addition to kongkoro's answer, there is a tool to lint Jenkinsfile.
https://www.npmjs.com/package/jflint
# install
$ npm install -g jflint
# usage
# JENKINS_URL=[root URL of Jenkins master]
$ jflint -j $JENKINS_URL Jenkinsfile
What the jflint does is the same as curl in the official document, and jflint works only with declarative pipelines too. But it's easier to use.
SSH
Methods using Jenkins SSH interface to run the linter:
Enable SSH service in the Configure Global Security page and assign the port (e.g. 2222).
Add your Public SSH Key in your user's profile in Jenkins (JENKINS_URL/user/USER/configure).
Confirm the SSH access by SSHing to Jenkins and run:
ssh -l admin -p 2222 localhost help
Validate your local Jenkinsfile using the following command on Jenkins box:
ssh -l admin -p 2222 localhost declarative-linter < ./Jenkinsfile
For further details, read Pipeline Development Tools.
Furthermore, to simplify, you can add the following section to your ~/.ssh/config:
Host jenkins-cli
HostName localhost
User admin
Port 2222
ProxyJump jenkins-host.example.com
Then run: ssh jenkins-cli declarative-linter < ./Jenkinsfile.
You can also consider creating the following shell alias (e.g. to your startup files):
alias jenkins-lint="ssh jenkins-cli declarative-linter < ./Jenkinsfile"
Then just run: jenkins-lint.
POST
Validate a Jenkinsfile by using the following curl command:
curl --user username:password -X POST -F "jenkinsfile=<Jenkinsfile" http://jenkins-url:8080/pipeline-model-converter/validate
For details, please read How to validate a Jenkinsfile page.
VS Code plugin
Using VS Code IDE editor, you can install Jenkins Pipeline Linter Connector plugin and configure accordingly to the instructions, so it can post your Jenkinsfile to your Jenkins Server via POST request.
If you want to lint Jenkins pipelines which can be scripted or declarative.
Then the best solution is to lint using the jenkins-cli.jar.
I tried whatever I could possibly get my hands at but this really looks like the best and most convenient to use.
Requirements would be - java
Download the cli jar
$ curl -O https://<jenkins-server>/jnlpJars/jenkins-cli.jar
Lint the Jenkins pipeline script - either Scripted or Declarative
$ java -jar jenkins-cli.jar -s '<jenkins-server-url' -auth <username>:<password> declarative-linter < Jenkinsfile
Its always best to use the jenkins server url where it will be placed as that takes care of checking if the necessary plugins, etc are in place for the pipeline to function correctly.
Jenkins-CLI
Related
My Jenkins is on host1 and I wish to trigger ansible which is on host2 using Jenkins pipeline. This can be done by creating a slave node on host2 and specifying agent in Jenkins pipeline.
However, I do not have a Jenkins slave on host2.
Instead, Jenkins has connectivity to host2 by means of Server Groups Center which can be found under Jenkins Global Configuration
Do I need a Jenkins slave on host2 ? If not, then how can I use Server Groups Center in Jenkins pipeline to trigger Ansible on host2? Sample code pleaseā¦
Do I need a Jenkins slave on host2?
No, if you have Linux on host2: you can simply run any command over SSH.
How can I use Server Groups Center in Jenkins pipeline to trigger Ansible on host2?
Server Groups Center block comes from SSH2 Easy plugin, which is very old and doesn't support Jenkins pipeline. So you can't use information from that block of settings in your pipeline.
But there are other plugins for SSH; try Publish over SSH plugin for example. This plugin adds Publish over SSH block to Jenkins Global Configuration, where you can specify host2 connection parameters.
And then you can write pipeline step as follows ({HOST2} is the name of host2 that you type in Publish over SSH block in Jenkins Global Configuration):
steps {
sshPublisher \
failOnError: true, \
publishers: [ \
sshPublisherDesc( \
configName: "{HOST2}", \
transfers: [ \
sshTransfer (execCommand: "ansible -m ping all -i inventory_file", execTimeout: 120000) \
] \
) \
]
}
I have a Jenkins CI and use it to build (mvn) and containerize (docker) my app using Jenkins scripted pipeline. Lastly, I want to deploy my container to Heroku dyno (I have already created an app).
I have followed this documentation https://devcenter.heroku.com/articles/container-registry-and-runtime and have been successfully pushed my docker image to registry.heroku.com/sunset-sailing-4049/web.
The issue is since this announcement https://devcenter.heroku.com/changelog-items/1426 I now need to explicitly execute "heroku container:release web" in order to get my docker container running from registry to app dyno. This is where I am royally stuck. See my below issues:
Heroku is not recognized by Jenkins. (My Jenkins is running on ec2, I have installed heroku toolbelt as ec2-user user. But Jenkins throws error: heroku: command not found). How do I resolve this issue?
How to do "heroku login" from Jenkins, since the login command prompts for browser login. I have added ssh key but I do not know how to use it from the command line, hence Jenkins "shell script"
The only other way I could think of is deploying via heroku pipeline using a dummy git repo onto which Jenkins will upload the source code on a successful build.
Would really appreciate your help solving the above 2 issues.
Thanks in Advance.
You need install heroku as user under which jenkins is running. Or if you installed it globally it may be not in PATH of user under which jenkins is running.
There are multiple options for setting PATH:
Set for specific command.
If your job is pipeline just wrap heroku command in withEnv closure:
withEnv(['PATH+HEROKU=/use/local/bin/']) {
your heroku command here
}
Set path for jenkins slave: go to [Manage Jenkins] -> [Manage Nodes], configure your node and set Environment variable PATH to $PATH:/use/local/bin/. This way all jobs running on slave will get environment variable injected.
For automated cli interactions heroku supports API tokens. You can either put it in ~/.netrc on build machine or supply as environment variable (see here).
(writing here incase someone is facing the same scenario)
ok I took #vladimir's suggestion and did the below:
Heroku command (for jenkins running on ec2):
The below command is needed to push a built docker image to heroku via jenkins/or other ci/cd tool; Because of a recent change (https://devcenter.heroku.com/changelog-items/1426) pushing to heroku registry isn't sufficient any longer. In order to execute the below command you need to install heroku toolbelt.
heroku container:release web
Install snap on amazon linux like below:
follow instruction to enable epel https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/add-repositories.html
Then Modify /etc/yum.repos.d/epel.repo. Under the section marked [epel] , change enabled=0 to enabled=1.
Then do
sudo yum install epel-release
sudo yum install yum-plugin-copr
sudo yum copr enable ngompa/snapcore-el7
sudo yum -y install snapd
sudo systemctl enable --now snapd.socket
Then install heroku toolbelt:
sudo snap install --classic heroku
Deploying to docker image to heroku:
In Jenkins scripted pipeline:
withCredentials([string(credentialsId: 'heroku-api-cred', variable: 'herokuRegistryApiCred')]) {
sh "docker login -u email#example.com -p ${herokuRegistryApiCred} registry.heroku.com"
}
// Tag docker img (in my case it was an image in dockerhub)
sh "docker tag dockerhubusername/pvtreponame:${imageTag} registry.heroku.com/your_app_name/release_type[ie>web]"
sh "docker push registry.heroku.com/your_app_name/web"
sh "/usr/local/bin/heroku container:release web --app=your_app_name"
sh "docker logout registry.heroku.com"
In order to run the app inside docker (in my case it was java) I had to add the below line (otherwise it was crashing because 1. tell app about heroku's port binding. 2. tell web process to run command. The ENTRYPOINT ["java","-jar","my_spring_boot_app-0.0.1-SNAPSHOT.jar"] does not work on heroku.):
CMD ["web", "java $JAVA_OPTS -Dserver.port=$PORT -jar /usr/app/my_spring_boot_app-0.0.1-SNAPSHOT.jar"]
I try to build gerrit with bazel (0.13.0rc2 and 0.20) on Ubuntu. My company uses a proxy that requires authentication. The environment variables http_proxy and https_proxy are set (and other tools like wget, maven, ... work) but bazel reports "HTTP/1.0 407 Proxy Authentication Required" errors:
$ bazel build gerrit
WARNING: ignoring http_proxy in environment.
ERROR: error loading package '': Encountered error while reading extension file 'closure/defs.bzl': no such package '#io_bazel_rules_closure//closure': Error downloading [https://github.com/bazelbuild/rules_closure/archive/08039ba8ca59f64248bb3b6ae016460fe9c9914f.tar.gz] to /home/xxxxxx/.cache/bazel/_bazel_xxxxxx/5fb8c741228852f6bc57df1fc04917a8/external/io_bazel_rules_closure/08039ba8ca59f64248bb3b6ae016460fe9c9914f.tar.gz: Unable to tunnel through proxy. Proxy returns "HTTP/1.0 407 Proxy Authentication Required"
Any suggestions are welcome!
If somebody else has a similar problem:
After searching a while and e. g. tried to specify the environment variables HTTP_PROXY and HTTPS_PROXY (see also bazel issue #587) I found this discussion and used gerritforge/jenkins-slave-bazel for building (proxy configured also for docker):
# use docker the docker container gerritforge/jenkins-slave-bazel to build gerrit
$ docker run --privileged -ti --entrypoint=bash --user jenkins gerritforge/jenkins-slave-bazel
$ cd && git clone --recursive https://gerrit.googlesource.com/gerrit
$ cd gerrit && bazel build gerrit
That at least solves the bazel proxy issue.
[Help]
Description of Problem
jenkins-cli not authenticating with provided ssh private key
Observed
when passing the jenkins-cli command:
java -jar ~/jenkins-cli.jar -s http://localhost:8080 -i ~/.ssh/ccdevops who-am-i
The console output is:
Authenticated as: anonymous
Authorities:
Desired
Jenkins should authenticate as the user with the matching public key in their profile
Relevant Information
jenkins v 2.46.3 and using the correct cli jar for the version
Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-78-generic x86_64)
only using recommend plugins
running on azure cloud in china east datacenter
azure network security group for the vm is configured to allow traffic
ssh key being used was created on the ubuntu machine jenkins is running on and the public key is in the users entry in the jenkins user database
Key was created using the instructions on the github site
jenkins-cli is running on the server and not from a remote host
Steps Tried Already
tried different keys with and without passphrases
tried the web address with localhost and the ip address
tried other jenkins-cli commands with same result
tried to create other users and putting an public ssh key in their profile. (no duplicate keys between users)
tried moving the location of the jenkins-cli jar from server root to jenkins home directory
You should also specify the SSH method and the user on the command line with -ssh and -user USER_NAME respectively. After that, your command would look like this:
java -jar ~/jenkins-cli.jar -s http://localhost:8080 -i ~/.ssh/ccdevops who-am-i -ssh -user USER_NAME
Also note that you'll need to be able to access the server via SSH as well.
As per the title, in Jenkins how can I add new slave nodes to my build cluster using the CLI, or if there is not a CLI option, is there another scriptable approche that can be used?
Basic CLI instruction can be found here.
The following CLI command should get the new node configuration XML as stdin:
java -jar jenkins-cli.jar -s [JENKINS_URL] create-node [NewNodeName]
For example, if you want to copy an existing node, you can use:
java -jar jenkins-cli.jar -s [JENKINS_URL] get-node [NodeToCopyFrom] | java -jar jenkins-cli.jar -s [JENKINS_URL] create-node [NewNodeName]
Many people use the Swarm Plugin to eliminate the need to actually add slaves manually. You would need to script the install of the swarm agent of course, but that should be pretty straight forward.