I am creating a new Pipeline on an existing, functioning Jenkins server.
At the moment it is a simple multibranch pipeline with the Jenkinsfile, and a deploy.yml
The issue is that the workspace folder is not being generated.
When I do ls /var/lib/jenkins/workspace I get a list of all the other deploy jobs that have created workspaces, and a workspace.txt file where I can see the names of the workspaces that SHOULD have been created, but there is no folder for that workspace.
If I do mkdir /var/lib/jenkins/workspace/expectedname and touch /var/lib/jenkins/workspace/expectedname/deploy.yml and run it, it creates a workspace called /var/lib/jenkins/workspace/expectedname#tmp which is empty, and puts the retreived files in the one i created. If I delete the #tmp folder, it works. but it breaks again without advance notice, and it isn't feasible to keep having to create the fodler manually.
Jenkinsfile
#!groovy
node {
currentBuild.displayName = "${env.BRANCH_NAME}-${env.BUILD_NUMBER}"
config = readYaml file: "${WORKSPACE}/deploy.yml"
}
pipeline {
agent any
stages {
stage ('Fiddling'){
steps {
echo "***************** TAG: ${config.scm.tag} *****************"
}
}
}
}
deploy.yml
env:
wwwroot: "/var/www/siteroot"
type: "test"
ssh:
servername: "mysecretserver.url"
username: "jenkins"
conf:
siteTitle: "Site Title"
siteFQDN: "site.subdomain.domain.tld"
defaultFromEmail: "test#domain.tld"
sessionname: "SessionName"
mysql:
host: "localhost"
database: "mycooldb"
username: "dbuser"
password: "s3cret"
scm:
tag: "v.1.2"
jenkins log
15:28:41 Started by user JoSSte
15:28:41 > git rev-parse --is-inside-work-tree # timeout=10
15:28:41 Setting origin to git#bitbucket.org:t******/****.git
15:28:41 > git config remote.origin.url git#bitbucket.org:t******/****.git # timeout=10
15:28:41 Fetching origin...
15:28:41 Fetching upstream changes from origin
15:28:41 > git --version # timeout=10
15:28:41 > git --version # 'git version 2.17.1'
15:28:41 > git config --get remote.origin.url # timeout=10
15:28:41 using GIT_SSH to set credentials YYYYYY
15:28:41 > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
15:28:42 Seen branch in repository origin/master
15:28:42 Seen branch in repository origin/noscm
15:28:42 Seen branch in repository origin/stable
15:28:42 Seen branch in repository origin/working
15:28:42 Seen 4 remote branches
15:28:44 Obtained Jenkinsfile from 5374b39f33bf6ye904134e845d60fae76a97bb09
15:28:44 Running in Durability level: MAX_SURVIVABILITY
15:28:44 [Pipeline] Start of Pipeline
15:28:44 [Pipeline] node
15:28:44 Running on Jenkins in /var/lib/jenkins/workspace/tc-d-tr_noscm
15:28:44 [Pipeline] {
15:28:44 [Pipeline] readYaml
15:28:44 [Pipeline] }
15:28:44 [Pipeline] // node
15:28:44 [Pipeline] End of Pipeline
15:28:44 java.io.FileNotFoundException: /var/lib/jenkins/workspace/tc-d-tr_noscm/deploy.yml does not exist.
15:28:44 at org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep$Execution.doRun(ReadYamlStep.java:113)
15:28:44 at org.jenkinsci.plugins.pipeline.utility.steps.AbstractFileOrTextStepExecution.run(AbstractFileOrTextStepExecution.java:32)
15:28:44 at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
15:28:44 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
15:28:44 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
15:28:44 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
15:28:44 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
15:28:44 at java.lang.Thread.run(Thread.java:748)
15:28:44 Finished: FAILURE
Why is this job failing when all the others are not? - the only thing I can see different is that the other jobs I have don't use readYaml
Update
I made a new repo, with basically the same structure, without readYaml, on the same jenkins server - this hasn't failed in the same way - I will try to ad that to it, and see if it breaks...
Related
I want to clone a github repository using Jenkins decriptive pipeline. I use the following code :
pipeline {
agent any
stages {
stage('Checkout') {
steps {
script {
// The below will clone your repo and will be checked out to master branch by default.
git credentialsId: 'jenkins-user-github', url: 'https://github.com/aakashsehgal/FMU'
// Do a ls -lart to view all the files are cloned. It will be clonned. This is just for you to be sure about it.
//bat "ls -lart ./*"
// List all branches in your repo.
bat "git branch -a"
// Checkout to a specific branch in your repo.
}
}
}
}
}
But Jenkins cannot log on to the Github account. I have double checked the credentials and they are correct. Here is the error :
Started by user Aakash Sehgal
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\Program Files (x86)\Jenkins\workspace\GitHub
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Checkout)
[Pipeline] script
[Pipeline] {
[Pipeline] git
using credential jenkins-user-github
> git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git.exe config remote.origin.url https://github.com/aakashsehgal/FMU # timeout=10
Fetching upstream changes from https://github.com/aakashsehgal/FMU
> git.exe --version # timeout=10
using GIT_ASKPASS to set credentials
> git.exe fetch --tags --force --progress -- https://github.com/aakashsehgal/FMU +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/aakashsehgal/FMU
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1131)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "git.exe fetch --tags --force --progress -- https://github.com/aakashsehgal/FMU +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: Logon failed, use ctrl+c to cancel basic credential prompt.
warning: invalid credential line: get
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/aakashsehgal/FMU/'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:81)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:569)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:907)
... 11 more
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE
I also searched online and someone mentions that it could be because of https. However not clear how to resolve it. Any idea what could i be doing wrong ?
So after some research I finally found the answer. The problem was that Jenkins wasnt able to resolve the path of the Git executable.
Go to Manage Jenkins -> Global Tool Configuration. Under the Git Tab, we need to give the absolute path of the Git executable. For a standard installation, the default path is C:\Program Files\Git\bin\git.exe. Save this and tried again. Works.
I have a scenario where I manage two pipelines within the same repo, with two different Jenkinsfiles.
I have set up two Jenkins multibranch pipelines to handle the two different Jenkinsfiles, by path discovery and set up github webhooks to trigger a build when a PR on particular branches is created.
I have not found a way to get changes in files for a particular PR so I thought of leveraging git by doing git diff --name-status origin/master...HEAD but it fails, since Jenkins only checks out the target branch.
The logstash:
using credential github-user-token-uname-pwd
Fetching changes from the remote Git repository
Fetching without tags
> git rev-parse --is-inside-work-tree # timeout=10
> git config remote.origin.url https://github.com/myreponame # timeout=10
Fetching upstream changes from https://github.com/myreponame
> git --version # timeout=10
using GIT_ASKPASS to set credentials Github token in uname-pwd form used by jenkins to register and manage webhooks
> git fetch --no-tags --force --progress https://github.com/myreponame +refs/heads/BRANCH-X:refs/remotes/origin/BRANCH-X
Checking out Revision 440df9b08667458fa4ade4be57ecbf59a4 (BRANCH-X)
Commit message: "move post build where it belongs"
> git config core.sparsecheckout # timeout=10
> git checkout -f 440df9b08667458fa4ade4be57ecbf59a4
> git rev-list --no-walk ab28e843c0fc7807c4cbd2d6f612e5d76b # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $SECRET_ACCESS_KEY or $ACCESS_KEY_ID
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] sh
+ git diff --name-status origin/master...HEAD
and the error I get:
fatal: ambiguous argument 'origin/master...HEAD': unknown revision or path not in the working tree.
Is there a way to retrieve the files' changes of a PR within multibranch pipelines or get Jenkins to be able to discover the source branch of said PR?
You could try to fetch origin and then diff with FETCH_HEAD
{ git fetch origin master
git diff —name-status FETCH_HEAD...HEAD
}
So here is what I had to do.
The error I was facing was due to Jenkins default checkout behaviour with multibranch pipelines: it checks out and track a branch and that branch only, it does not fetch other branches.
I eventually added
options {
skipDefaultCheckout true
}
and a stage that checks out all the origin remote branches
stage('Checking out repo'){
steps {
script {
checkout(
[
$class: 'GitSCM',
branches: [[name: 'origin/FEATUREBRANCH*']],
doGenerateSubmoduleConfigurations: false,
extensions: [[$class: 'LocalBranch'],
[$class: 'CleanBeforeCheckout']],
submoduleCfg: [],
userRemoteConfigs: [[credentialsId: 'github-deploy-key',
url: 'git#github.com:myawesomerepo.git']]
]
)
}
}
}
I'm creating a Jenkinsfile for use with GitHub Enterprise. I used the GUI settings in the pipeline job to specify a Jenkinsfile from a GitHub repo.
I'm using the scripted syntax instead of the declarative syntax.
I am able to checkout the repo in my Jenkinsfile using checkout scm. I want to use some information about the checkout in my script, such as the branch name and commit hash. However, I can't figure out how to access variables of the scm class.
When I run the job, it fails in the Checkout stage. The checkout from git seems to work properly, but it fails without printing any errors. If I delete the echo scm.GIT_BRANCH line it works fine.
node {
stage('Checkout') {
checkout scm
echo scm.GIT_BRANCH
}
}
Here's the output:
Started by user spark
Obtained nightly/Jenkinsfile from git https://github.enterprise.instance.com/spark/ci_flow_test
[Pipeline] node
Running on jenkins-server in /home/spark/ci_flow_test/pipeline_test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Checkout)
[Pipeline] checkout
> /apps/git/git18/bin/git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> /apps/git/git18/bin/git config remote.origin.url https://github.enterprise.instance.com/spark/ci_flow_test # timeout=10
Fetching upstream changes from https://github.enterprise.instance.com/spark/ci_flow_test
> /apps/git/git18/bin/git --version # timeout=10
using GIT_ASKPASS to set credentials
> /apps/git/git18/bin/git fetch --tags --progress https://github.enterprise.instance.com/spark/ci_flow_test +refs/heads/*:refs/remotes/origin/*
> /apps/git/git18/bin/git rev-parse refs/remotes/origin/working^{commit} # timeout=10
> /apps/git/git18/bin/git rev-parse refs/remotes/origin/origin/working^{commit} # timeout=10
Checking out Revision 396f172c6061ba2760a71cba817df24836ec7e3b (refs/remotes/origin/working)
Commit message: "try echo"
> /apps/git/git18/bin/git config core.sparsecheckout # timeout=10
> /apps/git/git18/bin/git checkout -f 396f172c6061ba2760a71cba817df24836ec7e3b
> /apps/git/git18/bin/git rev-list 778c36171927020bd1afbd7206d86bf94abd1ed8 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] mail
[Pipeline] echo
Post script
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: FAILURE
You can use the checkout scm return value to get git infomation
node {
stage('Checkout') {
def d = checkout scm
echo "branch: " + d.GIT_BRANCH
echo "commit: " + d.GIT_COMMIT
}
}
// supported fields
GIT_AUTHOR_EMAIL
GIT_AUTHOR_NAME
GIT_BRANCH
GIT_COMMIT
GIT_COMMITTER_EMAIL
GIT_COMMITTER_NAME
GIT_LOCAL_BRANCH
GIT_PREVIOUS_COMMIT
GIT_PREVIOUS_SUCCESSFUL_COMMIT
GIT_URL
following a YT tutorial Getting Started with Blue Ocean's Visual Pipeline Editor, I struggle to proceed further after the second round of modification my Jenkinsfile using the BO Visual Editor.
When I hit the 'Save & run' button, Jenkins BO pipeline starts running, but fails with this:
GitHub has been notified of this commit’s build result
hudson.plugins.git.GitException: Command "git checkout -b blue-ocean-editor 9c164f507fa831c33d44d83e1f458ec50b433f40" returned status code 128:
stdout:
stderr: fatal: A branch named 'blue-ocean-editor' already exists
If you look into my repo and the particular commit(s) Jenkins had been able to create the 'blue-ocean-editor' branch in my initial commit (caused by the new BO pipeline) and also push the next commit "Expand my Blue Ocean pipeline", so I doubt that my Jenkins won't have the right credentials as discussed here.
However there are some timeouts. Not sure how much is this of concern.
Anyway, here's the full output of my BO pipeline:
Branch indexing
Connecting to https://api.github.com using zxxxy/****** (GitHub Access Token)
Obtained Jenkinsfile from 9c164f507fa831c33d44d83e1f458ec50b433f40
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/jobs/HelloWorldProject/branches/blue-ocean-editor/workspace
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
using credential github
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/delphym/HelloWorldProject.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Fetching without tags
Fetching upstream changes from https://github.com/delphym/HelloWorldProject.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials GitHub Access Token
> git fetch --no-tags --progress https://github.com/delphym/HelloWorldProject.git +refs/heads/blue-ocean-editor:refs/remotes/origin/blue-ocean-editor # timeout=10
Checking out Revision 9c164f507fa831c33d44d83e1f458ec50b433f40 (blue-ocean-editor)
> git config core.sparsecheckout # timeout=10
> git checkout -f 9c164f507fa831c33d44d83e1f458ec50b433f40 # timeout=10
> git branch -a -v --no-abbrev # timeout=10
> git checkout -b blue-ocean-editor 9c164f507fa831c33d44d83e1f458ec50b433f40 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 0s
GitHub has been notified of this commit’s build result
hudson.plugins.git.GitException: Command "git checkout -b blue-ocean-editor 9c164f507fa831c33d44d83e1f458ec50b433f40" returned status code 128:
stdout:
stderr: fatal: A branch named 'blue-ocean-editor' already exists.
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2318)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$900(CliGitAPIImpl.java:81)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2627)
Caused: hudson.plugins.git.GitException: Could not checkout blue-ocean-editor with start point 9c164f507fa831c33d44d83e1f458ec50b433f40
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2649)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1228)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
I wonder what I could be doing wrong. I believe I have pretty much up-to-date Jenkins and all the plugins and OS, but briefly:
OS: Ubuntu Trusty 14.04.6 LTS
Jenkins: Jenkins ver. 2.161 (a bit behind, but I want to switch to the stable releases)
Plugins: Blue Ocean 1.13.2
I have a jenkins pipeline job with the job config pulling Jenkinsfile from a repo . Once the job runs and pulls the jenkinsfile, it clones the repo and I can see it in workspace icon for the job.
Now when in Jenkinsfile I do cd ${workspace} and ls , it doesnt display anything. How do I access the workspace of the repo of the Jenkinsfile ? Or does it just store the Jenkinsfile itself ?
This is my Jenkinsfile :
node ("master"){
// Mark the code checkout 'Checkout'....
stage 'Checkout'
sh "pwd ; ls"
}
As I run it, I get the following log:
> GitHub pull request #282 of commit
> 0045a729838aae0738966423ff19c250151ed636, no merge conflicts. Setting
> status of 0045a729838aae0738966423ff19c250151ed636 to PENDING with url
> https://10.146.84.103/job/test1/ and message: 'Pull request builder
> for this security group pr started' Using context: SG Terraform
> Validate/Plan
> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository
> > git config remote.origin.url https://github.xxx.net/Terraform/djin-sg/ # timeout=10 Fetching
> upstream changes from https://github.xxx.net/Terraform/djin-sg/
> > git --version # timeout=10 using GIT_ASKPASS to set credentials wsjbuild
> > git fetch --tags --progress https://github.xxx.net/Terraform/djin-sg/
> +refs/heads/*:refs/remotes/origin/*
> > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 9dd8491b7b18c47eac09cec5a4bff4f16df979bf (origin/master)
> > git config core.sparsecheckout # timeout=10
> > git checkout -f 9dd8491b7b18c47eac09cec5a4bff4f16df979bf First time build. Skipping changelog. [Pipeline] node Running on master in
> /var/lib/jenkins/workspace/test1 [Pipeline] { [Pipeline] stage
> (Checkout) Using the ‘stage’ step without a block argument is
> deprecated Entering stage Checkout Proceeding [Pipeline] wrap
> [Pipeline] { [Pipeline] sh [test1] Running shell script
> + cd /var/lib/jenkins/workspace/test1
> + ls
My question specifically is that to get the Jenkinsfile it clones the djin-sg repo. Its in the workspace as well. So When I do ls why does it show no files ?
As I go to the Jenkins job pipeline steps and open workspace in console I can see the full repo in workspace but I cant seem to access it in the job.
Try instead a Jenkins pipeline syntax, like:
pipeline {
agent { node { label 'master' } }
stages {
stage('After Checkout') {
steps {
sh 'pwd; ls'
}
}
}
}
You can do checkout scm to actually checkout a repository to the workspace or you can find it under ../${env.JOB_NAME}#script only on master.
It's better to always checkout checkout scm manually because slaves do not have ../$env.JOB_NAME#script folder.