I have a jenkins pipeline script to create an app using openshift template placed in a repository(Bit Bucket). However, i get the below mentioned error everytime.
Note: I have configured credentials in Jenkins as well.
Version
openshift v3.11.200
kubernetes v1.11.0+d4cacc0
Steps To Reproduce
PipeLine Script:
// path of the template to use
def templatePath = 'https://xxxx/git/users/pdeb/repos/mashery-local/raw/acs/tml-acs-template.json'
// name of the template that will be created
def templateName = 'mashery-local'
// NOTE, the "pipeline" directive/closure from the declarative pipeline syntax needs to include, or be nested outside,
// and "openshift" directive/closure from the OpenShift Client Plugin for Jenkins. Otherwise, the declarative pipeline engine
// will not be fully engaged.
pipeline {
agent any
stages {
stage('preamble') {
steps {
script {
openshift.withCluster() {
openshift.withProject() {
echo "Using project: ${openshift.project()}"
}
}
}
}
}
stage('cleanup') {
steps {
script {
openshift.withCluster() {
openshift.withProject() {
// delete everything with this template label
openshift.selector("all", [ template : templateName ]).delete()
// delete any secrets with this template label
if (openshift.selector("secrets", templateName).exists()) {
openshift.selector("secrets", templateName).delete()
}
}
}
} // script
} // steps
} // stage
stage('create') {
steps {
script {
openshift.withCluster() {
openshift.withProject() {
// create a new application from the templatePath
openshift.newApp(templatePath)
}
}
} // script
} // steps
} // stage
} // stages
} // pipeline
Current Result:
*ERROR: new-app returned an error;
{reference={}, err=error: unable to load template file "https://rndwww.nce.amadeus.net/git/users/pdeb/repos/mashery-local-on-acs/raw/acs/tml-acs-template.json": unable to decode "https://rndwww.nce.amadeus.net/git/users/pdeb/repos/mashery-local-on-acs/raw/acs/tml-acs-template.json": couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
error: git ls-remote failed with: fatal: https://rndwww.nce.amadeus.net/git/users/pdeb/repos/mashery-local-on-acs/raw/acs/tml-acs-template.json/info/refs not valid: is this a git repository?; local file access failed with: stat https://rndwww.nce.amadeus.net/git/users/pdeb/repos/mashery-local-on-acs/raw/acs/tml-acs-template.json: no such file or directory
error: unable to locate any images in image streams, templates loaded in accessible projects, template files, local docker images with name "https://rndwww.nce.amadeus.net/git/users/pdeb/repos/mashery-local-on-acs/raw/acs/tml-acs-template.json"
Argument 'https://rndwww.nce.amadeus.net/git/users/pdeb/repos/mashery-local-on-acs/raw/acs/tml-acs-template.json' was classified as an image, image~source, or loaded template reference.
The 'oc new-app' command will match arguments to the following types:
1. Images tagged into image streams in the current project or the 'openshift' project
- if you don't specify a tag, we'll add ':latest'
2. Images in the Docker Hub, on remote registries, or on the local Docker engine
3. Templates in the current project or the 'openshift' project
4. Git repository URLs or local paths that point to Git repositories
--allow-missing-images can be used to point to an image that does not exist yet.
See 'oc new-app -h' for examples., verb=new-app, cmd=oc --server=https://10.224.0.1:443 --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --namespace=lhmdw-tml-dev --token=XXXXX new-app https://rndwww.nce.amadeus.net/git/users/pdeb/repos/mashery-local-on-acs/raw/acs/tml-acs-template.json -o=json , out=, status=1}
Finished: FAILURE*
Expected Result:
To run the template successfully.
The following error message indicates that your template file has an incorrect format:
json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
So you should fix the content of the template tml-acs-template.json, as currently this file cannot be used.
Related
I'm struggling with deploying the war file to Nexus repository using Jenkinsfile via Gradle task.
The war is being created successfully. I have also no problem with deploying JARs (since there are examples everywhere how to do it).
So I have this publishing section in my build.grade:
publishing {
repositories {
maven {
URI releasesUrl = new URI("${UploadURL}/repository/releases")
URI snapshotsUrl = new URI("${UploadURL}/repository/snapshots")
afterEvaluate {
url version.endsWith("SNAPSHOT") ? snapshotsUrl : releasesUrl
}
credentials {
username "${user}"
password "${password}"
}
}
}
publications {
mavenWeb(MavenPublication) {
from components.web
artifact war.archivePath
}
}
}
With pluggins:
apply plugin: 'java'
apply plugin: 'maven'
apply plugin: 'maven-publish'
apply plugin: 'org.springframework.boot'
apply plugin: 'war'
The URL for repositories is also specified in the build script correctly (test publish with the jar works just fine)
And the Jenkinsfile:
stage ('Publish war') {
steps {
sh "sh gradlew publish"
}
}
Currently I'm getting this error from jenkins build:
Task :publishMavenWebPublicationToMavenRepository FAILED
FAILURE: Build failed with an exception.
What went wrong:
Execution failed for task ':publishMavenWebPublicationToMavenRepository'.
Failed to publish publication 'mavenWeb' to repository 'maven'
Invalid publication 'mavenWeb': multiple artifacts with the identical extension and classifier ('war', 'null').
I'm quite sure that the problem is within "publications" part of Gradle task.
For publishing the Jars I have been using it like this:
[...]
publications {
mavenJava(MavenPublication) {
from components.java
artifact sourceJar
}
}
[...]
task sourceJar(type: Jar) {
classifier 'sources'
from sourceSets.main.java
}
I do not know how to configure from, artifact and classifier for this task. I do not even know if all of these parameters should be configured... Could anyone help me with that?
It turned out, that the origin of the problem was this section:
afterEvaluate {
url version.endsWith("SNAPSHOT") ? snapshotsUrl : releasesUrl
}
This feature works with Gradle 5.X version however, I was using Gradle 4.8. That lead to null instead of propper url value...
Unfortunately, it took a while since the exception message does not suggest where the problem was.
I'm putting together a Jenkins pipeline job which will take a file parameter. I can trigger the job and point it at a file however I can't find where the file has ended up (In an ordinary freestyle job it would be in the workspace).
Where has the uploaded file gone? Or do file parameters not currently work with pipelines?
There is currently an issue with pipeline and file parameter
(https://issues.jenkins-ci.org/browse/JENKINS-27413).
Solved it the following way:
node {
deleteDir()
stage("upload") {
def inputFile = input message: 'Upload file', parameters: [file(name: 'data.zip')]
new hudson.FilePath(new File("$workspace/data.zip")).copyFrom(inputFile)
inputFile.delete()
}
stage("checkout") {
echo fileExists('data.zip').toString()
}
}
I know the solution is not that beautiful because the pipeline gets interrupted for the upload but it works.
Further the "copyFrom" is necessary, because the input stores the "data.zip" in the jobs directory and not in the workspace (don't know why)
Found a WA (Strictly for text based file input)
We can use Jenkins multi-line string parameter and ask user to paste file contents to it.
And in our pipeline, write contents of this parameter using pipeline step writeFile, as :
stage('File Param WA') {
writeFile file: 'demo.yaml', text: params.DEMO_YAML
}
I tried using the solution provided by #Christoph Forster , but the input File was not getting copied anywhere in the workspace .
So I used the workaround as provided in
https://bitbucket.org/janvrany/jenkins-27413-workaround-library/src/6b7dada8ea37?at=default
The library provides a new library - unstashParam - that saves the file build parameter into a workspace. Works fine with text and yaml file .
I also tried using the solution by #Christoph Forster but I received a script security error when Groovy Sandbox is enable
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use new hudson.FilePath java.io.File
However, it seems we can skip the file copying and deleting actions (and bypass the Groovy sandbox restriction) by simply requiring that the file is uploaded to the job workspace. Just add the workspace variable to the file name as follows:
stage("upload") {
def inputFile = input message: 'Upload file', parameters: [file(name: "$workspace/data.zip")]
}
I found a solution in the form of a global library here:
https://bitbucket.org/janvrany/jenkins-27413-workaround-library/src/default/
It contains calls to inner methods of Jenkins which are deprecated (I guess).
So I made my own version like this:
import hudson.FilePath
import hudson.model.ParametersAction
import hudson.model.FileParameterValue
import hudson.model.Executor
def call(String name, String fname = null) {
def paramsAction = currentBuild.rawBuild.getAction(ParametersAction.class);
if (paramsAction == null) {
error "unstashParam: No file parameter named '${name}'"
}
for (param in paramsAction.getParameters()) {
if (param.getName().equals(name)) {
if (! param instanceof FileParameterValue) {
error "unstashParam: not a file parameter: ${name}"
}
if (env['NODE_NAME'] == null) {
error "unstashParam: no node in current context"
}
if (env['WORKSPACE'] == null) {
error "unstashParam: no workspace in current context"
}
workspace = new FilePath(getComputer(env['NODE_NAME']), env['WORKSPACE'])
filename = fname == null ? param.getOriginalFileName() : fname
file = workspace.child(filename)
file.copyFrom(param.getFile())
return filename;
}
}
}
def getComputer(name){
for(computer in Jenkins.getInstance().getComputers()){
if(computer.getDisplayName() == name){
return computer.getChannel()
}
}
error "Cannot find computer for file parameter workaround"
}
You can insert it in a global library and then use it like:
library "file-workaround"
node {
def file_in_workspace = unstashParam "myFile"
sh "cat ${file_in_workspace}"
}
It's not pretty but it's working and as long as there is no official fix, it's my best shot.
Update
Turns out you might run into "No such file or directory". That's because nothing in the workaround triggers Jenkins to create the workspace directory. If that was triggered somewhere else in the pipeline good, otherwise you'll be scratching your head.
You might wanna throw a
touch "thisIsAFile"
in there
To handle an optional file parameter in pipeline (to handle the use case where no file should be accepted) you could use jenkinsci-unstashParam-library (add it in Jenkins>Manage Jenkins>Configure System>Global Pipeline Libraries https://github.com/janvrany/jenkinsci-unstashParam-library) with a try/catch in a script as this sample stage:
stage('upload') {
steps {
// delete workspace
cleanWs()
// handle file parameters in pipeline (JENKINS-27413)
script {
try {
// force workspace directory creation
sh "touch emptyFileToCreateWorkspace"
// https://stackoverflow.com/questions/59468464/fetching-uploaded-files-in-jenkins
def file_in_workspace = unstashParam 'MY_FILE.xlsx'
// https://unix.stackexchange.com/questions/125776/error-with-a-file-name-containing-parentheses
sh "mv '${file_in_workspace}' MY_FILE.xlsx"
}
catch (Exception e) {
echo e.getMessage()
echo "No file parameter, we will continue.."
}
}
}
}
File parameters provides 2 alternative parameters types for files (stashed for large files and base64 for small files).
Example, for base64File:
node {
sh 'echo $FILE | base64 -d'
withFileParameter('FILE') {
sh 'cat $FILE'
}
}
and stashedFile:
node {
unstash 'FILE'
sh 'cat FILE'
}
Tried what Christoph suggested and it didnt work for me. Here is what worked for me and the setup which I have, his should help others figure out what to do.
Problem:
I am executing my pipeline on dedicated nodes and use sanitized workspaces. After some research and troubleshooting I found out that by default the file upload only works with Master node. I realized this after digging through the file system and finding the file I am uploading in the workspace on the master
Solution:
stage('Upload Key') {
agent { label 'master' }
steps {
script {
// Uploads file via master node and stases it for other nodes to access
def inputFile = input message: 'Upload file', parameters: [file(name: "key.p12")]
new hudson.FilePath(new File("${workspace}/key.p12")).copyFrom(inputFile)
inputFile.delete()
}
stash name: 'key.p12' , includes: "key.p12"
}
}
stage('Register') {
steps {
ws (sanitizedWorkspaceName) {
echo "Registering"
unstash 'key.p12'
}
}
}
Execute the suggested file copy solution by Christoph. This stores the file in the job workspace on the master node
Allow the scripts in Manage Jenkins > In Process Script approval
use the stash step to stash the uploaded file
In the target stage "running on a different node" use the unstash
Hope this helps
I wasn't able to make Christoph's solution working if the file was uploaded on master node and needed on slave. The solution was to stash it on master and later unstash it on slave. Don't forget to remove the uploaded file on master node.
It's supported by the latest File Parameters plugin now. Please refer to: How to pass a file parameter to another build job in jenkins pipeline?
I have a private GitHub Rust project that depends on another private GitHub Rust project and I want to build the main one with Jenkins. I have called the organization Organization and the dependency package subcrate in the below code.
My Jenkinsfile looks something like
pipeline {
agent {
docker {
image 'rust:latest'
}
}
stages {
stage('Build') {
steps {
sh "cargo build"
}
}
etc...
}
}
I have tried the following in Cargo.toml to reference the dependency, it works fine on my machine
[dependencies]
subcrate = { git = "ssh://git#ssh.github.com/Organization/subcrate.git", tag = "0.1.0" }
When Jenkins runs I get the following error
+ cargo build
Updating registry `https://github.com/rust-lang/crates.io-index`
Updating git repository `ssh://git#github.com/Organization/subcrate.git`
error: failed to load source for a dependency on `subcrate`
Caused by:
Unable to update ssh://git#github.com/Organization/subcrate.git?tag=0.1.0#0623c097
Caused by:
failed to clone into: /usr/local/cargo/git/db/subcrate-3e391025a927594e
Caused by:
failed to authenticate when downloading repository
attempted ssh-agent authentication, but none of the usernames `git` succeeded
Caused by:
error authenticating: no auth sock variable; class=Ssh (23)
script returned exit code 101
How can I get Cargo to access this GitHub repository? Do I need to inject the GitHub credentials onto the slave? If so, how can I do this? Is it possible to use the same credentials Jenkins uses to checkout the main crate in the first place?
I installed the ssh-agent plugin and updated my Jenkinsfile to look like this
pipeline {
agent {
docker {
image 'rust:latest'
}
}
stages {
stage('Build') {
steps {
sshagent(credentials: ['id-of-github-credentials']) {
sh "ssh -vvv -T git#github.com"
sh "cargo build"
}
}
}
etc...
}
}
I get the error
+ ssh -vvv -T git#github.com
No user exists for uid 113
script returned exit code 255
Okay, I figured it out, No user exists for uid error is because of a mismatch between the users in the host /etc/passwd and the container /etc/passwd. This can be fixed by mounting /etc/passwd.
agent {
docker {
image 'rust:latest'
args '-v /etc/passwd:/etc/passwd'
}
}
Then
sshagent(credentials: ['id-of-github-credentials']) {
sh "cargo build"
}
Works just fine
I have a pipeline flow defined as:
node("linux_label") {
println("hostname".execute().txt)
def filename = "${WORKSPACE}/submoduleinfo.txt"
stage("Submodule info") {
def submoduleString = sh script: "git -C ${WORKSPACE} submodule status > ${filename}", returnStdout: true
}
String fileContents = new File("$filename}").text
operateOnFile(fileContents)
}
At "new File" I will get an error saying no such file exists. after some troublehshooting I see that the hostname printout will output the jenkins master and not the node "linux_label" where the workspace resides.
Is this how Piepeline should work, i.e. all code that is not part of stage/steps/etc are executed on the jenkins master and not on the wanted node?
What would be a good workaround where I do an operation in one stage and want to operate on the file in the node {} domain?
That is how pipeline works. You can use readFile to read file from a workspace. Since you are using just a content of the file for your processing, this will work.
From tutorial:
readFile step loads a text file from the workspace and returns its
content (do not try to use java.io.File methods — these will refer to
files on the master where Jenkins is running, not in the current
workspace).
In one of our use case, we added some additional functions using Shared pipeline library.
Try this:
if (env['NODE_NAME'].equals("master")) {
return new hudson.FilePath(path);
} else {
return new hudson.FilePath(Jenkins.getInstance().getComputer(env['NODE_NAME']).getChannel(), path);
}
I'm putting together a Jenkins pipeline job which will take a file parameter. I can trigger the job and point it at a file however I can't find where the file has ended up (In an ordinary freestyle job it would be in the workspace).
Where has the uploaded file gone? Or do file parameters not currently work with pipelines?
There is currently an issue with pipeline and file parameter
(https://issues.jenkins-ci.org/browse/JENKINS-27413).
Solved it the following way:
node {
deleteDir()
stage("upload") {
def inputFile = input message: 'Upload file', parameters: [file(name: 'data.zip')]
new hudson.FilePath(new File("$workspace/data.zip")).copyFrom(inputFile)
inputFile.delete()
}
stage("checkout") {
echo fileExists('data.zip').toString()
}
}
I know the solution is not that beautiful because the pipeline gets interrupted for the upload but it works.
Further the "copyFrom" is necessary, because the input stores the "data.zip" in the jobs directory and not in the workspace (don't know why)
Found a WA (Strictly for text based file input)
We can use Jenkins multi-line string parameter and ask user to paste file contents to it.
And in our pipeline, write contents of this parameter using pipeline step writeFile, as :
stage('File Param WA') {
writeFile file: 'demo.yaml', text: params.DEMO_YAML
}
I tried using the solution provided by #Christoph Forster , but the input File was not getting copied anywhere in the workspace .
So I used the workaround as provided in
https://bitbucket.org/janvrany/jenkins-27413-workaround-library/src/6b7dada8ea37?at=default
The library provides a new library - unstashParam - that saves the file build parameter into a workspace. Works fine with text and yaml file .
I also tried using the solution by #Christoph Forster but I received a script security error when Groovy Sandbox is enable
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use new hudson.FilePath java.io.File
However, it seems we can skip the file copying and deleting actions (and bypass the Groovy sandbox restriction) by simply requiring that the file is uploaded to the job workspace. Just add the workspace variable to the file name as follows:
stage("upload") {
def inputFile = input message: 'Upload file', parameters: [file(name: "$workspace/data.zip")]
}
I found a solution in the form of a global library here:
https://bitbucket.org/janvrany/jenkins-27413-workaround-library/src/default/
It contains calls to inner methods of Jenkins which are deprecated (I guess).
So I made my own version like this:
import hudson.FilePath
import hudson.model.ParametersAction
import hudson.model.FileParameterValue
import hudson.model.Executor
def call(String name, String fname = null) {
def paramsAction = currentBuild.rawBuild.getAction(ParametersAction.class);
if (paramsAction == null) {
error "unstashParam: No file parameter named '${name}'"
}
for (param in paramsAction.getParameters()) {
if (param.getName().equals(name)) {
if (! param instanceof FileParameterValue) {
error "unstashParam: not a file parameter: ${name}"
}
if (env['NODE_NAME'] == null) {
error "unstashParam: no node in current context"
}
if (env['WORKSPACE'] == null) {
error "unstashParam: no workspace in current context"
}
workspace = new FilePath(getComputer(env['NODE_NAME']), env['WORKSPACE'])
filename = fname == null ? param.getOriginalFileName() : fname
file = workspace.child(filename)
file.copyFrom(param.getFile())
return filename;
}
}
}
def getComputer(name){
for(computer in Jenkins.getInstance().getComputers()){
if(computer.getDisplayName() == name){
return computer.getChannel()
}
}
error "Cannot find computer for file parameter workaround"
}
You can insert it in a global library and then use it like:
library "file-workaround"
node {
def file_in_workspace = unstashParam "myFile"
sh "cat ${file_in_workspace}"
}
It's not pretty but it's working and as long as there is no official fix, it's my best shot.
Update
Turns out you might run into "No such file or directory". That's because nothing in the workaround triggers Jenkins to create the workspace directory. If that was triggered somewhere else in the pipeline good, otherwise you'll be scratching your head.
You might wanna throw a
touch "thisIsAFile"
in there
To handle an optional file parameter in pipeline (to handle the use case where no file should be accepted) you could use jenkinsci-unstashParam-library (add it in Jenkins>Manage Jenkins>Configure System>Global Pipeline Libraries https://github.com/janvrany/jenkinsci-unstashParam-library) with a try/catch in a script as this sample stage:
stage('upload') {
steps {
// delete workspace
cleanWs()
// handle file parameters in pipeline (JENKINS-27413)
script {
try {
// force workspace directory creation
sh "touch emptyFileToCreateWorkspace"
// https://stackoverflow.com/questions/59468464/fetching-uploaded-files-in-jenkins
def file_in_workspace = unstashParam 'MY_FILE.xlsx'
// https://unix.stackexchange.com/questions/125776/error-with-a-file-name-containing-parentheses
sh "mv '${file_in_workspace}' MY_FILE.xlsx"
}
catch (Exception e) {
echo e.getMessage()
echo "No file parameter, we will continue.."
}
}
}
}
File parameters provides 2 alternative parameters types for files (stashed for large files and base64 for small files).
Example, for base64File:
node {
sh 'echo $FILE | base64 -d'
withFileParameter('FILE') {
sh 'cat $FILE'
}
}
and stashedFile:
node {
unstash 'FILE'
sh 'cat FILE'
}
Tried what Christoph suggested and it didnt work for me. Here is what worked for me and the setup which I have, his should help others figure out what to do.
Problem:
I am executing my pipeline on dedicated nodes and use sanitized workspaces. After some research and troubleshooting I found out that by default the file upload only works with Master node. I realized this after digging through the file system and finding the file I am uploading in the workspace on the master
Solution:
stage('Upload Key') {
agent { label 'master' }
steps {
script {
// Uploads file via master node and stases it for other nodes to access
def inputFile = input message: 'Upload file', parameters: [file(name: "key.p12")]
new hudson.FilePath(new File("${workspace}/key.p12")).copyFrom(inputFile)
inputFile.delete()
}
stash name: 'key.p12' , includes: "key.p12"
}
}
stage('Register') {
steps {
ws (sanitizedWorkspaceName) {
echo "Registering"
unstash 'key.p12'
}
}
}
Execute the suggested file copy solution by Christoph. This stores the file in the job workspace on the master node
Allow the scripts in Manage Jenkins > In Process Script approval
use the stash step to stash the uploaded file
In the target stage "running on a different node" use the unstash
Hope this helps
I wasn't able to make Christoph's solution working if the file was uploaded on master node and needed on slave. The solution was to stash it on master and later unstash it on slave. Don't forget to remove the uploaded file on master node.
It's supported by the latest File Parameters plugin now. Please refer to: How to pass a file parameter to another build job in jenkins pipeline?