I'm trying to setup a Jenkins server with EC2 plugin such that all builds are executed by the EC2 instances, which act as agents for the master server.
While trying to save the cloud configuration for 'Amazon EC2' cloud in the Jenkins master, it fails with an exception. Checking in the Jenkins logs, I see this exception:
Caught unhandled exception with ID f6d45d51-fb00-4d1c-a474-0a55dd5ee710
org.kohsuke.stapler.WrongTypeException: Got type array but no lister class found for type class java.lang.String
at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:724)
at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:478)
at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:787)
Caused: java.lang.IllegalArgumentException: Failed to convert the instanceCapStr parameter of
the constructor public hudson.plugins.ec2.AmazonEC2Cloud
(java.lang.String,boolean,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.util.List,java.lang.String,java.lang.String)
at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:789)
at org.kohsuke.stapler.RequestImpl.access$200(RequestImpl.java:83)
at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:678)
Caused: java.lang.IllegalArgumentException: Failed to instantiate class hudson.plugins.ec2.AmazonEC2Cloud from
{
"cloudName":"ec2-cloud",
"includeUser":["false","false"],
"credentialsId":"",
"useInstanceProfileForCredentials":true,
"altEC2Endpoint":"",
"region":"eu-west-1",
"sshKeysCredentialsId":"jenkins-slave-ssh-key",
"instanceCapStr":["",""],
"noDelayProvisioning":false,
"roleArn":"",
"roleSessionName":"",
"templates":{
"description":"Amazon Linux 2 AMI",
"ami":"ami-0bb3fad3c0286ebd5",
"type":"T2Micro",
"ebsOptimized":true,
"monitoring":true,
"t2Unlimited":false,
"zone":"",
"securityGroups":"",
"remoteFS":"/var/lib/jenkins",
"remoteAdmin":"ec2-user",
"":"0",
"amiType":{
"rootCommandPrefix":"",
"slaveCommandPrefix":"",
"slaveCommandSuffix":"",
"sshPort":"22",
"stapler-class":"hudson.plugins.ec2.UnixData",
"$class":"hudson.plugins.ec2.UnixData"
},
"labelString":"ec2",
"mode":"EXCLUSIVE",
"idleTerminationMinutes":"30",
"initScript":"",
"tmpDir":"",
"userData":"",
"numExecutors":"",
"jvmopts":"",
"stopOnTerminate":false,
"subnetId":"",
"useDedicatedTenancy":false,
"name":"","value":""
},
"minimumNumberOfInstances":"0",
"minimumNumberOfSpareInstances":"0",
"iamInstanceProfile":"arn:aws:iam::xxxxxxxxxxxx:instance-profile/jenkins_server_role",
"deleteRootOnTermination":true,
"useEphemeralDevices":true,
"customDeviceMapping":"",
"launchTimeoutStr":"",
"associatePublicIp":false,
"connectionStrategy":"PRIVATE_IP",
"connectBySSHProcess":false,
"hostKeyVerificationStrategy":"CHECK_NEW_HARD",
"maxTotalUses":"-1",
"nodeProperties":{"stapler-class-bag":"true"}
},"stapler-class":"hudson.plugins.ec2.AmazonEC2Cloud","$class":"hudson.plugins.ec2.AmazonEC2Cloud"}
at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:681)
at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:478)
at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:474)
at hudson.model.Descriptor.newInstance(Descriptor.java:598)
I do see the property 'Instance cap' in two different locations in the Jenkins UI. My understanding is that one is for configuring a max limit on the total number of instances allowed in the entire cloud,
and the other property describes a max limit on the number of instances for the particular AMI.
Is this a Jenkins issue? Or is it something wrong with the configuration I've provided?
NOTE: I'm okay with providing the configuration as code rather than via the Jenkins UI. If anyone is able to provide the same configuration through code, that is also fine.
I eventually ended up configuring this using a groovy script run in the Script Console in the Jenkins UI. The groovy script I used is:
import hudson.model.*
import jenkins.model.*
import hudson.plugins.ec2.*
import com.amazonaws.services.ec2.model.InstanceType
def instance = Jenkins.getInstance()
def ec2_cloud_name = 'ec2-cloud'
def ec2_instance_cap = 5
def worker_description = 'jenkins-slave running in ec2 instance'
def worker_label_string = 'ec2-slave'
def ami_id = 'ami-xxxxxxxxxxxxxxxxx'
def security_groups = 'sg-xxxxxxxxxxxxxxxxxx'
def subnet_id = 'subnet-xxxxxxxx'
def instance_type = 't2.micro'
def instance_profile_arn = 'arn:aws:iam::xxxxxxxxxxxx:instance-profile/jenkins_server_role'
def number_of_executors = 2
def ec2_tags = [
new EC2Tag('Name', 'jenkins-slave-instance')
]
def priv_key_txt = '''
-----BEGIN RSA PRIVATE KEY-----
<My Private key>
-----END RSA PRIVATE KEY-----
'''
def worker_ami = new SlaveTemplate(
// String ami
ami_id,
// String zone
'',
// SpotConfiguration spotConfig
null,
// String securityGroups
security_groups,
// String remoteFS
'',
// InstanceType type
InstanceType.fromValue(instance_type),
// boolean ebsOptimized
false,
// String labelString
worker_label_string,
// Node.Mode mode
Node.Mode.NORMAL,
// String description
worker_description,
// String initScript
'',
// String tmpDir
'',
// String userData
'',
// String numExecutors
"${number_of_executors}",
// String remoteAdmin
'',
// AMITypeData amiType
new UnixData(null, null, null, null),
// String jvmopts
'',
// boolean stopOnTerminate
false,
// String subnetId
subnet_id,
// List<EC2Tag> tags
ec2_tags,
// String idleTerminationMinutes
'30',
// int minimumNumberOfInstances
0,
// int minimumNumberOfSpareInstances
0,
// string instanceCapStr
'3',
// string iamInstanceProfile
'arn:aws:iam::xxxxxxxxxxxx:instance-profile/jenkins_server_role',
// boolean deleteRootOnTermination
true,
// boolean useEphemeralDevices
true,
// boolean useDedicatedTenancy
false,
// String launchTimeoutStr
'1800',
// boolean associatePublicIp
false,
// String customDeviceMapping
'',
// boolean connectBySSHProcess
false,
// boolean monitoring
false,
// boolean t2Unlimited
false,
// Enum connectionStrategy
ConnectionStrategy.PRIVATE_IP,
// int maxTotalUses
3,
// List<? extends NodeProperty<?>> nodeProperties
[],
// HostKeyVerificationStrategyEnum
HostKeyVerificationStrategyEnum.CHECK_NEW_HARD
)
def new_cloud = new AmazonEC2Cloud(
// String cloudName
ec2_cloud_name,
// boolean useInstanceProfileForCredentials
true,
// String credentialsId
'',
// String region
'eu-west-1',
// String privateKey
priv_key_txt,
// String sshKeysCredentialsId
'jenkins-slave-ssh-key',
// String instanceCapStr
"3",
// List<? extends SlaveTemplate> templates
[worker_ami],
// String roleArn
null,
// String roleSessionName
null
)
instance.clouds.add(new_cloud)
Strange thing is, after creating the config by running this script, now I am able to edit and save the created config via the Jenkins UI.
Related
I'm trying to test a binary with latest commit id in Jenkins. Error happens at send slack message stage:
def kResPath = "/tmp/res.json" // global variable, where json file is dumped to; declared at the very beginning of Jenkinsfile
def check_result_and_notify(tier, result) {
def kExpected = 3
def kSuccessSlackColor = "#00CC00"
message = "Test ${tier} result: ${result}\n"
def test_result = readJSON file: kResPath
// 0 means index, "benchmarks", "real-time" is the key
real_time = test_result["benchmarks"][0]["real_time"]
if (real_time > kExpected) {
message += String.format("real time = %f, expected time = %f", real_time, kExpected)
}
slackSend(color: ${kSuccessSlackColor}, message: message.trim(), channel: "test-result")
}
The json file looks like:
{
"benchmarks": [
{
"real_time": 4,
},
{
"real_time": 5,
}
],
}
The error message I've received is hudson.remoting.ProxyException: groovy.lang.MissingPropertyException: No such property: kResPath for class: WorkflowScript
Can someone tell me what's wrong with my code?
Is there any way I could test it locally so that I don't need to commit it every single time? I googled and find it needs server id and password, which I don't think accessible to me :(
Your kResPath variable is undefined in the scope of that function or method (unsure which based on context). You can pass it as an argument:
def check_result_and_notify(tier, result, kResPath) {
...
}
check_result_and_notify(myTier, myResult, kResPath)
and even specify a default if you want:
def check_result_and_notify(tier, result, kResPath = '/tmp/res.json')
Using Timestamper plugin 1.11.2 with globally enabled timestamps, using the default format, I get the following console output:
00:00:41.097 Some Message
In Blue Ocean the output shows like:
[2020-04-01T00:00:41.097Z] Some Message
How can I make it so that Blue Ocean uses the short timestamp format? The long format is somewhat unreadable and clutters the details view of the steps.
I've looked at the Pipeline Options too, but there is only the timestamps option which doesn't have a parameter to specify the format.
Note: This question isn't a dupe, because it asks for differences in time zone only.
Edit:
⚠️ Unfortunately this workaround doesn't work in the context of node, see JENKINS-59575. Looks like I have to finally get my hands dirty with plugin development, to do stuff like that in a supported way.
Anyway, I won't delete this answer, as the code may still be useful in other scenarios.
Original answer:
As a workaround, I have created a custom ConsoleLogFilter. It can be applied as a pipeline option, a stage option or at the steps level. If you have the timestamp plugin installed, you should disable the global timestamp option to prevent duplicate timestamps.
Typically you would define the low-level code in a shared library. Here is a sample that can be copy-pasted right into the pipeline script editor (you might have to disable Groovy sandbox):
import hudson.console.LineTransformationOutputStream
import hudson.console.ConsoleLogFilter
import java.nio.charset.Charset
import java.nio.charset.StandardCharsets
pipeline{
agent any
/*
options{
// Enable timestamps for the whole pipeline, using default format
//withContext( myTimestamps() )
// Enable timestamps for the whole pipeline, using custom format
//withContext( myTimestamps( dateFormat: 'HH:mm:ss', prefix: '', suffix: ' - ' ) )
}
*/
stages {
stage('A') {
options {
// Enable timestamps for this stage only
withContext( myTimestamps() )
}
steps {
echo 'Hello World'
}
}
stage('B') {
steps {
echo 'Hello World'
// Enable timestamps for some steps only
withMyTimestamps( dateFormat: 'HH:mm:ss') {
echo 'Hello World'
}
}
}
}
}
//----- Code below should be moved into a shared library -----
// For use as option at pipeline or stage level, e. g.: withContext( myTimestamps() )
def myTimestamps( Map args = [:] ) {
return new MyTimestampedLogFilter( args )
}
// For use as block wrapper at steps level
void withMyTimestamps( Map args = [:], Closure block ) {
withContext( new MyTimestampedLogFilter( args ), block )
}
class MyTimestampedLogFilter extends ConsoleLogFilter {
String dateFormat
String prefix
String suffix
MyTimestampedLogFilter( Map args = [:] ) {
this.dateFormat = args.dateFormat ?: 'YY-MM-dd HH:mm:ss'
this.prefix = args.prefix ?: '['
this.suffix = args.suffix ?: '] '
}
#NonCPS
OutputStream decorateLogger( AbstractBuild build, OutputStream logger )
throws IOException, InterruptedException {
return new MyTimestampedOutputStream( logger, StandardCharsets.UTF_8, this.dateFormat, this.prefix, this.suffix )
}
}
class MyTimestampedOutputStream extends LineTransformationOutputStream {
OutputStream logger
Charset charset
String dateFormat
String prefix
String suffix
MyTimestampedOutputStream( OutputStream logger, Charset charset, String dateFormat, String prefix, String suffix ) {
this.logger = logger
this.charset = charset
this.dateFormat = dateFormat
this.prefix = prefix
this.suffix = suffix
}
#NonCPS
void close() throws IOException {
super.close();
logger.close();
}
#NonCPS
void eol( byte[] bytes, int len ) throws IOException {
def lineIn = charset.decode( java.nio.ByteBuffer.wrap( bytes, 0, len ) ).toString()
def dateFormatted = new Date().format( this.dateFormat )
def lineOut = "${this.prefix}${dateFormatted}${this.suffix}${lineIn}\n"
logger.write( lineOut.getBytes( charset ) )
}
}
Example output for stage "B":
Credits:
I got the idea from this answer.
I have Jenkins pipeline set up for Git branches with the last optional step of deploying to stage:
stage('Stage') {
if (gitBranch != "master") {
timeout(time: 1, unit: 'DAYS') {
input message: "Do you want to deploy ${shortCommit} from branch ${gitBranch} to STAGE?"
}
}
node {
stage('Deploy Stage') {
echo("Deploying to STAGE ${gitCommit}")
sh "NODE_ENV=stage yarn lerna-run --since ${sinceSha} deploy"
}
}
}
The problem is deploying a branch to stage is optional, but Jenkins doesn't return a success code to Github until it's done.
Is there any syntax to mark it as optional?
You can combine the timeout step with the input step, like we have it here:
/**
* Generates a pipeline {#code input} step that times out after a specified amount of time.
*
* The options for the timeout are supplied via {#code timeoutOptions}.
* The options for the input dialog are supplied via {#code inputOptions}.
*
* The returned Map contains the following keys:
*
* - proceed: true, if the Proceed button was clicked, false if aborted manually aborted or timed out
* - reason: 'user', if user hit Proceed or Abort; 'timeout' if input dialog timed out
* - submitter: name of the user that submitted or canceled the dialog
* - additional keys for every parameter submitted via {#code inputOptions.parameters}
*
* #param args Map containing inputOptions and timoutOptions, both passed to respective script
* #return Map containing above specified keys response/reason/submitter and those for parameters
*/
Map inputWithTimeout(Map args) {
def returnData = [:]
// see https://go.cloudbees.com/docs/support-kb-articles/CloudBees-Jenkins-Enterprise/Pipeline---How-to-add-an-input-step,-with-timeout,-that-continues-if-timeout-is-reached,-using-a-default-value.html
try {
timeout(args.timeoutOptions) {
def inputOptions = args.inputOptions
inputOptions.submitterParameter = "submitter"
// as we ask for the submitter, we get a Map back instead of a string
// besides the parameter supplied using args.inputOptions, this will include "submitter"
def responseValues = input inputOptions
echo "Response values: ${responseValues}"
// BlueOcean currently drops the submitterParameter
// https://issues.jenkins-ci.org/browse/JENKINS-41421
if (responseValues instanceof String) {
echo "Response is a String. BlueOcean? Mimicking the correct behavior."
String choiceValue = responseValues
String choiceKey = args.inputOptions.parameters.first().getName()
responseValues = [(choiceKey): choiceValue, submitter: null]
}
echo "Submitted by ${responseValues.submitter}"
returnData = [proceed: true, reason: 'user'] + responseValues
}
} catch (FlowInterruptedException err) { // error means we reached timeout
// err.getCauses() returns [org.jenkinsci.plugins.workflow.support.input.Rejection]
Rejection rejection = err.getCauses().first()
if ('SYSTEM' == rejection.getUser().toString()) { // user == SYSTEM means timeout.
returnData = [proceed: false, reason: 'timeout']
} else { // explicitly aborted
echo rejection.getShortDescription()
returnData = [proceed: false, reason: 'user', submitter: rejection.getUser().toString()]
}
} catch (err) {
// try to figure out, what's wrong when we manually abort the pipeline
returnData = [proceed: false, reason: err.getMessage()]
}
returnData
}
In addition to your requirements, this also returns, who submitted the dialog.
The script is not iterating through all the values of the 'modules' array.
class Module {
public String name = '';
public Boolean isCustom = false;
public Module(String name, Boolean custom){
this.name = name;
this.isCustom = custom;
}
}
//creates array from the ext_module env var
modules = [];
EXT_MODULE.split(',').each {
modules.add(new Module(it, false));
}
println modules;
modules.each {
println "MODULE NAME ::::: ${it.name}"
if(it.isCustom)
{
println "install custom";
} else {
println "install non custom";
}
};
This is the result of the run. The array shows 4 elements, but the code inside the .each black only executes once.
Running: Print Message
[Module#71f09325, Module#e1ddb41, Module#7069a674, Module#1f68f952]
Running: Print Message
MODULE NAME ::::: puppetlabs-ntp
Running: Print Message
install non custom
Running: End of Workflow
Finished: SUCCESS
The messages "Running: Print Message" and "Running: End of Workflow" indicate that you are using the new workflow plugin: https://wiki.jenkins-ci.org/display/JENKINS/Workflow+Plugin. This plugin currently has a bug causing at least some Groovy iterations involving a closure to be aborted after one iteration: https://issues.jenkins-ci.org/browse/JENKINS-26481
The workaround is to simply use an old school for loop (code below).
Also, NonCPS is another workaround.
There is an open issue for this matter. See here: https://issues.jenkins-ci.org/browse/JENKINS-26481
Update, Oct 24th, 2016
/**
* Dumps environment varibles to the log, using an old school for loop.
*/
import com.cloudbees.groovy.cps.NonCPS
def version = '1.0'
#NonCPS
def dumpEnvVars() {
def str = "Dumping build environment variables...\n"
for (Map.Entry<String, String> entry : currentBuild.build().environment) {
str += " ${entry.key} = ${entry.value}\n"
}
echo str
}
return this;
As of yesterday, the new Pipeline plugin was delivered in version 2.0 and correct this problem.
.each closures now work, but .collect still only iterate once.
I've got groovy class which is calling AntBuilder.sshexec method to execute remote command. This is working fine when I'm running this Groovy class as Java application. But, when I'm trying to call this class/method from controller, I've got error "Could not create type sshexec due to java.lang.NoClassDefFoundError: com/jcraft/jsch/UserInfo".
Groovy class:
package filemanager
import com.jcraft.jsch.*
import com.jcraft.jsch.ChannelSftp.*
class FileWork {
String servName
String servUser
String servPassword
String servFolder
String localFolder
int servPort
String fileName
FileWork (String p_servName, String p_servUser, String p_servPassword, String p_servFolder, String p_localFolder, int p_servPort,String p_fileName) {
println 'Exec constructor'
this.servName = p_servName
this.servUser = p_servUser
this.servPassword = p_servPassword
this.servFolder = p_servFolder
this.localFolder = p_localFolder
this.servPort = p_servPort
this.fileName = p_fileName
}
.....
String runRemoteCommand () {//(Session p_ses) {
try {
def result = ''
def ant = new AntBuilder()
ant.sshexec(
host: servName,
port: servPort,
trust: true,
username: servUser,
password: servPassword,
command: "unzip -o ${servFolder}${fileName} -d ${servFolder}",
outputproperty: 'result',
verbose: false
)
return result
} catch (Exception e) {
println 'This is filemanager.FileWork.runRemoteCommandException'
e.printStackTrace();
}
}
}
Controller:
package filemanager
import com.jcraft.jsch.*
import com.jcraft.jsch.ChannelSftp.*
class ChooseToController {
def index(params) {
params.max = Math.min(max ?: 10, 100)
//render params
//model:[destServ: DestServ, fileName:fileName]
}
def copyToRemote(params) {
def destServ = DestServ.get(params.id)
//FileWork fileWork = new FileWork (destServ.getDestServName(), destServ.getDestServUser(), destServ.getDestServPassword(), destServ.getDestServFolder(), "C:/tmp/", destServ.getDestServPort(), params.fileName)
//Session ses = fileWork.connect()
//fileWork.copyToRemoteServ(ses)
//ses.disconnect()
FileWork fileWork3 = new FileWork ("###########", "test", "Test123", "/home/test/IN/", "C:/tmp/", 22, "1.zip")
String result = fileWork3.runRemoteCommand()
println(result)
}
}
Dependencies:
runtime "com.jcraft:jsch:0.1.51"
runtime "org.apache.ant:ant-jsch:1.8.1"
Error:
Could not create type sshexec due to java.lang.NoClassDefFoundError: com/jcraft/jsch/UserInfo
at org.apache.tools.ant.AntTypeDefinition.createAndSet(AntTypeDefinition.java:278)
at org.apache.tools.ant.AntTypeDefinition.icreate(AntTypeDefinition.java:219)
at org.apache.tools.ant.AntTypeDefinition.create(AntTypeDefinition.java:206)fro.....
Seems that not all classes are visible from grails runtime context...