My sample yaml file looks like:
cluster:
a: val1
b: val2
c: val3
list:
k1:
- n11
- n2
k2:
- n1
So, the jenkins pipeline takes 2 inputs, i.e., the name of the input1 and input2. Post creation, I need to update the yaml file. I need to add the input1 based on input2, if input2=k1, I need to input1 in k1 section. I'm trying to execute the following command in my jenkins pipeline:
sh "a=$Input1 b=$Input2 yq -i '.cluster.list[strenv(b)] += strenv(a)' sample.yaml"
But it is throwing error:
jq: error: strenv/1 is not defined at <top-level>, line 1:
.cluster.vms[strenv(a)] += strenv(b)
Case 1: if your variable is declared in the pipeline:
yq -iY .cluster.list[$key] += $val
where, -iY will edit the yaml file, $key holds the value of input1 and $val holds the value of input2
Case 2: if $key and $val are not declared, following commands work:
yq -iY --arg key k1(or k2) --arg val value .cluster.list[$key] += $val
Related
I am writing a bwa mapping module using nextflow (dsl=2), modules/map_reads.nf to map single-end reads. When I execute this workflow it does not return error from the terminal and it also output bam files with the correct file names. However, I found that the bam files are not correctly mapped and I also found in .command.err an error:
[E::bwa_idx_load_from_disk] fail to locate the index files
I have checked the paths are correct and also execute shell command directly in terminal.
I appreciate any suggestions or solution to this problem.
modules/map_reads.nf
#!/usr/bin/env nextflow
nextflow.enable.dsl=2
process mapping {
conda 'envs/bwa.yml'
publishDir 'results/mapped', mode: 'copy'
input:
tuple val(sample_id), file(fastq)
file index
output:
tuple val(sample_id), file('*.bam')
script:
"""
bwa mem $index $fastq | samtools view -b - > ${sample_id}.bam
"""
}
workflow {
fastq_data = channel.fromPath( 'data/samples/*.fastq' ).map { file -> tuple(file.baseName, file) }
index = channel.fromPath( 'data/genome.fa' )
mapping( fastq_data, index )
}
Here is my directory structure:
envs/bwa.yml
name: bwa
channels:
- bioconda
- defaults
dependencies:
- bwa
- samtools=1.9
[E::bwa_idx_load_from_disk] fail to locate the index files
BWA MEM is expecting a number of index files to be provided with it's first argument, but you've only localized the genome FASTA file:
index = channel.fromPath( 'data/genome.fa' )
BWA MEM only requires the actual index files. I.e. it doesn't require the FASTA file (or FASTA index), so you can save some time and resources by skipping the localization of the FASTA (this is especially relevant if you localize from s3 for example, since the FASTA file is often a couple of gigabytes). Also ensure you use a value channel here, so that the channel can be used an unlimited number of times:
process mapping {
conda 'envs/bwa.yml'
publishDir 'results/mapped', mode: 'copy'
input:
tuple val(sample_id), path(fastq)
path bwa_index
output:
tuple val(sample_id), path("${sample_id}.bam")
script:
def idxbase = bwa_index[0].baseName
"""
bwa mem "${idxbase}" "${fastq}" | samtools view -b - > "${sample_id}.bam"
"""
}
workflow {
fastq_data = channel.fromPath( 'data/samples/*.fastq' ).map { file ->
tuple(file.baseName, file)
}
bwa_index = file( 'data/genome.fa.{,amb,ann,bwt,pac,sa}' )
mapping( fastq_data, bwa_index )
}
Also, the reason you didn't see an error on the command line is that, by default, the return value of a pipeline is the exit status of the last command. In your BWA MEM SAMtools pipeline, the last command (i.e. samtools) completes successfully and returns exit status zero. The option you want to add is called 'pipefail', man bash:
pipefail
If set, the return value of a pipeline is the value of the
last (rightmost) command to exit with a non-zero status, or zero if
all commands in the pipeline exit successfully. This option is
disabled by default.
Usually, you can just add the following to your nextflow.config to have it applied to all processes:
process {
shell = [ '/bin/bash', '-euo', 'pipefail' ]
}
I have to parse a config with section values in Jenkins Pipeline . Below is the example config file
[deployment]
10.7.1.14
[control]
10.7.1.22
10.7.1.41
10.7.1.17
[worker]
10.7.1.45
10.7.1.42
10.7.1.49
10.7.1.43
10.7.1.39
[edge]
10.7.1.13
Expected Output:
control1 = 10.7.1.17 ,control2 = 10.7.1.22 ,control3 = 10.7.1.41
I tried the below code in my Jenkins Pipeline script section . But it seems to be incorrect function to use
def cluster_details = readProperties interpolate: true, file: 'inventory'
echo cluster_details
def Var1= cluster_details['control']
echo "Var1=${Var1}"
Could you please help me with the approach to achieve the expected result
Regarding to documentation readProperties is to read Java properties file. But not INI files.
https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#readproperties-read-properties-from-files-in-the-workspace-or-text
I think to read INI file you have find available library for that,
e.g. https://ourcodeworld.com/articles/read/839/how-to-read-parse-from-and-write-to-ini-files-easily-in-java
Hi i got the solution for the problem
control_nodes = sh (script: """
manish=\$(ansible control -i inventory --list-host |sort -t . -g -k1,1 -k2,2 -k3,3 -k4,4 |awk '{if(NR>1)print}' |awk '{\$1=\$1;print}') ; \
echo \$manish
""",returnStdout: true).trim()
echo "Cluster Control Nodes are : ${control_nodes}"
def (control_ip1,control_ip2,control_ip3) = control_nodes.split(' ')
//println c1 // this also works
echo "Control 1: ${control_ip1}"
echo "Control 2: ${control_ip2}"
echo "Control 3: ${control_ip3}"
Explaination:
In the script section . I am getting the list of hostnames.Using sort i am sorting the hostname based on dot(.) delimeter. then using awk removing the first line in output. Using the later awk i am removing the leading white spaces.
Using returnStdout to save the shell variable output to jenkins property, which has list of ips separated by white space.
Now once i have the values in jenkins property variable, extracting the individual IPs using split methods.
Hope it helps.
I am using simple grep like
grep -E "input files : 0" *.stdout
and its output is following:
QCD_HT200to300_13TeV_ext.stdout:==> Total number of input files : 0
WJetsToLNu_HT_600To800_13TeV_ext1.stdout:==> Total number of input files : 0
If I use the same grep command within the for loop then its output is not nice. My script is
foreach file ( `grep -E "input files : 0" *.stdout` )
echo $file
end
the output of this is like:
QCD_HT200to300_13TeV_ext.stdout:==>
Total
number
of
input
files
:
0
WJetsToLNu_HT_600To800_13TeV_ext1.stdout:==>
Total
number
of
input
files
:
0
How I can get the same output within foreach loop as I get from grep command running independently.
Use the list of files to iterate through in the foreach loop and do the grep inside the foreach loop.
set files = `ls *.stdout`
foreach file ( $files )
grep -E "input files : 0” $file
end
my build steps:
cd $WORKSPACE
export TERM="linux"
. venv/bin/activate
pylint --rcfile=pylint.cfg $(find handlers -maxdepth 1 -name "*.py" -print) > pylint.log || exit 0
result of pylint.log:
************* Module handlers
C: 1, 0: Missing module docstring (missing-docstring)
C: 8, 0: Missing function docstring (missing-docstring)
************* Module handlers.foo
C: 1, 0: Black listed name "foo" (blacklisted-name)
C: 1, 0: Missing module docstring (missing-docstring)
C: 1, 0: Missing function docstring (missing-docstring)
E: 2,11: Undefined variable 'a' (undefined-variable)
E: 2,13: Undefined variable 'b' (undefined-variable)
Report
======
...
(the report continues with statistics by type, raw metrics, external dependencies)
the xml filename pattern for pylint is:
**/pylint.log
with the source path pattern being:
**/
Even after all this, and with pylint.log showing I have lint errors, the graph shows nothing.
any ideas how to get pylint and the violations plugin working nicely together?
it seems the correct pylint command is the following:
pylint --rcfile=pylint.cfg $(find handlers -maxdepth 1 -name "*.py" -print) --msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}" > pylint.log || exit 0
note the addition of the --msg-template param
I had this problem myself. I think it was a result of the way I installed the Violations plugin. It worked after restarting Jenkins:
$ sudo service jenkins restart
It seems that there is some work in progress to add support for this in the future:
https://github.com/fish-shell/fish-shell/issues/478
https://github.com/xiaq/fish-shell/tree/opt-parse
But in the meantime, what's the recommended way to deal with this? Should I just parse $argv? If so do you have some tips/best practices?
Parsing $argv is fine for basic scenarios but can otherwise become tedious and error prone.
Before fish had its own argument parsing solution, the community created:
getopts.fish
As of fish 2.7.0, you can use fish's built-in option parser: argparse
function foo --description "Example argparse usage"
set --local options 'h/help' 'n/count=!_validate_int --min 1'
argparse $options -- $argv
if set --query _flag_help
printf "Usage: foo [OPTIONS]\n\n"
printf "Options:\n"
printf " -h/--help Prints help and exits\n"
printf " -n/--count=NUM Count (minimum 1, default 10)"
return 0
end
set --query _flag_count; or set --local _flag_count 10
for i in (seq $_flag_count); echo foo; end
end
To see the full extent of what's possible, run argparse -h or argparse --help.
I'm note sure if it is a best practice, but in the meantime you could do something like this:
function options
echo $argv | sed 's|--*|\\'\n'|g' | grep -v '^$'
end
function function_with_options
for i in (options $argv)
echo $i | read -l option value
switch $option
case a all
echo all the things
case f force
echo force it
case i ignore
echo ignore the $value
end
end
end
Output:
➤ function_with_options -i thing -a --force
ignore the thing
all the things
force it
You can do this:
for item in $argv
switch "$item"
case -f --foo
case -b --bar
end
end
The above does not support writing short options in a single argument -fbz, option values --foo=baz, --foo baz or f baz, negative assignment --name!=value, end of options -- and dashes - and -- are always part of the argument.
To address these issues, I wrote a getopts function.
getopts -ab1 --foo=bar baz
Now look at the output.
a
b 1
foo bar
_ baz
The items on the left represent the option flags or keys associated with the CLI. The items on the right are the option values. The underscore _ character is the default key for arguments without a key.
Use read(1) to process the generated stream and switch(1) to match patterns:
getopts -ab1 --foo=bar baz | while read -l key option
switch $key
case _
case a
case b
case foo
end
end
See the documentation.