Using variable in jenkins file for robocopy command - jenkins

I am writing a Jenkins pipeline script to copy files from the source to the destination folder. The only difference from lots of articles I already visited is that my source and destination paths are not fixed but constructed using variables. I have a very strange issue that the variable names are used instead of their value, have been trying since the whole day. My Jenkins script is below
pipeline
{
agent any
parameters
{
string(name: 'SPRINT_NUMBER', defaultValue: '', description: 'Sprint Number')
}
environment
{
DESTINATION_PATH=""
SOURCE_PATH=""
}
stages
{
stage('Copy')
{
steps
{
script
{
/* APPROACH-1*/
DESTINATION_PATH="C:\\Release\\" + "${env.SPRINT_NUMBER}" + "\\" + "${env.BUILD_NUMBER}" +"\\Destination"
SOURCE_PATH="C:\\Source\\bin"
echo "DESTINATION_PATH: ${DESTINATION_PATH}"
echo "DESTINATION_PATH: ${SOURCE_PATH}"
/* the below doesnt work, echo prints correct DESTINATION_PATH and SOURCE_PATH though*/
bat '(robocopy ${SOURCE_PATH} ${DESTINATION_PATH}) ^& IF %ERRORLEVEL% LEQ 1 exit 0)'
/* APPROACH-2*/
bat label: '', script: '''Set src="C:\\Source\\bin"
Set dest="C:\\Release\\${params.SPRINT_NUMBER}\\${BUILD_NUMBER}\\Destination"
/* the below doesnt work, the variable name used instead of values*/
robocopy %src% %dest%'''
}
}
}
}
}
The Console out put from Jenkins is
For first approach
7:00:58 -------------------------------------------------------------------------------
17:00:58 ROBOCOPY :: Robust File Copy for Windows
17:00:58 -------------------------------------------------------------------------------
17:00:58
17:00:58 Started : Tuesday, July 28, 2020 5:00:58 PM
17:00:58 Source : C:\Program Files (x86)\Jenkins\workspace\CopyFilesJob\${SOURCE_PATH}\
17:00:58 Dest : C:\Program Files (x86)\Jenkins\workspace\CopyFilesJob\${DESTINATION_PATH}\
17:00:58
17:00:58 Files : .
17:00:58
17:00:58 Options : . /DCOPY:DA /COPY:DAT /R:1000000 /W:30
For second approach
17:00:58 -------------------------------------------------------------------------------
17:00:58 ROBOCOPY :: Robust File Copy for Windows
17:00:58 -------------------------------------------------------------------------------
17:00:58
17:00:58 Started : Tuesday, July 28, 2020 5:00:58 PM
17:00:58 Source : C:\Source\bin\
17:00:58 Dest : C:\Release\${params.SPRINT_NUMBER}\${BUILD_NUMBER}\Destination\
17:00:58
17:00:58 Files : .
17:00:58
17:00:58 Options : . /DCOPY:DA /COPY:DAT /R:1000000 /W:30

Related

why else block not executing scripted pipeline?

I have this if else statement if the condition is true the if statement executed but if the condition is false the else statement not executed
steps{
script{
echo '*Stage3 - Trying to build python program and export the Artifact*'
pyInstallerOutput = bat(returnStdout: true, script: '#python -m PyInstaller -v').trim()
if(!("$pyInstallerOutput".contains('No module named'))){
echo "+ PyInstaller Version: $pyInstallerOutput is up and running. good! building artifact"
directoryName = bat(returnStdout: true,script: '#echo %date:~-4,4%-%date:~-10,2%-%date:~7,2%-%time:~-11,2%%time:~-8,2%%time:~-5,2%').trim()
echo ("+ Current Artifact directory: $directoryName")
artifactResult = bat(returnStdout: true, script: "#python -m PyInstaller --specpath ./artifacts-repo/${directoryName}/spec --distpath ./artifacts-repo/${directoryName}/dist --workpath ./artifacts-repo/${directoryName}/build --onefile ./codes/SayHello.py")
} else{
echo '- PyInstaller not installed on node, trying to install it'
installResult = bat(returnStdout: true, script: '#pip install pyinstaller')
if("$installResult".contains('Successfully installed')){
echo '+ PyInstaller installed successfully, Now building Artifact.'
directoryName = bat(returnStdout: true,script: '#echo %date:~-4,4%-%date:~-10,2%-%date:~7,2%-%time:~-11,2%%time:~-8,2%%time:~-5,2%')
echo ("+ Current Artifact directory: $directoryName")
artifactResult = bat(
returnStdout: true,
script: "#python -m PyInstaller --specpath ./artifacts-repo/${directoryName}/spec --distpath ./artifacts-repo/${directoryName}/dist --workpath ./artifacts-repo/${directoryName}/build --onefile ./codes/SayHello.py"
)
}else{
error('- PyInstaller installation failed, nothing more can be done by pipeline, STOPING')
}
}
}
}
Log when the if condition is true
*Stage3 - Trying to build python program and export the Artifact*
[Pipeline] bat
[Pipeline] echo
+ PyInstaller Version: 5.5 is up and running. good! building artifact
[Pipeline] bat
[Pipeline] echo
+ Current Artifact directory: 2022-10-09-211243
[Pipeline] bat
376 INFO: PyInstaller: 5.5
376 INFO: Python: 3.10.7
384 INFO: Platform: Windows-10-10.0.22000-SP0
386 INFO: wrote ./artifacts-repo/2022-10-09-211243/spec\SayHello.spec
387 INFO: UPX is not available.
389 INFO: Extending PYTHONPATH with paths
['C:\\Users\\Ata '
'System\\AppData\\Local\\Jenkins\\.jenkins\\workspace\\Pipeline-01\\codes']
840 INFO: checking Analysis
841 INFO: Building Analysis because Analysis-00.toc is non existent
841 INFO: Initializing module dependency graph...
843 INFO: Caching module graph hooks...
854 INFO: Analyzing base_library.zip ...
3125 INFO: Loading module hook 'hook-heapq.py' from 'C:\\Users\\Ata System\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python310\\site-packages\\PyInstaller\\hooks'...
3188 INFO: Loading module hook 'hook-encodings.py' from 'C:\\Users\\Ata System\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python310\\site-packages\\PyInstaller\\hooks'...
4729 INFO: Loading module hook 'hook-pickle.py' from 'C:\\Users\\Ata System\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python310\\site-packages\\PyInstaller\\hooks'...
5910 INFO: Caching module dependency graph...
6004 INFO: running Analysis Analysis-00.toc
6011 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
required by C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2032.0_x64__qbz5n2kfra8p0\python.exe
6207 INFO: Analyzing C:\Users\Ata System\AppData\Local\Jenkins\.jenkins\workspace\Pipeline-01\codes\SayHello.py
6209 INFO: Processing module hooks...
6219 INFO: Looking for ctypes DLLs
6221 INFO: Analyzing run-time hooks ...
6222 INFO: Including run-time hook 'C:\\Users\\Ata System\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python310\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_inspect.py'
6224 INFO: Including run-time hook 'C:\\Users\\Ata System\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python310\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_subprocess.py'
6228 INFO: Looking for dynamic libraries
365 INFO: Extra DLL search directories (AddDllDirectory): []
366 INFO: Extra DLL search directories (PATH): ['C:\\Program Files (x86)\\VMware\\VMware Workstation\\bin\\', 'C:\\WINDOWS\\system32', 'C:\\WINDOWS', 'C:\\WINDOWS\\System32\\Wbem', 'C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\', 'C:\\WINDOWS\\System32\\OpenSSH\\', 'C:\\Program Files\\NVIDIA Corporation\\NVIDIA NvDLISR', 'C:\\Program Files\\Microsoft VS Code\\bin', 'C:\\Program Files (x86)\\Xshell 7\\', 'C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common', 'C:\\Program Files\\Git\\cmd', 'C:\\Users\\Ata System\\AppData\\Local\\Microsoft\\WindowsApps', 'C:\\Program Files\\JetBrains\\PhpStorm 2022.2.1\\bin', 'C:\\xampp\\php', 'C:\\Users\\Ata System\\AppData\\Local\\GitHubDesktop\\bin', 'C:\\apps\\opt\\spark-3.3.0-bin-hadoop3\\bin', 'C:\\apps\\opt\\spark-3.3.0-bin-hadoop3\\bin', 'C:\\Program Files\\Java\\jdk-17\\bin']
7142 INFO: Looking for eggs
7142 INFO: Using Python library C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2032.0_x64__qbz5n2kfra8p0\python310.dll
7142 INFO: Found binding redirects:
[]
7144 INFO: Warnings written to C:\Users\Ata System\AppData\Local\Jenkins\.jenkins\workspace\Pipeline-01\artifacts-repo\2022-10-09-211243\build\SayHello\warn-SayHello.txt
7161 INFO: Graph cross-reference written to C:\Users\Ata System\AppData\Local\Jenkins\.jenkins\workspace\Pipeline-01\artifacts-repo\2022-10-09-211243\build\SayHello\xref-SayHello.html
7188 INFO: checking PYZ
7188 INFO: Building PYZ because PYZ-00.toc is non existent
7188 INFO: Building PYZ (ZlibArchive) C:\Users\Ata System\AppData\Local\Jenkins\.jenkins\workspace\Pipeline-01\artifacts-repo\2022-10-09-211243\build\SayHello\PYZ-00.pyz
7392 INFO: Building PYZ (ZlibArchive) C:\Users\Ata System\AppData\Local\Jenkins\.jenkins\workspace\Pipeline-01\artifacts-repo\2022-10-09-211243\build\SayHello\PYZ-00.pyz completed successfully.
7397 INFO: checking PKG
7398 INFO: Building PKG because PKG-00.toc is non existent
7398 INFO: Building PKG (CArchive) SayHello.pkg
8688 INFO: Building PKG (CArchive) SayHello.pkg completed successfully.
8690 INFO: Bootloader C:\Users\Ata System\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe
8690 INFO: checking EXE
8690 INFO: Building EXE because EXE-00.toc is non existent
8690 INFO: Building EXE from EXE-00.toc
8690 INFO: Copying bootloader EXE to C:\Users\Ata System\AppData\Local\Jenkins\.jenkins\workspace\Pipeline-01\artifacts-repo\2022-10-09-211243\dist\SayHello.exe.notanexecutable
8693 INFO: Copying icon to EXE
8694 INFO: Copying icons from ['C:\\Users\\Ata System\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python310\\site-packages\\PyInstaller\\bootloader\\images\\icon-console.ico']
8705 INFO: Writing RT_GROUP_ICON 0 resource with 104 bytes
8705 INFO: Writing RT_ICON 1 resource with 3752 bytes
8705 INFO: Writing RT_ICON 2 resource with 2216 bytes
8705 INFO: Writing RT_ICON 3 resource with 1384 bytes
8705 INFO: Writing RT_ICON 4 resource with 37019 bytes
8705 INFO: Writing RT_ICON 5 resource with 9640 bytes
8705 INFO: Writing RT_ICON 6 resource with 4264 bytes
8705 INFO: Writing RT_ICON 7 resource with 1128 bytes
8707 INFO: Copying 0 resources to EXE
8707 INFO: Embedding manifest in EXE
8707 INFO: Updating manifest in C:\Users\Ata System\AppData\Local\Jenkins\.jenkins\workspace\Pipeline-01\artifacts-repo\2022-10-09-211243\dist\SayHello.exe.notanexecutable
8717 INFO: Updating resource type 24 name 1 language 0
8719 INFO: Appending PKG archive to EXE
8732 INFO: Fixing EXE headers
8947 INFO: Building EXE from EXE-00.toc completed successfully.
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] echo
Attemping some cleanup works.
[Pipeline] echo
Log when the if condition not true you can see that the else skipped!!!
*Stage3 - Trying to build python program and export the Artifact*
[Pipeline] bat
C:\Users\Ata System\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\python.exe: No module named PyInstaller
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] echo
Attemping some cleanup works.
[Pipeline] echo
you can find the complete jenkins file Here
One option is to surround your bat command with a try-catch or a catchError block
catchError {
pyInstallerOutput = bat(returnStdout: true, script: '#python -m PyInstaller -v').trim()
}

Copy content of a directory to another Jenkins pipeline

I'm trying to copy the content of a directory into another directory on a Jenkins file.
My first attempt was using xcopy, but I'm getting an error ( Invalid number of parameters).
bat 'xcopy cashplus-backoffice/cashplus-backoffice-ui/build/web cashplus-backoffice/src/main/resources/static /e /h'
My second attempt was to using File Operation plugin as follows:
fileOperations([fileCopyOperation(
excludes: '',
flattenFiles: false,
includes: 'cashplus-backoffice/cashplus-backoffice-ui/build/web/**',
targetLocation: 'cashplus-backoffice/src/main/resources/static'
)])
The problem with this solution is that it copy the whole tree and the content of the source folder ( cashplus-backoffice/cashplus-backoffice-ui/build/web. I want just the content of the web folder ).
How can I achieve this?
You can use xcopy command and use \\ instead of \ for windows.
Example:
bat "xcopy /E /H C:\\Data\\myfiles C:\\Data\\Documents\\"

Jenkins Groovy Script not recognizing windows Drive

Using the following groovy script in my Jenkinsfile to do some file operations for preparing my build package:
pipeline {
agent any
stages {
stage('package-windows') {
when {
expression { isUnix() == false && env.JOB_NAME == 'my-job-webapi'}
}
steps {
bat label: 'unzip all files', script: 'FOR /R .\\archive %%I IN (*.zip) DO "C:\\Program Files\\7-Zip\\7z.exe" x "%%I" -aou -o"%%~dpI\\*"'
}
}
}
}
When i run the job its failing with the following error:
\Program was unexpected at this time.
C:\Program Files (x86)\Jenkins\workspace\my-job-webapi>FOR /R .\archive \Program Files\7-Zip\7z.exe" x "~dpI\*"[Pipeline] }
For some reason its unable to recognize the drive letter C: in the path "C:\\Program Files\\7-Zip\\7z.exe". What is the right way to provide the path with windows drive letter in Groovy script ? Or is there a different way this needs to be handled ?
Just needed to use / instead of \\. C:/Program Files/7-Zip/7z.exe worked

Jenkins Pipeline Utility Steps - zip zipFile

I am trying to zip the folders which are created as output of my jenkins pipeline job using pipeline script. By googling i came to know the Jenkins
Pipeline Utility Steps - zip zipFile
https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#code-zip-code-create-zip-file to zip folders/files but could not get exact pipeline syntax to zip.
In my job workspace, I have a folder by name 'Test' which has 2 sub folders as 'Test1', 'Test2'. Each sub folder will have .dll files. So, I would like to zip entire 'Test' folder with all subfolder.
node(Jenkinks_1)
{
echo "ZIP"
zip zipFile: 'Test.zip', dir:'C:\\workspace\\Build_Sample\\Test'
echo "END - ZIP"
}
Below are the Console Output from Jenkins:
Started by user XXXXX
[Pipeline] node
Running on Jenkinks_1 in C:\workspace\Build_Sample
[Pipeline] {
[Pipeline] echo
ZIP
[Pipeline] echo
END - ZIP
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
Looking for some guidance to zip the folders using pipeline syntax. Appreciate your inputs.
I wanted to zip some files as output of my jenkins pipeline job
First, try the same operation in stages and step, as in here:
pipeline {
agent any
stages {
stage ('push artifact') {
steps {
sh 'mkdir archive'
sh 'echo test > archive/test.txt'
zip zipFile: 'test.zip', archive: false, dir: 'archive'
archiveArtifacts artifacts: 'test.zip', fingerprint: true
}
}
...
}
It uses archiveArtifacts to record the result.
If using an absolute path does now work, try a relative one ('..')
As seen by the OP Sri, zip zipFile is part of, and requires the JENKINS Pipeline Utility Steps Plugin.
See "Implemented Steps".
Regarding the syntax to be used for multi-criteria file selection, NicolasW notes in the comments that the documentation is vague: "use glob ant-style syntax"...
He got it to work though, with a basic coma separated syntax.
E.g.
zip zipFile: 'test.zip', archive: false, glob: 'config-/**/,scripts/**/*.*
But, as noted by Tanvir in the comments, issue 44078 means you need to replace zip by:
script{ zip zipFile: 'test.zip', archive: false, dir: 'archive' }
Meaning you need to use a script block.
Was able to Zip after installing the Pipeline Utility Steps plugin.
I came across this because zip was ... not installed on the host.
Reminder to self : If you need zip, install it first.
sudo yum install zip
you can just use sh (jenkins server need install zip);
sh '''
zip -r algo.zip algo
'''
pipeline script like this
node {
stage('Clean'){
cleanWs()
}
stage('Checkout') {
git branch: 'develop', url: 'ssh://user#ip:29418/prj.git'
}
stage('Zip') {
dir('algo-python') {
sh '''
zip -r algo.zip algo
'''
}
}
stage('Upload zip'){
dir('algo-python') {
sh '''
source /etc/profile
export HADOOP_USER_NAME=dev
hdfs dfs -put -f algo.zip /user/dev/zipfile/
'''
}
}
}

FileNameFinder().getFileNames fails on one Jenkins node

Getting a list of files from jenkins workspace using groovy fails on one node but it works on another.
Here is the code in the pipeline:
def pd = pwd()
bat "dir $pd"
def bat_files = new FileNameFinder().getFileNames(pd, 'G*.bat')
Output:
C:\Jenkins\Slave\workspace\TestFolder\CodeTestPipe>dir C:\Jenkins\Slave\workspace\TestFolder\CodeTestPipe
Volume in drive C is OSDisk
Volume Serial Number is AAA1-73FA
Directory of C:\Jenkins\Slave\workspace\TestFolder\CodeTestPipe
01/23/2017 05:34 PM <DIR> .
01/23/2017 05:34 PM <DIR> ..
01/23/2017 05:34 PM 4 GOL.bat
1 File(s) 4 bytes
2 Dir(s) 134,906,617,856 bytes free
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
C:\Jenkins\Slave\workspace\TestFolder\CodeTestPipe does not exist.
at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:483)
at org.codehaus.groovy.ant.FileIterator.setNextObject(FileIterator.java:119)
at org.codehaus.groovy.ant.FileIterator.hasNext(FileIterator.java:81)
at groovy.util.FileNameFinder.getFileNames(FileNameFinder.groovy:44)
at groovy.util.FileNameFinder$getFileNames.callCurrent(Unknown Source)
at groovy.util.FileNameFinder.getFileNames(FileNameFinder.groovy:31)
at
Pipelines are executed on the Jenkins master, and only through the magic of remoting-enabled APIs do things happen on the selected node. So File, and everything using File, doesn't work, and never will: It always executes on master.
Source: https://groups.google.com/forum/#!topic/jenkinsci-users/yBiYbwWjg-I
I was able to get the files by using the dir in bat command:
def bat_out = bat( returnStdout: true, script: '#echo off & dir /b G*.bat').trim()

Resources