I want to copy/clone only particular files from repo.
I tried many ways using cm cmd line executable of plastic.
cm getfile --help
Downloads the content of a given revision.
Usage:
cm getfile | cat <revspec> [--file=<output_file>] [--debug]
[--symlink] [--raw]
revspec Object specification. (Use 'cm help objectspec' to learn
more about specs.)
Options:
--file File to save the output. By default, it is printed on the
standard output.
--debug When a directory specification is used, the command
shows all the items in the directory, its revision id
and file system protection.
--symlink Applies the operation to the symlink and not to the
target.
--raw Displays the raw data of the file.
Examples:
cm cat myfile.txt#br:/main
(Obtains the last revision in branch 'br:/main' of 'myfile.txt'.)
cm getfile myfile.txt#cs:3 --file=tmp.txt
(Obtains the changeset 3 of 'myfile.txt' and write it to file 'tmp.txt'.)
cm cat serverpath:/src/foo.c#br:/main/task003#myrepo
(Obtains the contents of '/src/foo.c' at the last changeset of branch
'/main/task003' in repository 'myrepo'.)
cm cat revid:1230#rep:myrep#repserver:myserver:8084
(Obtains the revision with id 1230.)
cm getfile rev:info\ --debug
(Obtains all revisions in the 'info' directory.)
Related
Is it possible to obtain the Job Description or the Job Parameter Description in run-time or later like the BUILD_ID or JOB_NAME?
I search for plugins or workarounds and nothing.
Thanks.
This would be Tip/workaround
https://<<yourjenkinsdomain>>/job/<<yourjobname>>/configure (will open the configuration of your job)
However
https://<<yourjenkinsdomain>>/job/<<yourjobname>>/config.xml (will give the job configuration in an xml format)
You can download this xml via curl at run time or using jenkins cli and use a grep with -B option to find description per value.
Considering you have copied the with name "config.xml"
cat config.xml | grep -B 1 "description"
Will give you description and build parameter name
Grep command
-B NUM, --before-context=NUM
Print NUM lines of leading context before matching lines.
Places a line containing a group separator (--) between
contiguous groups of matches. With the -o or --only-matching
option, this has no effect and a warning is given.
Sample output :
cat config.xml | grep -B 1 "description"
<actions/>
<description>Job description : Automation </description>
--
<name>branch</name>
<description>mandatory parameter , used for automation</description>
--
Alternative :
jenkins cli has an option to set value
set-build-description Sets the description of a build.
set-build-parameter Update/set the build parameter of the current build in progress. [deprecated]
you can write a small script and get the values into variables and use them
As part of automating the process of running secure code analysis, I have a Jenkins job which uses the sourceanalyzer command line tool to generate an .fpr results file. At the moment I'm opening this results file in Audit Workbench application to view the results and check if there's any newly introduced issues etc, and generating a report from there in PDF/XML format.
Does anyone is it possible to invoke Audit Workbench through the command line and generate a report on the issues, which we could then leverage through a Jenkins script and also then mail the results? Looking online the command line usage seems to stop at the fpr generation stage.
Thanks in advance!
There is a command-line utility to generate an Report from the FPR file.
Currently there are two report generators: Legacy and BIRT. The BIRT report engine was introduced into Audit Workbench with version 4.40.
Here is an example using the BIRT Report engine to generate a DISA STIG report
BIRTReportGenerator -template "DISA STIG" -source HelloWorld_second.fpr
-output BirtReport.pdf -format PDF -showSuppressed --Version "DISA STIG 3.9"
-UseFortifyPriorityOrder
Using the legacy one is a little more involved. The command is:
ReportGenerator -format pdf -f LegacyReport.pdf -source HelloWorld_second.fpr
-template DisaStig3.10.xml -showSuppressed -showHidden
You can either use one of the predefined template reports located in the <SCA Install Dir>/Core/config/reports directory or generate one using the Report Wizard and saving the template which gets stored in the C:\Users\<USER>\AppData\Local\Fortify\config\AWB-XX.XX\reports\ directory in Windows.
On Linux/Mac look at the configuration file <SCA Install Dir>/Core/config/fortify.properties for the com.fortify.WorkingDirectory property, this is where the reports will be stored
#SBurris,
If you don't want to show Suppressed/Hidden is it just -hideSuppressed and -hideHidden?
Also, is there a way to add custom filters to not show things like "nones" from the STIG/SANS/OWASP like you can create in the AWB GUI?
Basically, I need a command(s) to merge two FPRs and then compare them based on what is found new on the scanned code vs. the old FPR.
Merge should be:
FPRUtility -merge -project <newest_scan.fpr> -source <previous_scan.fpr> -f <BUILDXX_MergedWith_BUILDXY.fpr>
The custom filter I need after the merge is:
"[OWASP Top 10 2013]:!<none> OR [SANS Top 25 2011]:!<none> OR [STIG 3.9]:!<none> AND [Detected On]:!/^/"
Where the Detected On field is a custom tag that I need to carry through from the previous FPR file into the newly merged one.
AND THEN output the report from that newly merged fpr in pdf and xml format to a location/filename I specify. Something along the lines of:
~AWB_Installation_Dir/bin/ReportGenerator -format pdf -f [BUILDXX_MergedWith_BUILDXY].pdf -source output.fpr
-template DisaStig3.10.xml -hideSuppressed -hideHidden
Obviously this can be a multitude of commands as long as we can get it back to Bamboo. Any help would be greatly appreciated. Thanks.
FPRUtility interprets the space-separated conditions in the -information -search -query ... parameter by applying the boolean AND operator. To obtain a union of 2 conditions A || B, I figured I could intersect negations of other conditions that complement the former: !C && !D (where A || B || C || D always holds true). I.e., to find all high and critical issues, I use
FORTIFY_ROOT\jre\bin\java -d64 -Xmx4096M -jar FORTIFY_ROOT\Core\lib\exe\fpr-utility-exe.jar -project APP_VER_DATE.fpr -information -search -query "[OWASP Top 10 2017]:A [fortify priority order]:!low [fortify priority order]:!medium" -categoryIssueCounts -listIssues > issues.txt
In case of an audit, I figured I needed the older report generation utility to include suppressed issues (and their comments),
sed -e 's/\(IssueListing limit=\)"[^"]\+"/\1"-1"/' -i "FORTIFY_ROOT/Core/config/reports/DeveloperWorkbook.xml"
cmd /c call ReportGenerator -template DeveloperWorkbookAll.xml -format pdf -source APP_VER_DATE.fpr -showSuppressed -f "APP_VER_DATE_with_suppressed.pdf"
How we can generate FortiFy report using command ??? on linux.
In command, how we can include only some folders or files for analyzing and how we can give the location to store the report. etc.
Please help....
Thanks,
Karthik
1. Step#1 (clean cache)
you need to plan scan structure before starting:
scanid = 9999 (can be anything you like)
ProjectRoot = /local/proj/9999/
WorkingDirectory = /local/proj/9999/working
(this dir is huge, you need to "rm -rf ./working && mkdir ./working" before every scan, or byte code piles underneath this dir and consume your harddisk fast)
log = /local/proj/9999/working/sca.log
source='/local/proj/9999/source/src/**.*'
classpath='local/proj/9999/source/WEB-INF/lib/*.jar; /local/proj/9999/source/jars/**.*; /local/proj/9999/source/classes/**.*'
./sourceanalyzer -b 9999 -Dcom.fortify.sca.ProjectRoot=/local/proj/9999/ -Dcom.fortify.WorkingDirectory=/local/proj/9999/working -logfile /local/proj/working/9999/working/sca.log -clean
It is important to specify ProjectRoot, if not overwrite this system default, it will put under your /home/user.fortify
sca.log location is very important, if fortify does not find this file, it cannot find byte code to scan.
You can alter the ProjectRoot and Working Directory once for all if your are the only user: FORTIFY_HOME/Core/config/fortify_sca.properties).
In such case, your command line would be ./sourceanalyzer -b 9999 -clean
2. Step#2 (translate source code to byte code)
nohup ./sourceanalyzer -b 9999 -verbose -64 -Xmx8000M -Xss24M -XX:MaxPermSize=128M -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+UseParallelGC -Dcom.fortify.sca.ProjectRoot=/local/proj/9999/ -Dcom.fortify.WorkingDirectory=/local/proj/9999/working -logfile /local/proj/9999/sca.log -source 1.5 -classpath '/local/proj/9999/source/WEB-INF/lib/*.jar:/local/proj/9999/source/jars/**/*.jar:/local/proj/9999/source/classes/**/*.class' -extdirs '/local/proj/9999/source/wars/*.war' '/local/proj/9999/source/src/**/*' &
always unix background job (&) in case your session to server is timeout, it will keep working.
cp : put all your known classpath here for fortify to resolve the functiodfn calls. If function not found, fortify will skip the source code translation, so this part will not be scanned later. You will get a poor scan quality but FPR looks good (low issue reported). It is important to have all dependency jars in place.
-extdir: put all directories/files you don't want to be scanned here.
the last section, files between ' ' are your source.
-64 is to use 64-bit java, if not specified, 32-bit will be used and the max heap should be <1.3 GB (-Xmx1200M is safe).
-XX: are the same meaning as in launch application server. only use these to control the class heap and garbage collection. This is to tweak performance.
-source is java version (1.5 to 1.8)
3. Step#3 (scan with rulepack, custom rules, filters, etc)
nohup ./sourceanalyzer -b 9999 -64 -Xmx8000M -Dcom.fortify.sca.ProjectRoot=/local/proj/9999 -Dcom.fortify.WorkingDirectory=/local/proj/9999/working -logfile /local/ssap/proj/9999/working/sca.log **-scan** -filter '/local/other/filter.txt' -rules '/local/other/custom/*.xml -f '/local/proj/9999.fpr' &
-filter: file name must be filter.txt, any ruleguid in this file will not be reported.
rules: this is the custom rule you wrote. the HP rulepack is in FORTIFY_HOME/Core/config/rules directory
-scan : keyword to tell fortify engine to scan existing scanid. You can skip step#2 and only do step#3 if you did notchange code, just want to play with different filter/custom rules
4. Step#4 Generate PDF from the FPR file (if required)
./ReportGenerator -format pdf -f '/local/proj/9999.pdf' -source '/local/proj/9999.fpr'
I Have more than 1000 jobs in Jenkins,
And I would like to go through all of them in order to clean unused jobs.
What is the recommended way to do so?
I guess in every job "xml" file there is an indication to when it last ran,
Can anyone point me where this file is located?
I ended up filter the jobs by the "View job Filters" plugin,
You can use "Filter by Build Trend" option as follows:
Create a view for "All jobs" -> go to edit view -> in "add job filter " choose "Build Trend Filter" -> choose the filter you desire.
This is what I did:
I don't think you can do this in one step. But you can do this in 2 steps.
Find the URLs of all jobs with this:
https://jenkins-server/api/json?tree=jobs[url]
Get more info about each job by using the urls returned from step 1:
url-from-step1/api/json
This will give you the healthreport, last failed/successful build etc. If you need more info about these builds you can make a new request with :
url-from-step1/last-build-number/api/json
I recommend using JSON, and using JQ (http://stedolan.github.io/jq/, https://jqplay.org/) to parse your JSON
Happy coding!
You can leverage the REST API. The following urls might be relevant for you:
https://ci.jenkins-ci.org/api/xml?tree=jobs[name] -- to get a list of jobs
https://ci.jenkins-ci.org/job/{jobName}/lastBuild/buildTimestamp?format=yyyy-MM-dd-HH-mm-ss -- to get the time of last build of job {jobName}
Feel free to change xml to json/python...
I can provide a following shell script as a rough example:
#!/bin/bash
jenkinsUrlBase='https://ci.jenkins-ci.org'
callJenkins() {
curl --silent --show-error -g "$jenkinsUrlBase${1}"
}
callJenkins '/api/xml?tree=jobs[name]' | xmlstarlet sel -t -v '//hudson/job/name' | while read projectName ; do
timestamp=$(callJenkins "/job/${projectName}/lastBuild/buildTimestamp?format=yyyy-MM-dd-HH-mm-ss")
echo "Last build of ${projectName}: ${timestamp}"
done
You can exploit directory and file structure in ${JENKINS_HOME}:
cd ${JENKINS_HOME}/jobs/${JOB_NAME}/builds
ls -lt | head -2 | tail -1 | awk '{print $9}'
Example output:
2015-08-13_11-48-25
I have a rails site. I'd like, on mongrel restart, to write the current svn version into public/version.txt, so that i can then put this into a comment in the page header.
The problem is getting the current local version of svn - i'm a little confused.
If, for example, i do svn update on a file which hasn't been updated in a while i get "At revision 4571.". However, if i do svn info, i get
Path: .
URL: http://my.url/trunk
Repository Root: http://my.url/lesson_planner
Repository UUID: #########
Revision: 4570
Node Kind: directory
Schedule: normal
Last Changed Author: max
Last Changed Rev: 4570
Last Changed Date: 2009-11-30 17:14:52 +0000 (Mon, 30 Nov 2009)
Note this says revision 4570, 1 lower than the previous command.
Can anyone set me straight and show me how to simply get the current version number?
thanks, max
Subversion comes with a command for doing exactly this: SVNVERSION.EXE.
usage: svnversion [OPTIONS] [WC_PATH [TRAIL_URL]]
Produce a compact 'version number' for the working copy path
WC_PATH. TRAIL_URL is the trailing portion of the URL used to
determine if WC_PATH itself is switched (detection of switches
within WC_PATH does not rely on TRAIL_URL). The version number
is written to standard output. For example:
$ svnversion . /repos/svn/trunk
4168
The version number will be a single number if the working
copy is single revision, unmodified, not switched and with
an URL that matches the TRAIL_URL argument. If the working
copy is unusual the version number will be more complex:
4123:4168 mixed revision working copy
4168M modified working copy
4123S switched working copy
4123:4168MS mixed revision, modified, switched working copy
If invoked on a directory that is not a working copy, an
exported directory say, the program will output 'exported'.
If invoked without arguments WC_PATH will be the current directory.
Valid options:
-n [--no-newline] : do not output the trailing newline
-c [--committed] : last changed rather than current revisions
-h [--help] : display this help
--version : show version information
I use the following shell script snippet to create a header file svnversion.h which defines a few constant character strings I use in compiled code. You should be able to something very similar:
#!/bin/sh -e
svnversion() {
svnrevision=`LC_ALL=C svn info | awk '/^Revision:/ {print $2}'`
svndate=`LC_ALL=C svn info | awk '/^Last Changed Date:/ {print $4,$5}'`
now=`date`
cat <<EOF > svnversion.h
// Do not edit! This file was autogenerated
// by $0
// on $now
//
// svnrevision and svndate are as reported by svn at that point in time,
// compiledate and compiletime are being filled gcc at compilation
#include <stdlib.h>
static const char* svnrevision = "$svnrevision";
static const char* svndate = "$svndate";
static const char* compiletime = __TIME__;
static const char* compiledate = __DATE__;
EOF
}
test -f svnversion.h || svnversion
This assumes that you would remove the created header file to trigger the build of a fresh one.
If you just want to print latest revision of the repository, you can use something like this:
svn info <repository_url> -rHEAD | grep '^Revision: ' | awk '{print $2}'
You can use capistrano for deployment, it creates REVISION file, which you can copy to public/version.txt
It seems that you are running svn info on the directory, but svn update on a specific file. If you update the directory to revision 4571, svn info should print:
Path: .
URL: http://my.url/trunk
Repository Root: http://my.url/lesson%5Fplanner
Repository UUID: #########
Revision: 4571
[...]
Last Changed Rev: 4571
Note that the "last changed revision" does not necessarily align with the latest revision of the repository.
Thanks to everyone who suggested capistrano and svninfo.
We do actually use capistrano, and it does indeed make this REVISION file, which i guess i saw before but didn't pay attention to. As it happens, though, this isn't quite what i need because it only gets updated on deploy, whereas sometimes we might sneakily update a couple of files then restart, rather than doing a full deploy.
I ended up doing my own file using svninfo, grep and awk as many people have suggested here, and putting it in public. This is created on mongrel start, which is part of the deploy process and the restart process so gets done both times.
thanks all!