Maintain folder structure jFrog Artifactory Deployer TFS plugin - tfs

I'm working on a new CI proof of concept. I'm using TFS build and attempting to integrate jFrog Artifactory.
I'm trying to create a folder structure within my Artifactory repository like so:
[repository]/[sub-repository]/[Artifacts Folder]/[Versioned Artifact Folder]/[Versioned Artifact Zip Archive]
I've scripted the creation of the following correct structure in my Artifactory staging directory with PowerShell:
[Artifacts Folder]\[Versioned Artifact Folder]\[Versioned Artifact Zip Archive]
... and finally compressed my [Artifacts Folder] into a [Artifacts Folder].zip archive for deployment to Artifactory repository.
Now, although jFrog documentation indicates the introduction of an --explode option in jFrog 1.7 for this purpose, attempts to upload using this option returned an Incorrect Usage error:
2018-10-01T10:21:28.3168258Z running 'C:\jfrog\jfrog.exe' rt upload '[Artifactory Staging Directory]\[Artifacts Folder]\*' '[repository]/[sub-repository]/[Artifacts Folder]' --url=https://www.artifactrepository.xxx.net/artifactory --explode=true --user=******** --password=******** --props='build.number=[build_number];build.name=[build_name]'
2018-10-01T10:21:28.3168258Z
2018-10-01T10:21:28.3168258Z
2018-10-01T10:21:29.6761967Z Incorrect Usage.
2018-10-01T10:21:29.6761967Z
2018-10-01T10:21:29.6761967Z NAME:
2018-10-01T10:21:29.6761967Z jfrog rt upload - Upload files
2018-10-01T10:21:29.6761967Z
2018-10-01T10:21:29.6761967Z USAGE:
2018-10-01T10:21:29.6761967Z jfrog rt upload [command options] [arguments...]
2018-10-01T10:21:29.6761967Z
2018-10-01T10:21:29.6761967Z OPTIONS:
2018-10-01T10:21:29.6761967Z --url [Optional] Artifactory URL
2018-10-01T10:21:29.6761967Z --user [Optional] Artifactory username
2018-10-01T10:21:29.6761967Z --password [Optional] Artifactory password
2018-10-01T10:21:29.6761967Z --apikey [Optional] Artifactory API key
2018-10-01T10:21:29.6761967Z --ssh-key-path [Optional] SSH key file path
2018-10-01T10:21:29.6761967Z --props [Optional] List of properties in the form of "key1=value1;key2=value2,..." to be attached to the uploaded artifacts.
2018-10-01T10:21:29.6761967Z --deb [Optional] Used for Debian packages in the form of distribution/component/architecture.
2018-10-01T10:21:29.6917936Z --recursive [Default: true] Set to false if you do not wish to collect artifacts in sub-folders to be uploaded to Artifactory.
2018-10-01T10:21:29.6917936Z --flat [Default: true] If set to false, files are uploaded according to their file system hierarchy.
2018-10-01T10:21:29.6917936Z --regexp [Default: false] Set to true to use a regular expression instead of wildcards expression to collect files to upload.
2018-10-01T10:21:29.6917936Z --threads [Default: 3] Number of artifacts to upload in parallel.
2018-10-01T10:21:29.6917936Z --dry-run [Default: false] Set to true to disable communication with Artifactory.
2018-10-01T10:21:29.6917936Z
I using jFrog Artifactory Deployer 2.1.1 TFS build task.
This command line option is described here: https://www.jfrog.com/confluence/display/CLI/CLI+for+JFrog+Artifactory#CLIforJFrogArtifactory-UploadingFiles
However, it seems that jFrog.exe which is on our TFS servers doesn’t understand --explode command line option.
(Note: I am unsure what version of jFrog.exe is running on our build servers; currently awaiting details from responsible team, update to follow.)
Is the issue that the jFrog.exe version is older (pre 1.7) and does not support the --explode command option? If so, is there an alternative way to achieve multiple artifact upload while preserving staging folder structure?
(Note: I applied the --flat=false option but the staging folder hierarchy was preserved right back to the root; this is not what's required either).
insights appreciated, thanks for looking..

In the end, we were able to work around the absence of the '--explode' command option by using placeholders like so:
In the jFrog Artifactory Deployer task:
Path to the Artifacts: [Artifacts Folder]\(**)\(*)
Target Repository [repository]/[sub-repository]/[Artifacts Folder]/{1}/
The use of placeholders in this way accomplished the preservation of folder structure in the push to the Artifactory repository as required.

Related

Nexus repository path

I have uploaded some release artifacts from Jenkins to Nexus. I can see they have been created under Nexus as per the -Dversion I have provided.
Example: AppName-BuildNumber-Snapshot.zip
However, somehow the repository path is autogenerated, it has extra yyyymmdd.hhmmss appended to it.
Example: AppName-BuildNumber-Snapshot-yyyymmdd.hhmmss.zip
I would like to download the release artifact from Jenkins but due to the autogenerated Nexus repository path I am not able to download it in the separate job.
How can I force it to stop adding yyyymmdd.hhmmss to the repository path? Or is there anyway I can retrieve the repository path using AppName and BuildNumber?
If you are uploading a SNAPSHOT, then Nexus dynamically stores it with a timestamp. That's because you can replace -SNAPSHOTs with newer copy w/same GAV. This is unlike real # jars which cannot/should not be redeployed, thus preserving their integrity.
To download a -SNAPSHOT, you just reference it as -SNAPSHOT; maven, aether, etc will retrieve the latest.
Maven (by default) only checks the remote repository for versions newer than your local once daily, unless you use -U option or change the updatePolicy.
If you want "solid" number, then you must release jar.
btw, you should see it's -yyyymmdd.hhmmss-nn, where nn is an incremental number for that version.
Nexus supports many different repository formats. If you only require maven, use Nexus 2 as it better supports maven.
Repository types: Maven has two distinct types:Release and Snapshot Repositories.
If you want to "upload a zip file containing release artifacts and retrieve it back based on the build version number?", then that's what you should do.
You must configure a repository of type release, not snapshots (gleened from: -DrepositoryId=dsnexus-snapshots) (or in addition to type snapshots).
To pass the "build version number", then presumably, you have:
[ X ] Create a formatted version number
Environment Variable Name [ label ]
Make sure your maven step has:
[ X ] Inject build variables
Then, mvn deploy:deploy-file -DgroupId=com.my.gid -DartifactId=AppName -Dversion=${label} -DrepositoryId=dsnexus-release
ps: you'll also want to implement a cleanup strategy in your Nexus repository if you are pumping every build as a new artifact.

Modules in Artifactory - What are they?

There's a concept in Artifactory of "Modules". In the UI, when you go to the "Builds" tab, and enter some specific build, you get a list of Module IDs, which is, from what i've seen, just the name of the build.
So when I do this:
jfrog rt upload --build-name Kuku --build-number 1 File.bin Repo/path/
I get a Build with name "kuku", the number "1", and inside I have a list of Module IDs, with only 1 entry - Kuku, and if I click on it then I can see the list of artifacts that are part of this build.
So my question is, what are these Modules? How can I add more modules (Or why should I) to a specific build?
Just mention that I'm using a Generic repo
Module are part of the JFrog build info open integration layer for the CI servers and build tools which is used for capturing build metadata.
Build info can be captured using CI integrations, build tools, the JFrog CLI or the REST API. Once it is published to Artifactory, you can query it using the REST API, Artifactory Query Language (AQL) and GraphQL.
As part of build info, a module is a way to group artifacts which where produced by as build and dependencies which where used when building them. A module, for example, can represent a micro service.
The JFrog CLI, which you are already using, can be used for collecting build info.
You can add artifacts when uploading them in the following way:
jfrog rt upload --build-name Kuku --build-number 1 --module Momo File.bin Repo/path/
And dependencies can be added when you are downloading them:
jfrog rt dl "my-local-repo/yoyo.zip" kuku-dependencies/ --build-name Kuku --build-number 1 --module Momo
To publish build info, use the following command:
jfrog rt bp kuku 1

How to update Artifact Build Module ID

I am using Jenkins CI(single pipeline Job) to push different files to JFrog Artifactory.
For each and every build I am publishing the build info to Artifactory. But every build has same Module ID.
for example: In Artifactory Repository Browser select any artifact then go to Build tab, then it is showing all builds instead of respective artifact build information
So I would like to know how to modify the Artifact Module ID while publishing the artifact information to Artifactory
The build information and Artifact storage are to different mechanisms. The build information is only associated with artifacts in the sense that you can define what artifacts are produced from a build and what artifacts are a dependency of a build. The module ID has no association with the build info. The module ID is defined by the repository layout where the Artifact is stored. See the jfrog documentation here: https://www.jfrog.com/confluence/display/RTF/Repository+Layouts
You cannot change the module ID. Artifactory assigns a module ID based on the repository layout. This means the path to the artifact matters for defining the module ID.
For example the repository layout
[org]/[module]/[module]_[base_rev].[ext] will give module IDs to any artifact stored following that pattern as [org]:[module]:[base_rev].
Artifacts are associated to builds by both a build.name, build.number property on the artifact and publishing a build info from a .json file. The .json must include a section denoting what artifacts were produced from the build.
In general, it sounds like you need to ensure each artifact is being deployed to a unique path in artifactory with the build.name and build.number property being set. The other thing to look at is the file hash. it must match in the build info being published. So if the file hash is not different that means there is no change to the artifact from the previous build.
I don't know how much this will help, but there is a lot of nuances that go into getting the artifacts and builds to line up. You mentioned using Jenkins, have you looked into using the Jenkins Artifactory plugin. It should take care of a lot of this work for you, although we use Bamboo, so I am not as familiar with how it differs.

Publish msi file within the artifactory

I'm using the free artifactory version OSS and would like to deploy some msi that was built by jenkins.
So I'm using the artifactory plugin --> Generic Artifactory integration.
I created my own repository (lest call it MY_REPO) based on simple-layout.
In "Published Artifacts" jenkins field we provide the msi path.
So the question is:
how to deploy the msi in desired path : MY_REPO\component\ ver...
I would like to use the current plugin and to pass the parameters but I don't see where it's done.
Thanks in advance,
Dima.
in the Published Artifacts section
XXX/*.msi => com/aaa//1.0.$BUILD_NUMBER
where XXX is the path of your msi file , com/aaa is the relative path on your artifactory & BUILD_NUMBER is the current build number

Resolving Artifacts using Jenkins job with the parent directory

I am looking to download the artifacts using Jenkins job to resolve the artifacts from Artifactory. Specifying the file type and the path to the artifact works, However, unable to resolve all artifacts from the root directory.
Actual Artifactory Path:
repo_key:Group/Artifact/Version/path/to/artifact1/file.zip
repo_key:Group/Artifact/Version/path/to/artifact2/file.zip
Below Configuration in Jenkins job to Resolved Artifacts doesn't works:
repo_key:Group/Artifact/*=>Output
How do I download all files under the Artifact directory to the Output directory.
You need to use the format JBaruch mentioned and add the build metadata as matrix params, to support wildcard resolution for multiple files.
For instance:
repo_key:Group/Artifact/**/*#publishing_build_name#LATEST
Will get you the latest artifacts published by the job "publishing_build_name".
There's some helpful information and examples when clicking on the Question Mark next to the "Resolved Artifacts" field.
Artifact/* will resolve files, directly located under Artifact directory (and there are none). What you need is Artifact/**/*.

Resources