does anyone know whether (and how) it is possible to retrieve the deployment timestamp of a Maven artifact from the repository programmatically?
I need to write a plugin that checks the project's dependencies and their dependencies as well. The plugin must check the deployment timestamps of the dependencies and their dependencies to ensure version compatibility during the release process of a very large legacy application with strong coupling between its modules. => Just for the background ;)
So I've tried several Maven APIs (maven-core, maven-artifact, ...) but even the Metadata interfaces don't have something like a timestamp feature.
Related
I'm trying to extend our Jenkins job (which builds the entire project) to deploy the built artifacts to our Artifactory but then I faced some problems related to the versioning of the artifacts. If I try to redeploy an artifact whose version didn't change (not a snapshot), I get an error 403 (user 'foo' needs DELETE permission) which is understandable, I should not replace an already released artifact. If the artifact version contains -SNAPSHOT then there are no problems, it's always uploaded. My question is: how we should approach the scenario of having locked overwriting in Artifactory?
Shouldn't the artifactory plugin from Jenkins just ignore the deploy of the artifact in case is already deployed instead of failing the job?
Or should we use always -SNAPSHOT (during development) even the artifact has not changed?
Do we increase the version on every release even the artifact has not changed?
Shouldn't the artifactory plugin from Jenkins just ignore the deploy
of the artifact in case is already deployed instead of failing the
job?
The job should fail if the artifact is already deployed with a fixed version (non -SNAPSHOT). For instance on a manual job trigger, I would like to know if I tried to build and deploy using a version name that is already published (maybe by someone else in the team)
Or should we use always -SNAPSHOT (during development) even the
artifact has not changed?
-SNAPSHOT is made for development. Yes we usually push the artifact at the end of the build, even if it did not change because you updated for instance a README and the job was triggered.
Usually SNAPSHOT have a lifetime depending on how you binary repository (here Artifactory) is configured. SNAPSHOT can be cleaned every 2 weeks for instance.
The link shared by Manuel has other interesting definitions like
Usually, only the most recently deployed SNAPSHOT,
for a particular version of an artifact is kept in the artifact repository.
Although the repository can be configured to maintain a rolling archive
with a number of the most recent deployments of a given artifact
https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN401
Do we increase the version on every release
even the artifact has not changed?
yup we increase the version number at every release. I call release what the customer will get. Except an exceptional occasion, you wont go through the process of release if the artifact didn't change. A release usually involves a lot of people in an organization, even people that are not from Development. A popular standard is to use semantic versioning https://semver.org/ Sometime people prefer to version with the date. My advice is to use semver and have a file in the artifact with the date of the build. This file could be used by the artifact itself to tell its version at runtime.
You could work with build numbers, and you wouldn't overwrite existing versions. Instead a buildNumber could include some bugfixes/security fixes.
https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#A1000661
If you're using the depenfency, you can handle the versions with expressions. Exact version or expression which covers the buildNumber.
Build process of java code is currently generating artifacts with name having no version number.
For ex: serial-framework-SNAPSHOT.jar
We are currently in build phase of CI/CD pipeline
All the artifacts generated through maven build has no version number for back-end services, in build phase of CI/CD pipeline
Dependent artifacts that are required to build a specific jar are only stored in JFrog artifactory
1)
Does it require versioning of artifacts for Build/QA/prod phase of CI/CD pipeline?
2)
Does it require to store every artifact in JFrog artifactory? Because only dependent artifacts that are required by pom.xml during maven build are stored in JFrog
The versioning was well explained in the first answer by snukone.Note the below points which might be helpful regarding versioning
For development always follow the version as “versionNumber-SNAPSHOT “(capital letters)
Eg:- 1.0 -SNAPSHOT
2) For test/prod branch follow the version as “versionNumber-RELEASE”
Eg:- 1.0 – RELEASE
a) Snapshots are mutable, so they are used for development purpose.
b) Releases are immutable. Once committed we cannot override the artifact in the
artifactory. So releases are used for higher environments.
c) Snapshots capture a work in progress and are used during development. A Snapshot artifact has both a version number such as “1.3.0” or “1.3” and a timestamp. For example, a snapshot artifact for commons-lang 1.3.0 might have the name commons-lang-1.3.0-20090314.182342-1.jar
So in your case if you are using "serial-framework-SNAPSHOT" it will store as "serial-framework-version-timestamp.jar" in your artifactory.
Similarly if you are using "serial-framework-RELEASE" it will store as "serial-framework-version.jar" in your artifactory.
How versioning helps:
Versioning helps in case you want to restore an older version of your application (due to bugs that are heavily decreasing performance in production)
If you are running integrationtests on api or ui level, you can specify which versions are fitting together (ie via contract testing: https://github.com/pact-foundation/pact_broker)
Default cleaning processes helps you to prevent your artifactory from huge storage usage
Storing every artifact or not?
My personal experience: Just store the artifacts which are dependencies to other artifacts. Like Libs for example. If you are working with Docker Container you should think about to version the Docker images which you are producing on every build.
Is there any Repository manager that manages the binary dll files and also integrates well with the Jenkins?
Can Nexus be used to manage the dll files as these files are created as a part of Embedded C/C++ Projects and not sure if Nexus Artifact Manager supports/integrates well with such Projects as it mainly supports the Java projects?
Is there a way to automatically manage the upload and download of such project artifacts from Nexus/other artifact managers without the use of POM file?
Suggest in case there are other Artifact Managers that supports binary artifacts.
Artifactory can be used to store any type of binaries.
Starting with Artifactory 4.0, you can create generic repositories which allows uploading packages of any type. You will not need to upload any POM files and Artifactory will not need to calculate any metadata (for example Maven metadata).
To deploy files you can use the REST API or the UI, for example:
curl -uUSER:PASS -T file.dll http://localhost:8081/artifactory/dll-local/path/to/file.dll
If you have a certain layout you would like to use for this repository you can create a custom layout and associate it with the repository. This can be useful for automatic snapshot/integration versions cleanup and other module management tasks.
Disclaimer: I'm affiliated with Artifactory
The Nexus repository manager is java oriented, but can be used to store any files you want. Binaries of all types or even just text configuration files.
To automate the file upload process, you can use maven from command line:
mvn deploy:deploy-file -DgroupId=com.you -DartifactId=file -Dversion=1.0 -Dpackaging=exe -Dfile=c:\out\file.exe -Durl=http://yourserver/nexus/content/repositories/releases -DrepositoryId=releases
Then, to get the file, you should be able to get it directly with the following URL:
wget http://yourserver/nexus/content/repositories/releases/com/you/file/1.0/file-1.0.exe
This is a simple approach to using Nexus as a general artifact repository.
I hope this helps.
The open source version of Nexus (Nexus OSS) is supports many repository formats out of the box including Maven, NuGet, NPM, RubyGems and others. Nexus just runs on Java (e.g. like Jenkins). It is not Java only...
Depending on how you plan to get the DLL files from the repository, different formats might be more or less suited to your usage. You could even use a custom format, but then you rely custom tools.
The scenarios I have seen at many customers are
using a Maven repo and pulling the files in either in a Maven build together with the Maven NAR Plugin (used for native development with C/C++)
using a Maven repo and pulling via plan HTTP GET calls using your scripting language/build tool of choice
using NuGet format and store the DLLs in NuGet packages in the repo and using nuget to retrieve them for the projects
All of these work well.
I am looking to separate development and production environments by publishing jars to ivy with live and dev qualifiers.
I am looking for a way to trigger ivy from projects that have these dependencies to automatically grab the latest from these environments based on the ant build file.
I am new to ant and ivy and am not finding documentation on if this is possible or not.
Basically, build-live in ant would trigger resolve-live that would use ref="[1.live.0,)", however we would also need a default one for developers in an Eclipse environment to automatically pick up dependencies through the plugin.
You have not indicated what type of repository you are using. I'm going to assume you are not using a Maven repository manager to manage your release repository. Some of these support remote workflow to manage what I like to call "release candidates" (for an example take a look at the staging feature provided by Sonatype Nexus)
For a pure ivy solution I first recommend reading the best practice documentation, specifically the section titled "Dealing with integration versions".
When publishing a new ivy module version one can set the status field. Out of the box ivy supports "integration", "milestone" or "release" but even these can be extended. The status is a label or metadata attribute that appears in the info field of the published ivy file within the ivy repository.
How does this work? When publishing the module as follows:
<ivy:publish resolver="???" pubrevision="1.0.1" status="integration">
<artifacts pattern="build/artifacts/jars/[artifact].[ext]" />
<artifacts pattern="build/artifacts/zips/[artifact].[ext]" />
</ivy:publish>
This states that the release 1.0.1 is an integration release.
This then enables the functionality you're looking for. Ivy's dynamic revisions capability can be used to automatically download the latest version with a particular status as follows:
<dependency org="acme" name="foo" rev="latest.integration" />
Update
Once a module is published into the repository it cannot be changed. Doing so could potentially break builds that rely on that version. Think about, if you change the status of a build how to communicate that change? Instead you use the "status" to indicate how stable a version is. Open source projects will frequently publish several "general availability" or "milestone" releases before the finally approved major version.
To do what you want to do requires server-side repository file management. I recommend looking at the "staging" suite in Sonatype Nexus. This feature keeps each pending release in a temp repo until it's finally approved and merged into the main release area.
We have the build system, which builds a lot of components. Built components publish their artifacts by ivy in artifactory.
I want to attach an extra meta-artifact to each component during its publication, but transparently for developers. This meta-artifact will contain the information about build-agent, build-log, changelog, etc.
I have access to build agents, so I can:
change some ivy configuration files on build agents
install a customized ivy with a patch on build agents
modify the ivysettings.xml file
I didn’t find any information in ivy documentation. However, I know that there are methods like “options.getExtraArtifacts()” in the ivy source code, so I guess there is a way.
Waiting for your suggestions.
Thanks in advance!
UPD:
I figured out “options.getExtraArtifacts()” serve an "artifact" subtag in a "publish" tag:
https://ant.apache.org/ivy/history/2.4.0/use/publish.html
Option1 : Artifactory Metadata
Are you using Artifactory Pro? I'm not a user myself but the documentation describes a capability to attach arbitrary XML based metadata to any artifact stored in Artifactory. Standalone REST API that should be simple to call from your build:
https://www.jfrog.com/confluence/display/RTF2X/Attaching+and+Reading+Metadata
The big advantage of this approach is that this data is searchable
Option 2: Jenkins plugin
Again I'm not an Artifactory user, but have always been intrigued by this plugin that reportedly delivers tight integration between Jenkins and Artifactory. The relevent bit is the so called "Build info" that is published by the plugin. I would be surprised if this doesn't include some of the info your looking for.
https://wiki.jenkins-ci.org/display/JENKINS/Artifactory+Plugin
Option 3: Attach a custom metadata file
A typical Maven repository has no support for metadata. Both Artifactory and Nexus have bespoke extensions and it appears that in both products this capability is a paid feature...
If all you want to do is store metadata and don't care about the lack of a search interface, then why not simply attach a file containing your metadata as an additional Module file? The following example documents how to do this:
how to publish 3rdparty artifacts with ivy and nexus
Option 4: Ivy extra attributes
This is only an option if you're storing data in an ivy repository (You are more likely using a Maven repo format).
Ivy has the ability to store custom attributes in the ivy module file.
http://ant.apache.org/ivy/history/latest-milestone/concept.html#extra
The following answer describes how these can also be used to influence the layout of your ivy repository.
Dynamically updating Ivy extra attributes