Github-actions: cache repo to speed up maven builds - maven-3

Maven dependencies are downloaded every time my build workflow is triggered.
Travis CI provides a way to cache maven repository. Does Github actions provider similar functionality that allows to cache maven repository?

For completeness, this is an example of how to cache the local Maven repository on subsequent builds:
steps:
# Typical Java workflow steps
- uses: actions/checkout#v1
- name: Set up JDK 11
uses: actions/setup-java#v1
with:
java-version: 11
# Step that does that actual cache save and restore
- uses: actions/cache#v1
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-
# Step that runs the tests
- name: Run tests
run: mvn test
See this page for more details.

As of 2021, the most up to date official answer would be
- name: Cache local Maven repository
uses: actions/cache#v2
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-
taken from an official github example at https://github.com/actions/cache/blob/master/examples.md#java---maven

Apparently as of 10 September 2019 caching of build dependencies is not present in Github Actions. Github staff have acknowledged that this feature is necessary, and have replied that "We're working on caching packages and artifacts between workflow executions, we'll have it by mid-November [2019]."
Source: https://github.community/t5/GitHub-Actions/Caching-files-between-GitHub-Action-executions/m-p/30974/highlight/true#M630

Dependency caching is now available:
https://help.github.com/en/actions/automating-your-workflow-with-github-actions/caching-dependencies-to-speed-up-workflows

For maven you can now use setup-java to cache the dependencies. Example:
- uses: actions/setup-java#v2.3.1
name: Install Java
with:
java-version: 8
distribution: 'adopt'
cache: 'maven'
Note that there are the same with other setup-*: node, python and ruby.

I started to use skjolber's Maven Cache Github Action, see https://github.com/marketplace/actions/maven-cache. My first tests are quite positive. The first build downloaded the dependencies from Maven Central; the subsequent build found the dependencies in the cache.

For multimodule projects one could speedup builds by building only modules affected by the recent changes using https://github.com/avodonosov/hashver-maven-plugin/
This will work given the possibility to cache the local maven repository, as described in the #Rob van der Leek's answer.
mvn install the project artifacts after every build, and on the next build the hashver-maven-plugin will find artifacts of the unaffected modules and will skip them.

Related

Codefresh git-commit with husky yarn pre-commit

We are using the git-commit Codefresh step to push updates during a build. We use husky and have pre-commit hooks that rely on yarn. This fails within the step because the dependency is missing.
How should we handle these hooks?
Here is the current step configuration:
push_snapshots:
stage: deploy
title: Commit and push snapshot updates.
type: git-commit
working_directory: ${{clone}}
arguments:
repo: "${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}"
git: github
commit_message: Update snapshot post infra deploy
add:
- "./tenants/*/snapshot/*"
when:
steps:
- name: deploy_infra
on:
- success
and our .husky/pre-commit looks like:
yarn lint-staged
We've tried a couple of things:
Looked for a way to pass the --no-verify flag, but it looks like the only flag supported in the step is --allow-empty, and there is no way to pass arbitrary flags
Tried to override the image with a custom image (that extends git-commit:1.0 and then adds npm and yarn). That image was not actually read by the step, the step just used its usual image.

No files were found with the provided path: No artifacts will be uploaded. Github Actions Fastlane

I'm following this tutorial on setting up CI/CD with github actions for a ios app. I want the action to build and deploy my ios app. I'm fairly sure I'm followed the directions, but when I trigger my action with a push, I get this error:
Run actions/upload-artifact#v2
Warning: No files were found with the provided path: /Users/runner/work/test.ipa
/Users/runner/work/test/*.app.dSYM.zip. No artifacts will be uploaded.
YML Code:-
name: iOS binary build & upload
on:
workflow_dispatch:
push:
jobs:
deploy:
runs-on: macos-latest
steps:
- uses: actions/checkout#v2
- name: Set up ruby env
uses: ruby/setup-ruby#v1
with:
ruby-version: 3.0.0
bundler-cache: true
- name: Upload app-store ipa and dsyms to artifacts
uses: actions/upload-artifact#v2
with:
name: app-store ipa & dsyms
path: |
${{ github.workspace }}/ test.ipa
${{ github.workspace }}/*.app.dSYM.zip
I thought this error might have something to do with my build directory's location, but its in the standard place. Any thoughts on what this might be?
Question: Can someone please explain to me how to solve this and create .ipa, I've tried with above code but no results yet.
Can someone please explain to me How to get Progress?
Any help would be greatly appreciated.
Thanks in advance.

How to implement a multistage Docker build with GitHub Actions?

Problem
I have a multistage (two stages) docker build for my container, lets name it cont, that I want so automate via GitHub Actions. The first stage/docker-image of the build-process does seldomly change and takes very long to build; lets call this cont-build. I want to to reduce build duration by not building cont-build every time I build the whole project.
When running that build locally, I have the image cont-build easily available through my local docker instance. I struggle to transfer this simple availability to GitHub Actions.
I checked the Docker and GitHub docs, but was unable to find a way of implementing this. It is so simple on a local machine, so I thought it cannot be that hard on GitHub-Actions...
Approach
To persist the cont-build image, there seem to be different approaches
Use some sort of GitHub cache. I am not sure about the duration images are cached for.
Pull image from DockerHub, which in the case of long build times may be much faster than building
The second one seems more straight forward and less complex to me. So my approach was to publish cont-build to DockerHub and pull cont-build in the GitHub Action every time I want to build cont.
I tried using uses: Docker://${{ secrets.DOCKERHUB_USERNAME }}/cont-build, but do not know where to place it.
Question
Where/how do I pull the image cont-build that is required by the Dockerfile-cont "Build and push" in the workflow below? Also, if my approach is bad, how is the general approach to multi-stage builds where one stage of the build does not/seldomly change, especially taking into account the fact that GitHub-caches might be deleted after a while?
I realise that I can use something like FROM mydockerID/cont-build:latest in Dockerfile-cont, but that does not seem to be the solution that leverages the whole GitHub-Workflow environment. This would also mean that I have to enter my docker-ID in clear text as opposed to using a GitHub-Secret.
name: CI for cont
on: workflow_dispatch
jobs:
docker:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout#v2
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action#v1
-
name: Login to DockerHub
uses: docker/login-action#v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Build and push
id: docker_build
uses: docker/build-push-action#v2
with:
context: ./Code/
file: ./Code/Dockerfile-cont
push: true
tags: ${{ secrets.DOCKERHUB_USERNAME }}/cont:latest
-
name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
The problem with multi-stage builds is that if you want caching to work you need:
Access to the intermediate stages as well as part of the rebuild.
To use --cache-from to refer to the previous images, including intermediate steps.
If you think about how rebuilds would work, if you are missing intermediate stages the builder will go "huh I guess I don't have that in cache" and rebuild; it can't tell if final stage would need to be rebuilt or not until it's gone through all previous steps.
So you need to do the following song and dance, assuming two stages, "build" and runtime:
Pull "yourimage:latest" and "yourimage:build".
Build and tag each intermediate stage, e.g. "yourimage:build", "yourimage:latest", with --cache-from=yourimage:build --cache-from=yourimage:latest.
Push both those images.
You can see specific details and more extended explanation, and example solution, at https://pythonspeed.com/articles/faster-multi-stage-builds/

Can I run a different workflow for develop merges into master with CircleCI?

My .circleci/config.yml is:
# Javascript Node CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-javascript/ for more details
#
version: 2
jobs:
build:
docker:
# specify the version you desire here
- image: circleci/node:8.11.1
working_directory: ~/repo
steps:
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ checksum "package.json" }}
# fallback to using the latest cache if no exact match is found
- v1-dependencies-
- run: yarn install
- save_cache:
paths:
- node_modules
key: v1-dependencies-{{ checksum "package.json" }}
# run tests!
- run:
command: yarn test
environment:
SOME: 'stuff'
This works well with github currently for my feature branches PR'ed against develop. I want to set up another workflow for when I have a PR of develop into master. I want to trigger the yarn integration script with different environment variables.
There's no built-in way to see the target/base branch of PR from within CircleCI. You'd need to create a GitHub personal token and do some scripting with their API in order to accomplish this.

Gitlab ci cache is empty when on retry

I've got a problem with gitlab-ci multi runner. I have several stages in my setup. Let's pretend build, test. Build works fine, but when it comes to the test stage the job is failing because of some infrastructure issue. Then I fix the reason of failure and want to repeat only the last step assuming the cache between stages is alive. But it fails again because of the empty cache. Here is an example to demonstrate my layout
eg.
stages:
- build
- test
build_step:
stage: build
tags:
- docker
cache:
key: ${CI_PIPELINE_ID}
untracked: true
paths:
- bld/
script:
- rm -rf bld
- mkdir -p bld
- cd bld
- touch build_here
test:
stage: test
cache:
key: ${CI_PIPELINE_ID}
untracked: true
paths:
- bld/
tags:
- docker
script:
- cd bld
- ls -all
Here is my gitlab-runner version:
# gitlab-ci-multi-runner --version
Version: 9.5.1
Git revision: 96b34cc
Git branch: 9-5-stable
GO version: go1.8.3
Built: Wed, 04 Oct 2017 16:26:27 +0000
OS/Arch: linux/amd64
Thanks for your help!
Cache is served on a best-effort basis; to pass data through jobs you need to use artifacts, as it explained in the documentation:
cache - Use for temporary storage for project dependencies. Not useful for keeping intermediate build results, like jar or apk files. Cache was designed to be used to speed up invocations of subsequent runs of a given job, by keeping things like dependencies (e.g., npm packages, Go vendor packages, etc.) so they don't have to be re-fetched from the public internet. While the cache can be abused to pass intermediate build results between stages, there may be cases where artifacts are a better fit.
artifacts - Use for stage results that will be passed between stages. Artifacts were designed to upload some compiled/generated bits of the build, and they can be fetched by any number of concurrent Runners. They are guaranteed to be available and are there to pass data between jobs. They are also exposed to be downloaded from the UI.
You need to use dependencies along with artifacts to obtain what you want

Resources