To automate our CI process, I need run the Bitbucket pipelines only when the title not starts with "Draft" or "WIP". Atlassian has only this features https://support.atlassian.com/bitbucket-cloud/docs/use-glob-patterns-on-the-pipelines-yaml-file/.
I tried with the regex ^(?!Draft:|WIP:).+ like this:
pipelines:
pull-requests:
'^(?!Draft:|WIP:).+':
- step:
name: Tests
but the pipeline not start under any circumstances (with or withour Draft:/WIP:). Any suggestions?
Note the PR pattern you define in the pipelines is matched against the source branch, not the PR title. Precisely, I used to feature an empty pipeline for PRs from wip/* branches, e.g.
pipelines:
pull-requests:
wip/*:
- step:
name: Pass
script:
- exit 0
"**":
- step:
name: Tests
# ...
But this workflow requires you to work on wip/* branches and changing their source branch later on. This is somewhat cumbersome and developers just did not opt-in.
This works, though.
When I push an empty branch to my GitHub repo (perhaps by creating a branch through the GitHub web UI), CircleCI sees it and runs all the tests for the commit at the tip of that branch.
This makes sense, but I've discovered an unfortunate side effect: the CircleCI jobs for the branch clobber the GitHub statuses for master with inaccurate information.
Here's the sequence of events:
I push commit A to master. This runs my whole CircleCI workflow and posts information on test success / failure for A to GitHub using the GitHub Status API.
I create a new branch, mybranch, using the GitHub UI. Its tip is also at commit A.
CircleCI runs all the tests for mybranch. These are also on commit A, so CircleCI re-posts statuses to the GitHub status API for commit A.
This might be OK, except that I have an optimization: I early-out of tests that shouldn't be affected by a PR branch. So if the branch contains no TypeScript files in its diff from master, I won't run the TypeScript tests. Unfortunately, this means that step 3 overrides the GitHub status for the full run on commit A with one that early outs (since there are no commits, there's no diff).
Here's what the "skip" logic looks like:
skip-job-if-no-ts:
description: Skips the rest of the job if files under ts/ have not been modified
steps:
- when:
condition:
not:
or:
# Always run all tests if this is a commit to master branch
- equal: [master, << pipeline.git.branch >>]
steps:
- run:
name: skip-job-if-no-ts
command: |
# Skip the rest of the job if ts/ wasn't modified
TS_CHANGED=$(git diff $(git merge-base HEAD origin/master) --name-only ts/)
if [ ! -n "$TS_CHANGED" ]; then
circleci-agent step halt
fi
How can I prevent CircleCI from running my job on an empty branch? I found an old discussion of this issue on the CircleCI forums, but it was closed with no comments.
I am trying to create a base image for my repo that is optionally re-built when branches (merge requests) make changes to dependencies.
Let's say I have this pipeline configuration:
stages:
- Test
- Build
variables:
- image: main
Changes A:
stage: Test
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
changes:
- path/to/a
script:
- docker build -t a .
- docker push a
- echo 'image=a' > dotenv
artifacts:
reports:
dotenv: dotenv
Build:
stage: Build
image: $image
script:
- echo build from $image
Let's say I push to a new branch and the first commit changes /path/to/a, the docker image is build and pushed, the dotenv is updated and the Build job successfully uses image=a.
Now, let's say I push a new commit to the same branch. However, the new commit does not change /path/to/a so the Changes A job does not run. Now, the Build stage pulls the "wrong" default image=main while I would like it to still pull image=a since it builds on top of the previous commit.
Any ideas on how to deal with this?
Is there a way to make rules.changes refer to origin/main?
Any other ideas on how to achieve what I am trying to do?
Is there a way to make rules.changes refer to origin/main?
Yes, there is, since GitLab 15.3 (August 2022):
Improved behavior of CI/CD changes with new branches
Improved behavior of CI/CD changes with new branches
Configuring CI/CD jobs to run on pipelines when certain files are changed by using rules: changes is very useful with merge request pipelines.
It compares the source and target branches to see what has changed, and adds jobs as needed.
Unfortunately, changes does not work well with branch pipelines.
For example, if the pipeline runs for a new branch, changes has nothing to compare to and always returns true, so jobs might run unexpectedly.
In this release we’re adding compare_to to rules:changes for both jobs and workflow:rules, to improve the behavior in branch pipelines.
You can now configure your jobs to check for changes between the new branch and the defined comparison branch.
Jobs that use rules:changes:compare will work the way you expect, comparing against the branch you define.
This is useful for monorepos, where many independent jobs could be configured to run based on which component in the repo is being worked on.
See Documentation and Issue.
You can use it only as part of a job, and it must be combined with rules:changes:paths.
Example:
docker build:
script: docker build -t my-image:$CI_COMMIT_REF_SLUG .
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
paths:
- Dockerfile
compare_to: 'refs/heads/branch1'
In this example, the docker build job is only included when the Dockerfile has changed relative to refs/heads/branch1 and the pipeline source is a merge request event.
There is a project setting, which defines how your MR pipelines are setup. This is only working for Merge requests and can be found in Settings -> Merge Requests under the section Merge options
each commit individually - nothing checked
this means, each commit is treated on its own, and changes checks are done against the triggering commit on it's own
Enabled merged results pipeline
This will merge your MR with the target branch before running the CI Jobs. This also will evaluate all your changes, and take a look at all of them within the MR and not commit wise.
Merge trains
This is a whole different chapter, and for this usecase not relevant. But for completeness i have to mention it see https://gitlab.com/help/ci/pipelines/merge_trains.md
What you are looking for is Option 2 - Merged pipeline results. but as i said, this will only work in Merge Request pipelines and not general pipelines. So you would also need to adapt your rules to something like:
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- path/to/a
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
I would like to set up github actions in order to perform security scans on docker images. However, the build and push of the image I would like to scan is made by CircleCI. Hence I need to make sure that the images have been properly built and pushed on my repo before performing the security scan.
I know there exists an action to trigger a CircleCI job, but is there any way to trigger the execution of my Github action from Circle CI?
Thank you for the answers or any workaround you could provide :)
You could use a repository dispatch event from you CircleCI pipeline to start a repository workflow (through CURL or through a script).
In that case, your Github repository workflow file will need a repository_dispatch event to trigger it, to perform your security scan job.
Dispatch Request Example
curl \
-X POST \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/<USERNAME>/<REPOSITORY-NAME>/dispatches \
-d '{"event_type":"event_type"}'
Workflow Trigger Example
on: [repository_dispatch]
jobs:
scan:
runs-on: ubuntu-latest
steps:
- name: security scan
run:|
...
Is there a way to show the Jenkins build status on my project's GitHub Readme.md?
I use Jenkins to run continuous integration builds. After each commit it ensures that everything compiles, as well as executes unit and integration tests, before finally producing documentation and release bundles.
There's still a risk of inadvertently committing something that breaks the build. It would be good for users visiting the GitHub project page to know the current master is in that state.
Ok, here's how you can set up Jenkins to set GitHub build statuses. This assumes you've already got Jenkins with the GitHub plugin configured to do builds on every push.
Go to GitHub, log in, go to Settings, Developer Settings, Personal access tokens and click on Generate new token.
Check repo:status (I'm not sure this is necessary, but I did it, and it worked for me).
Generate the token, copy it.
Make sure the GitHub user you're going to use is a repository collaborator (for private repos) or is a member of a team with push and pull access (for organization repos) to the repositories you want to build.
Go to your Jenkins server, log in.
Manage Jenkins → Configure System
Under GitHub Web Hook select Let Jenkins auto-manage hook URLs, then specify your GitHub username and the OAuth token you got in step 3.
Verify that it works with the Test Credential button. Save the settings.
Find the Jenkins job and add Set build status on GitHub commit to the post-build steps
That's it. Now do a test build and go to GitHub repository to see if it worked. Click on Branches in the main repository page to see build statuses.
You should see green checkmarks:
In the meanwhile the UI of Jenkins and GitHub has changed a bit and it took me a while to figure out how to configure Jenkins now correctly. The explanation here is based on Jenkins version 2.121.1.
I also assume that you have already configured your Jenkins Job be triggered by a webhook or by polling.
Those are the steps that I have taken to get it working:
Configure Github: Create Personal Access Token with OAuth Scope repo:status
Configure Jenkins: Configure System and add the OAuth Secret as a GitHub Server - use Secret Text as an authentication method to put the OAuth Secret in there.
Configure your Jenkins Job: Add Set GitHub commit status as Post-build action. Set the Status Result to One of the default messages and statuses.
Check your result on GitHub: Check if you get the build status and build execution duration on your GitHub commit.
Configure Github
Configure Jenkins
Configure Jenkins Job
Result
You will now see the status for your commits and branches:
What I did is quite simple:
Install the Hudson Post Task Plugin
Create a Personal Access Token here : https://github.com/settings/tokens
Add a Post Task Plugin that always put success
curl -XPOST -H "Authorization: token OAUTH TOKEN" https://api.github.com/repos/:organization/:repos/statuses/$(git rev-parse HEAD) -d "{
\"state\": \"success\",
\"target_url\": \"${BUILD_URL}\",
\"description\": \"The build has succeeded!\"
}"
Add a Post Task Plugin that will put failure if "marked build as failure"
curl -XPOST -H "Authorization: token OAUTH TOKEN" https://api.github.com/repos/:organization/:repos/statuses/$(git rev-parse HEAD) -d "{
\"state\": \"failure\",
\"target_url\": \"${BUILD_URL}\",
\"description\": \"The build has failed!\"
}"
You can also add a call to pending at the beginning of tests
curl -XPOST -H "Authorization: token OAUTH TOKEN" https://api.github.com/repos/:organization/:repos/statuses/$(git rev-parse HEAD) -d "{
\"state\": \"pending\",
\"target_url\": \"${BUILD_URL}\",
\"description\": \"The build is pending!\"
}"
This plugin should work: https://wiki.jenkins-ci.org/display/JENKINS/Embeddable+Build+Status+Plugin
You should be able to embed badges like this into your README.md file:
The Commit Status API allows you to see the "Repo Statuses API".
And since April 26th 2013, you now can see the build status on your GitHub repo branch page:
That means it is another way, by visiting the GitHub project page, to see those statuses instead of having only Jenkins.
Starting April 30th, 2013, the API endpoint for commit statuses has been extended to allow branch and tag names, as well as commit SHAs.
There's also the plugin Embeddable Build Status that will give you a badge URL that you can post in your README.md file, and it looks like this:
If you have the GitHub plugin installed on your Jenkins, you can do it in the Post build actions like this:
Add the below line in your README.md file and change both URLs according to your Jenkins project.
[![Build Status](https://jenkins../..project/lastBuild/buildStatus)](https://jenkins../..project/lastBuild/)
In regards with setting up Jenkins and GitHub's protected branch. I'm using Jenkins 2.6, and these are the steps I did to make it work:
On your repository's GitHub webpage:
Navigate to Settings > Branches.
Under Protect branches, click on
the Choose a branch drown down menu and select the branch you want
to set as a Protected branch.
Enable the options as needed.
On the Jenkins Server:
(Make sure you have the Git and GitHub Plugin installed)
Navigate to Manage Jenkins > Configure System.
Under GitHub, set the API URL to https://api.github.com. Though this is the default value.
Select your generated token for the Credentials. If you haven't generated a token yet, click on Advanced... then on Additional actions, you can convert your login and password to token and use it as your credential.
Also, make sure the GitHub account that your Jenkins is using is a collaborator for the repository. I've set it with write permission level.
Hope this helps.
I followed the directions from Alex and it worked.
But, for GitHub Enterprise you need to modify the API URL when adding the server to Jenkins.
For example, if your company is creditcard.com, your URL would be
https://github.creditcard.com/api/v3/
Edit:
I'm no longer using this approach, please use one of the other answers.
Update: what I ended up doing, for our specific case: (above answers were great - thanks!)
Because our build server is not on the internet, we have a script to publish the build status to the gh-pages branch in github.
Start of build stamps failing
End of build stamps success
Project runs after main project to publish results -> build-status, API docs, test reports and test coverage.
GitHub caches images, so we created .htaccess file, that instructs a short cache timeout for the build-status image.
Put this in the directory with the build-status image:
ExpiresByType image/png "access plus 2 minutes"
Here's the build script. The target that publishes to gh-pages is '--publish.site.dry.run'
With less than 400 lines of config, we have:
Compile checks
unit & integration tests
Test Reports
Code Coverage Reports
API Docs
Publishing to Github
. . and this script can be run in or outside of Jenkins, so that:
Developers can run this script before commit, reducing the chance of a broken build that impacts others.
A failure is easy to reproduce locally.
The Results:
Project main page has the build status, updated after each build, along with latest API Docs, test results and test coverage.
Jently updates your GitHub commit status (as described by vonc), but unfortunately they have yet to implement the Repo Status API.
For pipeline projects, you can use the post section like so:
void setBuildStatus(String message, String state) {
step([
$class: "GitHubCommitStatusSetter",
reposSource: [$class: "ManuallyEnteredRepositorySource", url: "https://github.com/my-user/my-repo"],
contextSource: [$class: "ManuallyEnteredCommitContextSource", context: "ci/jenkins/build-status"],
errorHandlers: [[$class: "ChangingBuildStatusErrorHandler", result: "UNSTABLE"]],
statusResultSource: [ $class: "ConditionalStatusResultSource", results: [[$class: "AnyBuildResult", message: message, state: state]] ]
]);
}
pipeline {
agent any
triggers {
githubPush()
}
stages {
stage('Hello') {
steps {
echo 'Hello World'
}
}
}
post {
success {
setBuildStatus("Build succeeded", "SUCCESS");
}
failure {
setBuildStatus("Build failed", "FAILURE");
}
}
}
Just change "https://github.com/my-user/my-repo" to meet your GitHub repository.
Reference: How to set GitHub commit status with Jenkinsfile NOT using a pull request builder
I am adding to this answer, and also to this one. They have answered the question, but they didn't give us a proper intuition about the problem
So, here are my intuitions:
We need to add status to our GitHub commit. This status is based upon the results of our Jenkins build.
To do this, we need to give Jenkins access to the GitHub API, not to the repo. We do this through OAuth, and we can create the token going to the GitHub settings → Developer settings → Personal access tokens. Then for a public GitHub repository just check repo:status, and for a private repository, check the whole repository section
After creating your access token you need to add it to your Jenkins server:
Copy and paste the access token to the GitHub plugin section settings, under your Jenkins configurations
Create a GitHub server. It defaults to api.github.com. And add the access token as a secret text credentials.
The last step is to add a post build settings when you create your pipeline.
Resources:
Here is a section of a YouTube video in which he goes over the above steps.
Read about OAuth