To automate our CI process, I need run the Bitbucket pipelines only when the title not starts with "Draft" or "WIP". Atlassian has only this features https://support.atlassian.com/bitbucket-cloud/docs/use-glob-patterns-on-the-pipelines-yaml-file/.
I tried with the regex ^(?!Draft:|WIP:).+ like this:
pipelines:
pull-requests:
'^(?!Draft:|WIP:).+':
- step:
name: Tests
but the pipeline not start under any circumstances (with or withour Draft:/WIP:). Any suggestions?
Note the PR pattern you define in the pipelines is matched against the source branch, not the PR title. Precisely, I used to feature an empty pipeline for PRs from wip/* branches, e.g.
pipelines:
pull-requests:
wip/*:
- step:
name: Pass
script:
- exit 0
"**":
- step:
name: Tests
# ...
But this workflow requires you to work on wip/* branches and changing their source branch later on. This is somewhat cumbersome and developers just did not opt-in.
This works, though.
Related
I am little confused here regarding PR being triggered against main branch?
All branches:
(I know this will trigger pull request from any branch to any branch)
pipelines:
pull-requests:
'**':
Main branch:
(Does this trigger pull request if created from feature/pe-1234 to main?)
pipelines:
pull-requests:
'main':
I want to know what happens if I mention only main. It is not clear in documentation or may be I didn't get it right
The branch name / glob pattern in the pull-request pipeline definition is the source branch that should trigger that pipeline, not the target branch.
E.g. if you were following git-flow instead of github-flow, it would make sense to override the pipeline run by the PR from main to a release/whatever branch so that it simply passes, or does an integration test, but does not perform the usual tests, linting, coverage and whatnot.
pipelines:
pull-requests:
'**': # triggers if no other specific pipeline was triggered
- parallel:
- step: *linting-step
- step: *testing-step
main: # triggers from main to anywhere else
- step:
name: Pass
script:
- exit 0
If following github-flow, you will probably never make a PR from main to anywhere else, so you can safely skip this definition. Only if you wanted PRs from feature/AAA-NNNN branches to trigger a special pipeline besides the testing workflow, you can write an alternate pipeline like
pipelines:
pull-requests:
'**': # triggers if no other specific pipeline was triggered
- parallel:
- step: *linting-step
- step: *testing-step
feature/*: # triggers from feature/* to anywhere else (including to main)
- parallel:
- step: *linting-step
- step: *testing-step
- step: *maybe-hook-issue-tracker-step # ?
so that the simpler default '**' pipeline will not run. But it will run irrespective of the target branch, usually main but not necessarily.
I have a monorepo project that is deployed to 3 environments - testing, staging and production. Deploys to testing come from the next branch, while staging and production from the master branch. Testing deploys should run automatically on every commit to next (but I'm also fine with having to trigger them manually), but deploys from the master branch should be triggered manually. In addition, every deploy may consist of a client push and server push (depending on the files changed). The commands to deploy to each of the hosts are exactly the same, the only thing changing is the host itself and the environment variables.
Therefore I have 2 questions:
Can I make Bitbucket prompt me the deployment target when I manually trigger the pipeline, thus basically letting me choose the set of the env variables to inject into the set sequence of commands? I've seen a screenshot for this in a tutorial, but I lost it and can't find it since.
Can I have parallel sequences of commands? I'd like the server and the client push to run simultaneously, but both of them have different steps. Or do I need to merge those into the same step with multiple scripts to achieve that?
Thank you for your help.
The answer to both of your questions is 'Yes'.
The feature that makes it possible is called custom pipelines. Here is a neat doc that demonstrates how to use them.
There is a parallel keyword which you can use to define parallel steps. Check out this doc for details.
If I'm not misinterpreting the description of your setup, your final pipeline should look very similar to this:
pipelines:
custom:
deploy-to-staging-or-prod: # As you say the steps are the same, only variable values will define the destination.
- variables: # List variable names under here, and Bitbucket will prompt you to supply their values.
- name: VAR1
- name: VAR2
- parallel:
- step:
- ./deploy-client.sh
- step:
- ./deploy-server.sh
branches:
next:
- step:
script:
- ./deploy-to-testing.sh
UPD
If you need to use Deployments instead of providing each variable separately, use can utilise manual type of trigger:
definitions:
steps:
- step: &RunTests
script:
- ./run-tests.sh
- step: &DeployFromMaster
script:
- ./deploy-from-master.sh
pipelines:
branches:
next:
- step:
script:
- ./deploy-to-testing.sh
master:
- step: *RunTests
- parallel:
- step:
<<: *DeployFromMaster
deployment: staging
trigger: manual
- step:
<<: *DeployFromMaster
deployment: production
trigger: manual
Key docs for understanding this pipeline is still this one and this one for yaml anchors. Keep in mind that I introduced a 'RunTests' step on purpose, as
Since a pipeline is triggered on a commit, you can't make the first step manual.
It will act as a stopper for the deploy step which can only be manual due to your requirements.
I am trying to create a base image for my repo that is optionally re-built when branches (merge requests) make changes to dependencies.
Let's say I have this pipeline configuration:
stages:
- Test
- Build
variables:
- image: main
Changes A:
stage: Test
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
changes:
- path/to/a
script:
- docker build -t a .
- docker push a
- echo 'image=a' > dotenv
artifacts:
reports:
dotenv: dotenv
Build:
stage: Build
image: $image
script:
- echo build from $image
Let's say I push to a new branch and the first commit changes /path/to/a, the docker image is build and pushed, the dotenv is updated and the Build job successfully uses image=a.
Now, let's say I push a new commit to the same branch. However, the new commit does not change /path/to/a so the Changes A job does not run. Now, the Build stage pulls the "wrong" default image=main while I would like it to still pull image=a since it builds on top of the previous commit.
Any ideas on how to deal with this?
Is there a way to make rules.changes refer to origin/main?
Any other ideas on how to achieve what I am trying to do?
Is there a way to make rules.changes refer to origin/main?
Yes, there is, since GitLab 15.3 (August 2022):
Improved behavior of CI/CD changes with new branches
Improved behavior of CI/CD changes with new branches
Configuring CI/CD jobs to run on pipelines when certain files are changed by using rules: changes is very useful with merge request pipelines.
It compares the source and target branches to see what has changed, and adds jobs as needed.
Unfortunately, changes does not work well with branch pipelines.
For example, if the pipeline runs for a new branch, changes has nothing to compare to and always returns true, so jobs might run unexpectedly.
In this release we’re adding compare_to to rules:changes for both jobs and workflow:rules, to improve the behavior in branch pipelines.
You can now configure your jobs to check for changes between the new branch and the defined comparison branch.
Jobs that use rules:changes:compare will work the way you expect, comparing against the branch you define.
This is useful for monorepos, where many independent jobs could be configured to run based on which component in the repo is being worked on.
See Documentation and Issue.
You can use it only as part of a job, and it must be combined with rules:changes:paths.
Example:
docker build:
script: docker build -t my-image:$CI_COMMIT_REF_SLUG .
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
paths:
- Dockerfile
compare_to: 'refs/heads/branch1'
In this example, the docker build job is only included when the Dockerfile has changed relative to refs/heads/branch1 and the pipeline source is a merge request event.
There is a project setting, which defines how your MR pipelines are setup. This is only working for Merge requests and can be found in Settings -> Merge Requests under the section Merge options
each commit individually - nothing checked
this means, each commit is treated on its own, and changes checks are done against the triggering commit on it's own
Enabled merged results pipeline
This will merge your MR with the target branch before running the CI Jobs. This also will evaluate all your changes, and take a look at all of them within the MR and not commit wise.
Merge trains
This is a whole different chapter, and for this usecase not relevant. But for completeness i have to mention it see https://gitlab.com/help/ci/pipelines/merge_trains.md
What you are looking for is Option 2 - Merged pipeline results. but as i said, this will only work in Merge Request pipelines and not general pipelines. So you would also need to adapt your rules to something like:
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- path/to/a
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
I am setting up Github Actions for a project repository.
The workflow consists of the following steps:
Building a docker image
Pushing the image to a container registry
Rollout a Kubernetes deployment.
However, I have two different Kubernetes deployments: one for development, and one for production. Hence, I have also two Github Action workflows.
The Github Action workflow for development is triggered everytime that a commit is pushed:
on:
push:
branches:
- master
But I don't want that for my production workflow. I would need a manual trigger, like a Send to production button. I didn't see anything close to that in the docs.
Is there a way to trigger a workflow manually in Github Actions?
How can I split my development and my production workflows to achieve what I want, either on Github Actions, Docker or Kubernetes?
Is there a way to trigger a workflow manually in Github Actions?
You might consider, from July2020:
GitHub Actions: Manual triggers with workflow_dispatch
(Note: or multiple workflows, through the new Composite Run Steps, August 2020)
You can now create workflows that are manually triggered with the new workflow_dispatch event.
You will then see a 'Run workflow' button on the Actions tab, enabling you to easily trigger a run.
You can choose which branch the workflow is run on.
philippe adds in the comments:
One thing that's not mentioned in the documentation: the workflow must exist on the default branch for the "Run workflow" button to appear.
Once you add it there, you can continue developing the action on its own branch and the changes will take effect when run using the button
The documentation goes on:
In addition, you can optionally specify inputs, which GitHub will present as form elements in the UI. Workflow dispatch inputs are specified with the same format as action inputs.
For example:
on:
workflow_dispatch:
inputs:
logLevel:
description: 'Log level'
required: true
default: 'warning'
tags:
description: 'Test scenario tags'
The triggered workflow receives the inputs in the github.event context.
For example:
jobs:
printInputs:
runs-on: ubuntu-latest
steps:
- run: |
echo "Log level: ${{ github.event.inputs.logLevel }}"
echo "Tags: ${{ github.event.inputs.tags }}"
shim adds in the comments:
You can add workflow_dispatch to a workflow that also has other triggers (like on push and / or schedule)
For instance:
on:
workflow_dispatch:
push:
branches:
- master
pull_request:
types: [opened, synchronize, reopened]
EDITED :
Great tweet explaining the use of workflow dispatch :
https://twitter.com/github/status/1321859709075394563?s=19
Is there a way to trigger a workflow manually in Github Actions?
I've got a little hack to do so...
With the watch event, you can manually trigger an action by star or unstar the repo. The code for the event in your workflow is :
on:
watch
types: [started]
I know it's weird but it works! Nevertheless, it's not the best way if it's a public repo with potential stars.
How can I split my development and my production workflows to achieve what I want, either on Github Actions, Docker or Kubernetes?
In Github Actions I mean, you can do multiple workflows / jobs and filter by targeted branches or events. You can combine multiple events for example trigger a workflow for push and with a cron on midnight.
Update: For a slash command style "ChatOps" solution see slash-command-dispatch action. This can allow you to trigger workflows with slash commands (e.g. /deploy) from issue and pull request comments.
Here is a basic example for a deploy slash command. REPO_ACCESS_TOKEN is a repo scoped Personal Access Token
name: Slash Command Dispatch
on:
issue_comment:
types: [created]
jobs:
slashCommandDispatch:
runs-on: ubuntu-latest
steps:
- name: Slash Command Dispatch
uses: peter-evans/slash-command-dispatch#v1
with:
token: ${{ secrets.REPO_ACCESS_TOKEN }}
commands: deploy
The command can be processed in this workflow.
name: Deploy Command
on:
repository_dispatch:
types: [deploy-command]
There are many more options and different setups. See slash-command-dispatch for full usage instructions.
Original Answer:
A repository_dispatch workflow can be manually triggered by a call to the GitHub API as follows.
on:
repository_dispatch:
types: [production-deploy]
[username] is a GitHub username
[token] is a repo scoped Personal Access Token
[repository] is the name of the repository the workflow resides in.
curl -XPOST -u "[username]:[token]" \
-H "Accept: application/vnd.github.everest-preview+json" \
-H "Content-Type: application/json" \
https://api.github.com/repos/[username]/[repository]/dispatches \
--data '{"event_type": "production-deploy"}'
Another way to resolve this with the current Github Action offering is to create a production branch from master when a deploy is needed & trigger deploy action on the production branch. The production branch is essentially a mirror of the master.
on:
push:
branches:
- master
Dev builds/push can happen whenever there is a commit to the master.
on:
push:
branches:
- production
At some point in the release schedule, you can raise the PR to the production branch. This will take care of the prod build/deploy.
Although Sarah's post was the closest and simplest answer to the original question, it is somewhat hacky so we eventually ended up by creating a dev branch to use the following triggers:
Development workflow: triggered when a push is made on the dev branch:
on:
push:
branches:
- dev
Production workflow: triggered when a pull request / merge is made from dev to master:
on:
pull_request:
branches:
- master
Edited for more detail/explanation.
One thing that you can do is call to repository_dispatch. You can view the GitHub documentation for using a repository_dispatch here.
For example, if you have a GitHub Actions workflow that looks like this:
on:
repository_dispatch:
types: [run_tests]
name: Run tests
jobs:
test:
name: Run your tests
runs-on: ubuntu-latest
steps:
- run: |
echo "I just ran all your tests!"
You can create a repository dispatch event by following the steps that are explained on the GitHub v3 API Documentation.
First, create a personal access token (PAT) on GitHub for authentication.
Then, you can run curl like so:
curl \
-H "Authorization: token $YOUR_PAT" \
--request POST \
--data '{"event_type": "run_tests"}' \
https://api.github.com/repos/$USER/$REPOSITORY/dispatches
At the same time, I also wanted to share a small project that I've been working on with a buddy that solves this exact problem.
https://www.actionspanel.app/
ActionsPanel uses this same repository_dispatch API but does so with a GitHub App token so that you don't need to worry about managing your own PAT. This also makes it much easier to trigger your actions across teams with multiple people.
Based on user requests and feedback, we've built in features to specify which branch to send the repository_dispatch to, and we've even built in a way to inject parameters when you want to execute the action.
You configure your buttons with a declarative yaml file that you leave in the repo, and ActionsPanel will read that file and dynamically create your UI for you to trigger your actions.
What GitHub cryptic documentation fails to clarify is that you can have multiple workflow files under .github/workflows, each with its own trigger. For instance, I've a workflow that builds and runs tests on every push and pull request, and another that is triggered manually to publish the artifact.
(ci.yml)
name: CI Pipeline
on: [push, pull_request]
---
(publish.yml)
name: Publish
on:
workflow_dispatch:
We are wondering whether there is any way to add filters to Travis matrix items. In our particular case, we wish to run certain jobs only on specific branches.
The following example would be an ideal way for configuring this scenario, however it doesn't seem to work:
matrix:
include:
- env: BUILD_TYPE=release
branches:
only:
- master
- env: BUILD_TYPE=ci
branches:
only:
- develop
As a workaround, we can exit from the build script immediately by checking the appropriate env vars (TRAVIS_BRANCH), but it is very far from ideal as launching the slave machine and cloning the repo takes a considerable amount of time.
You can now achieve this with the beta feature Conditional Build Stages
jobs:
include:
- stage: release
if: branch = master
env: BUILD_TYPE=release
- stage: ci
if: branch = develop
env: BUILD_TYPE=ci