Is it possible to share steps between branches and still run branch specific steps? For example, the develop and release branch has the same build process, but uploaded to separate S3 buckets.
pipelines:
default:
- step:
script:
- cd source
- npm install
- npm build
develop:
- step:
script:
- s3cmd put --config s3cmd.cfg ./build s3://develop
staging:
- step:
script:
- s3cmd put --config s3cmd.cfg ./build s3://staging
I saw this post (Bitbucket Pipelines - multiple branches with same steps) but it's for the same steps.
Use YAML anchors:
definitions:
steps:
- step: &Test-step
name: Run tests
script:
- npm install
- npm run test
- step: &Deploy-step
name: Deploy to staging
deployment: staging
script:
- npm install
- npm run build
- fab deploy
pipelines:
default:
- step: *Test-step
- step: *Deploy-step
branches:
master:
- step: *Test-step
- step:
<<: *Deploy-step
name: Deploy to production
deployment: production
trigger: manual
Docs: https://confluence.atlassian.com/bitbucket/yaml-anchors-960154027.html
Although it's not officially supported yet, you can pre-define steps now.
You can use yaml anchors.
I got this tip from bitbucket staff when I had an issue running the same steps across a subset of branches.
definitions:
step: &Build
name: Build
script:
- npm install
- npm build
pipelines:
default:
- step: *Build
branches:
master:
- step: *Build
- step:
name: deploy
# do some deploy from master only
I think Bitbucket can't do it. You can use one pipeline and check the branch name:
pipelines:
default:
- step:
script:
- cd source
- npm install
- npm build
- if [[ $BITBUCKET_BRANCH = develop ]]; then s3cmd put --config s3cmd.cfg ./build s3://develop; fi
- if [[ $BITBUCKET_BRANCH = staging ]]; then s3cmd put --config s3cmd.cfg ./build s3://staging; fi
The two last lines will be executed only on the specified branches.
You can define and re-use steps with YAML Anchors.
anchor & to define a chunk of configuration
alias * to refer to that chunk elsewhere
And the source branch is saved in a default variable called BITBUCKET_BRANCH
You'd also need to pass the build results (in this case the build/ folder) from one step to the next, which is done with artifacts.
Combining all three will give you the following config:
definitions:
steps:
- step: &build
name: Build
script:
- cd source
- npm install
- npm build
artifacts: # defining the artifacts to be passed to each future step.
- ./build
- step: &s3-transfer
name: Transfer to S3
script:
- s3cmd put --config s3cmd.cfg ./build s3://${BITBUCKET_BRANCH}
pipelines:
default:
- step: *build
develop:
- step: *build
- step: *s3-transfer
staging:
- step: *build
- step: *s3-transfer
You can now also use glob patterns as mentioned in the referenced post and steps for both develop and staging branches in one go:
"{develop,staging}":
- step: *build
- step: *s3-transfer
Apparently it's in the works. Hopefully available soon.
https://bitbucket.org/site/master/issues/12750/allow-multiple-steps?_ga=2.262592203.639241276.1502122373-95544429.1500927287
Related
I have 1 common project including both frontend and backend, sometimes backend sometimes frontend is getting new commits but my pipeline yml is working for both of them and deploying both to server even if they have no change. In other words, If I add 1 line of code to frontend, pipeline is deploying backend too. Here is my bitbucket-pipeline.yml
This is an example Starter pipeline configuration
pipelines:
branches:
master:
- step:
name: 'Frontend Build'
image: node:16.4.2
script:
- cd myfrontend
- npm install
- step:
name: 'Backend Build and Package'
image: maven:3.8.3-openjdk-17
script:
- cd myfolder
- mvn clean package
artifacts:
- mybackend/target/mybackend-0.0.1-SNAPSHOT.jar
- step:
name: 'Deploy artifacts to Droplet'
deployment: production
script:
- pipe: atlassian/scp-deploy:0.3.3
variables:
USER: root
SERVER: 138.138.138.138
REMOTE_PATH: '/root/artifacts/target/'
LOCAL_PATH: mybackend/target/mybackend-0.0.1-SNAPSHOT.jar
- pipe: atlassian/scp-deploy:0.3.3
variables:
USER: root
SERVER: 138.138.138.138
REMOTE_PATH: '/root/artifacts'
LOCAL_PATH: mybackend/Dockerfile
- pipe: atlassian/scp-deploy:0.3.3
variables:
USER: root
SERVER: 138.138.138.138
REMOTE_PATH: '/root/automation-temp-folder'
LOCAL_PATH: mybackend/README.MD
In this example the frontend is not deployed but I will activate it. So What I need is that I want to execute a step according to that which folder/project got commit to in it. e.g. If there is a commit under mybackend then only deploy mybackend and if front end.. so on. Is it possible to execute a step for a specific folder ?
Yes, this is achievable by using condition keyword:
This allows steps to be executed only when a condition or rule is satisfied. Currently, the only condition supported is changesets. Use changesets to execute a step only if one of the modified files matches the expression in includePaths.
Your end result should look similar to this:
pipelines:
branches:
master:
- step:
name: 'Frontend Build'
image: node:16.4.2
script:
- cd myfrontend
- npm install
condition:
changesets:
includePaths:
- "myfrontend/**"
- step:
name: 'Backend Build and Package'
image: maven:3.8.3-openjdk-17
script:
- cd myfolder
- mvn clean package
condition:
changesets:
includePaths:
- "myfolder/**"
artifacts:
- mybackend/target/mybackend-0.0.1-SNAPSHOT.jar
- step:
name: 'Deploy artifacts to Droplet'
deployment: production
script:
- pipe: atlassian/scp-deploy:0.3.3
variables:
USER: root
SERVER: 138.138.138.138
REMOTE_PATH: '/root/artifacts/target/'
LOCAL_PATH: mybackend/target/mybackend-0.0.1-SNAPSHOT.jar
- pipe: atlassian/scp-deploy:0.3.3
variables:
USER: root
SERVER: 138.138.138.138
REMOTE_PATH: '/root/artifacts'
LOCAL_PATH: mybackend/Dockerfile
- pipe: atlassian/scp-deploy:0.3.3
variables:
USER: root
SERVER: 138.138.138.138
REMOTE_PATH: '/root/automation-temp-folder'
LOCAL_PATH: mybackend/README.MD
condition:
changesets:
includePaths:
- "myfolder/**"
See here for more details.
Below script giving an error
The deployment environment 'staging' in your bitbucket-pipelines.yml file occurs multiple times in the pipeline. Please refer to our documentation for valid environments and their ordering.
image: python:3.8
options:
docker: true
pipelines:
branches:
master:
- step:
deployment: staging
name: Setup stage
script:
- echo ${db_name}
- step:
deployment: staging
name: Setup cli prod
script:
- echo ${db_name}
- step:
deployment: staging
name: Setup cli sandbox
script:
- echo ${db_name}
I want to use same environment variable (staging) in all steps of my pipeline. Please guide me how to do this.
This can't be done because steps marked as deployments must be unique.
Deployment variables are those needed to deploy the VCS to a particular deployment stage. If you are setting up administrative tasks as pipelines, those are NOT deployments.
Or if those are actual deployments, abide by the error message you are getting and make sure the declared deployment stage is unique:
image: python:3.8
options:
docker: true
definitions:
yaml-anchors:
- &deploy-step
script:
- echo ${db_name}
pipelines:
branches:
master:
- step:
<<: *deploy-step
deployment: staging
name: Deploy staging
- step:
<<: *deploy-step
deployment: production
name: Deploy prod
- step:
deployment: sandbox
name: Deploy sandbox
ALM used Bitbucket Cloud
CI system used Bitbucket Cloud
Languages of the repository: Angular (Other (for JS, TS, Go, Python, PHP, …))
Error observed
ERROR: Error during SonarScanner execution
ERROR: Not authorized. Please check the property sonar.login or SONAR_TOKEN env variable
Steps to reproduce
SONAR_TOKEN already generated and added to my ENV_VAR
Bitbucket.yaml
image: ‘node:12.22’
clone:
depth: full # SonarCloud scanner needs the full history to assign issues properly
definitions:
caches:
sonar: ~/.sonar/cache # Caching SonarCloud artifacts will speed up your build
steps:
step: &build-test-sonarcloud
name: Build, test and analyze on SonarCloud
caches:
- sonar
script:
- pipe: sonarsource/sonarcloud-scan:1.2.1
variables:
EXTRA_ARGS: ‘-Dsonar.host.url=https://sonarcloud.io -Dsonar.login=${SONAR_TOKEN}’
step: &check-quality-gate-sonarcloud
name: Check the Quality Gate on SonarCloud
script:
- pipe: sonarsource/sonarcloud-quality-gate:0.1.4
pipelines:
branches
Potential workaround
No idea.
if you already install the sonar cloud app to your workspace environment, there is no need to give the sonar url again. The integration process is handling the URL part. Also, you should add your Sonar token variable to Workspace or repo environment. After that, you should login to Sonar Cloud organization account and bind your repo to SonarCloud to be able to evaluate it by Sonar Cloud. Here is my Sonar Cloud setup;
bitbucket-pipelines.yml file,
image:
name: <base image>
clone:
# SonarCloud scanner needs the full history to assign issues properly
depth: full
definitions:
caches:
# Caching SonarCloud artifacts will speed up your build
sonar: ~/.sonar/cache
pipelines:
pull-requests:
'**':
- step:
name: "Code Quality and Security on PR"
script:
- pipe: sonarsource/sonarcloud-scan:1.2.1
variables:
SONAR_TOKEN: '$SONAR_CLOUD_TOKEN'
SONAR_SCANNER_OPTS: -Xmx512m
DEBUG: "true"
branches:
master:
- step:
name: "Code Quality and Security on master"
script:
- pipe: sonarsource/sonarcloud-scan:1.2.1
variables:
SONAR_TOKEN: '$SONAR_CLOUD_TOKEN'
SONAR_SCANNER_OPTS: -Xmx512m
DEBUG: "true"
tags:
'*.*.*-beta*':
- step:
name: "Image Build & Push"
services:
- docker
caches:
- docker
clone:
depth: 1
script:
- <build script>
- step:
name: "Deploy"
deployment: beta
clone:
enabled: false
script:
- <deploy script>
'*.*.*-prod':
- step:
name: "Image Build & Push"
services:
- docker
caches:
- docker
clone:
depth: 1
script:
- <build script>
- step:
name: "Deploy"
deployment: prod
clone:
enabled: false
script:
- <deploy script>
sonar-project.properties file,
sonar.organization=<sonar cloud organization name>
sonar.projectKey=<project key>
sonar.projectName=<project name>
sonar.sources=<sonar evaluation path>
sonar.language=<repo language>
sonar.sourceEncoding=UTF-8
I have this simple config:
version: 2
jobs:
build:
machine: true
steps:
- checkout
- run: |
docker-compose -f docker-compose.test.yml build
test:
machine: true
working_directory: ~/app
steps:
- checkout
- run:
command: make test
name: Test
workflows:
version: 2
build_and_test:
jobs:
- build
- test:
requires:
- build
My test job, the second one, fails for some reasons. But whenever I check the logs on CircleCI, I can see that the image is always built in this job. I was expecting that the job would use the container that was built in the build job. So question is, why are containers not shared across jobs?
Struggling for a few last days to migrate from CircleCI 1.0 to 2.0 and while the build process is done, deployment is still a big issue. CircleCI documentation is not really of a big help.
Here is a similar config.yml to what I have:
version 2
jobs:
build:
docker:
- image: circleci/node:8.9.1
steps:
- checkout
- setup_remote_docker
- run
name: Install required stuff
command: [...]
- run:
name: Build
command: docker build -t project .
deploy:
docker:
- image: circleci/node:8.9.1
steps:
- checkout
- run:
name: Deploy
command: |
bash scripts/deploy/deploy.sh
docker tag project [...]
docker push [...]
workflows:
version: 2
build-deploy:
jobs:
- build
- deploy:
requires:
- build
filters:
branches:
only: develop
The issue is in deploy job. I have to specify the docker: -image point but I want to reuse the environment from build job where all the required stuff is installed already. Surely, I could just install them in deploy job, but having multiple deploy jobs leads to code duplication which is something I do not want.
you probably want to persist to workspace and attach it in your deploy job.
you wont need to use '- checkout' after that
https://circleci.com/docs/2.0/configuration-reference/#persist_to_workspace
jobs:
build:
docker:
- image: circleci/node:8.9.1
steps:
- checkout
- setup_remote_docker
- run
name: Install required stuff
command: [...]
- run:
name: Build
command: docker build -t project .
- persist_to_workspace:
root: ./
paths:
- ./
deploy:
docker:
- image: circleci/node:8.9.1
steps:
- attach_workspace:
at: ./
- run:
name: Deploy
command: |
bash scripts/deploy/deploy.sh
docker tag project [...]
docker push [...]
workflows:
version: 2
build-deploy:
jobs:
- build
- deploy:
requires:
- build
filters:
branches:
only: develop
If you label the image built by the build stage, you can then reference it in the deploy stage: https://docs.docker.com/compose/compose-file/#labels