I am rewriting my CircleCI config. Everything was put in only one job and everything was working well, but for some good reasons I want more structure.
Now I have two jobs build and test, and I want the second job to reuse the machine exactly where the build job stopped.
I will later have a third and four job.
My desire would be a line that says I want to reuse the previous machine/executor, built-in from CircleCI.
Other options are Workspaces that save data on CircleCI machine, or building and deploying my own docker that represents the machine after the build job
What is the easiest way to achieve what I want to do ?
Currently, I have basically in my yaml:
jobs:
build:
docker:
- image: cypress/base:14.16.0
steps:
- checkout
- node/install:
install-yarn: true
node-version: '16.13'
- other-long-commands
test:
# NOT GOOD: need an executor
steps:
- run:
name: 'test'
command: 'npx cypress run'
environment:
TEST_SUITE: SMOKE
workflows:
build-and-test:
jobs:
- build
- smoke:
requires:
- build
Can't be done. Workspaces is the solution instead.
My follow up would be, why do you need two jobs? Depending on your use case, pulling steps out into reusable commands might help, or even an orb.
Related
I have a monorepo project that is deployed to 3 environments - testing, staging and production. Deploys to testing come from the next branch, while staging and production from the master branch. Testing deploys should run automatically on every commit to next (but I'm also fine with having to trigger them manually), but deploys from the master branch should be triggered manually. In addition, every deploy may consist of a client push and server push (depending on the files changed). The commands to deploy to each of the hosts are exactly the same, the only thing changing is the host itself and the environment variables.
Therefore I have 2 questions:
Can I make Bitbucket prompt me the deployment target when I manually trigger the pipeline, thus basically letting me choose the set of the env variables to inject into the set sequence of commands? I've seen a screenshot for this in a tutorial, but I lost it and can't find it since.
Can I have parallel sequences of commands? I'd like the server and the client push to run simultaneously, but both of them have different steps. Or do I need to merge those into the same step with multiple scripts to achieve that?
Thank you for your help.
The answer to both of your questions is 'Yes'.
The feature that makes it possible is called custom pipelines. Here is a neat doc that demonstrates how to use them.
There is a parallel keyword which you can use to define parallel steps. Check out this doc for details.
If I'm not misinterpreting the description of your setup, your final pipeline should look very similar to this:
pipelines:
custom:
deploy-to-staging-or-prod: # As you say the steps are the same, only variable values will define the destination.
- variables: # List variable names under here, and Bitbucket will prompt you to supply their values.
- name: VAR1
- name: VAR2
- parallel:
- step:
- ./deploy-client.sh
- step:
- ./deploy-server.sh
branches:
next:
- step:
script:
- ./deploy-to-testing.sh
UPD
If you need to use Deployments instead of providing each variable separately, use can utilise manual type of trigger:
definitions:
steps:
- step: &RunTests
script:
- ./run-tests.sh
- step: &DeployFromMaster
script:
- ./deploy-from-master.sh
pipelines:
branches:
next:
- step:
script:
- ./deploy-to-testing.sh
master:
- step: *RunTests
- parallel:
- step:
<<: *DeployFromMaster
deployment: staging
trigger: manual
- step:
<<: *DeployFromMaster
deployment: production
trigger: manual
Key docs for understanding this pipeline is still this one and this one for yaml anchors. Keep in mind that I introduced a 'RunTests' step on purpose, as
Since a pipeline is triggered on a commit, you can't make the first step manual.
It will act as a stopper for the deploy step which can only be manual due to your requirements.
I am trying to implement CI/CD. So far what I have is a CI using CircleCI, and
circleci/aws-ecr#6.15.0
orb, so when I push to master the image is created and pushed to ECR. That works perfect, but what I want to implement is the deployment.
I think that I am not searching well in google, but I can not find any tutorial or explanation of how to do it.
I read that the orb
circleci/aws-ecs#01.4.0
and seems to do the job but I really dont know how to proceed.
This is my yaml file:
version: 2.1
orbs:
node: circleci/node#4.1.0
aws-ecr: circleci/aws-ecr#6.15.0
aws-ecs: circleci/aws-ecs#01.4.0
workflows:
build_and_push_image:
jobs:
- aws-ecr/build-and-push-image:
repo: node.js-api
tag: "latest,v0.1.${CIRCLE_BUILD_NUM}"
dockerfile: "Dockerfile"
path: .
- aws-ecs/deploy-service-update:
requires:
- aws-ecr/build-and-push-image
family: "${AWS_RESOURCE_NAME_PREFIX}-service"
cluster-name: "${AWS_RESOURCE_NAME_PREFIX}-cluster"
container-image-name-updates: "container=${AWS_RESOURCE_NAME_PREFIX}-service,tag=${CIRCLE_SHA1}"
Error:
An error occurred (ClientException) when calling the
DescribeTaskDefinition operation: Unable to describe task definition.
What I want to achieve is an automated deploy, I choose CircleCI because I thought it would be a good option also with my bitbucket repo, but if there is another way to achieve this I am totally open to hear suggestions.
Thanks!!
If I have a .circleci/config.yml file like so:
version: 2
jobs:
build-node8:
docker:
- image: oresoftware/lmx-circleci:8
steps:
- checkout
- run: ./scripts/circleci/run.sh
build-node9:
docker:
- image: oresoftware/lmx-circleci:9
steps:
- checkout
- run: ./scripts/circleci/run.sh
build-node10:
docker:
- image: oresoftware/lmx-circleci:10
steps:
- checkout
- run: ./scripts/circleci/run.sh
build-node11:
docker:
- image: oresoftware/lmx-circleci:11
steps:
- checkout
- run: ./scripts/circleci/run.sh
build-node12:
docker:
- image: oresoftware/lmx-circleci:12
steps:
- checkout
- run: ./scripts/circleci/run.sh
there are 5 jobs listed here, but when the builds start, only 4 jobs run in parallel. Is there a way to run more than 4 jobs in parallel is there a hard limit there?
My guess is that under workflows, I can change the parallelism level?
workflows:
version: 2
build_nodejs:
parallelism: 5
jobs:
- build-node8
- build-node9
- build-node10
- build-node11
- build-node12
perhaps this requires a paid account tho?
Short Answer:
CircleCi lets you run as many jobs in parallel as you want, as long as your payment plan has enough containers to serve each job.
I suspect that your plan only has 4 containers. You can check to see how many containers you have in the settings tab in CircleCi
In the below example I have a total of 2 containers available: 1 paid + 1 free. So right now at max I can only run 2 jobs in parallel. I can pay an extra $50 per month per container to add additional containers thoughs.
Additional Details:
This article gives a great overview of how to configure circle ci jobs to run in parallel (and it actually has an example where 5 jobs run in parallel). https://circleci.com/blog/decrease-your-build-times-by-running-jobs-in-parallel-with-workflows/
Regarding the config file code snippet you pasted in your question - It looks fine (Tho you don't need the parallelism: 5 flag, since circle will use all available plan capacity automatically)
Can you please check to see how many containers are in your plan and then report back?
FYI - CircleCi container and concurrent job plan info:
https://circleci.com/pricing/
Is it possible to have another job run in the context of another job? I have some jobs that have some steps in common, and I don't want to repeat these steps in the different jobs.
push-production-image:
docker:
- image: google/cloud-sdk:latest
working_directory: ~/app
steps:
- setup-gcp-docker
- run: docker push [image]
No you cannot however YAML itself has a way to solve this problem with what is called YAML Anchors and Aliases.
Here's a blog post I wrote on how to do specifically this: https://circleci.com/blog/circleci-hacks-reuse-yaml-in-your-circleci-config-with-yaml/
I'm iterating on adding database migrations to a project. For the first step, I've made a repository that runs migrations. Now I need to make it so these migrations run against the stage/prod environment. I do not want this to happen on every commit. Does circle ci provide a way to have a button that I can click on to run a job?
I think ideally I'd have 2 buttons. One for running migrations on stage, one for running them on prod. Is this possible?
There is a manual approval process for workflows.
https://circleci.com/docs/2.0/workflows/#holding-a-workflow-for-a-manual-approval
workflows:
version: 2
build-test-and-approval-deploy:
jobs:
- build
- test1:
requires:
- build
- test2:
requires:
- test1
- hold:
type: approval
requires:
- test2
- deploy:
requires:
- hold
It's pretty limited. You can't use it to start a build.