We are trying to get our multi-stack application deployed using the cdk pipeline library.
We have recently disabled the publishAssetsInParallel flag, as with the default setting our pipeline would create >20 FileAsset objects under the Assets stage, which AWS then complains as being too many CodeBuild projects running parallel.
However, with this property now disabled, I'm getting the following error for the Assets stage:
[Container] 2022/11/14 12:04:24 Phase complete: DOWNLOAD_SOURCE State: FAILED
[Container] 2022/11/14 12:04:24 Phase context status code: YAML_FILE_ERROR Message: stat /codebuild/output/src112668013/src/buildspec-c866864112c35d54804951dbe96b99440c9b891fde-FileAsset.yaml: no such file or directory
I'm assuming this is supposed to be a build spec that is create by cdk pipeline, as we didn't need to create a build spec when things were running in parallel.
Here is the current pipeline code:
const pipeline = new CodePipeline(this, 'Pipeline', {
publishAssetsInParallel: false,
selfMutation: false,
pipelineName: fullStackName('Pipeline', app),
synth: new CodeBuildStep('SynthStep', {
input: CodePipelineSource.codeCommit(repo, repoBranchName, {codeBuildCloneOutput: true}),
buildEnvironment: {computeType: ComputeType.MEDIUM},
installCommands: [
'npm install -g yarn',
'yarn install',
'cd apps/cloud-app',
'yarn install',
'yarn global add aws-cdk'
],
commands: [
'yarn build',
'cdk synth'
],
primaryOutputDirectory: 'apps/cloud-app/cdk.out'
}
)
});
UPDATE:
I reverted the publishAssetsInParallel flag to its default setting to compare, and it seems there is a fundamental difference in the way it creates the FileAsset CodeBuild projects based on this flag. With it enabled, when I inspect the build details for one of the FileAsset projects that is created, I can see under the buildspec section it contains a concrete implementation of a build spec, eg:
{
"version": "0.2",
"phases": {
"install": {
"commands": [
"npm install -g cdk-assets#2"
]
},
"build": {
"commands": [
"cdk-assets --path \"MyStack.assets.json\" --verbose publish \"2357296280127ce793d8dbb13e6c907db22f5dcc57a173ba77fcd19a76d8f444:12345678910-eu-west-2\""
]
}
}
}
With the flag disabled, the buildspec simply contains a pointer to a buildspec file as below, which it then fails to find...
buildspec-c866864112c35d54804951dbe96b99440c9b891fde-FileAsset.yaml
Self-mutation has to be enabled - currently, asset updates mutate the pipeline.
Reference: https://github.com/aws/aws-cdk/issues/9080
Related
My desire is to build the CDK Cloud Formation stacks using AWS Code Pipeline, from the CDK library aws-cdk-lib/pipelines. When running cdk ls in the CLI, everything works as expected. I can successfully deploy the pipeline as well with cdk deploy.
Error Message:
[Container] 2022/12/30 09:18:36 Running command npx cdk synth
Error: .git/HEAD does not exist
at gitHeadPath (/codebuild/output/src224694107/src/backend/node_modules/git-branch/index.js:36:11)
at branch (/codebuild/output/src224694107/src/backend/node_modules/git-branch/index.js:14:28)
at /codebuild/output/src224694107/src/backend/src/context/getContext.ts:11:41
at new Promise (<anonymous>)
at Object.exports.getContext (/codebuild/output/src224694107/src/backend/src/context/getContext.ts:9:12)
at createStack (/codebuild/output/src224694107/src/backend/bin/template.ts:9:25)
at Object.<anonymous> (/codebuild/output/src224694107/src/backend/bin/template.ts:18:1)
at Module._compile (internal/modules/cjs/loader.js:1085:14)
at Module.m._compile (/codebuild/output/src224694107/src/backend/node_modules/ts-node/src/index.ts:1618:23)
at Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
(node:179) UnhandledPromiseRejectionWarning: undefined
(Use `node --trace-warnings ...` to show where the warning was created)
(node:179) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:179) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
CDK Pipeline Code:
this.codePipeline = new CodePipeline(this, `${environment}-pipeline-${appName}`, {
pipelineName: `${environment}-pipeline-${appName}`,
selfMutation: true,
crossAccountKeys: false,
role: this.codePipelineRole,
synth: new ShellStep("Deployment", {
input: CodePipelineSource.codeCommit(this.codeRepository, environment),
installCommands: ["npm uninstall -g aws-cdk", "npm i -g npm#latest", "npm install -g aws-cdk"],
commands: ["cd backend", "npm ci", "npm run build", "npx cdk synth"],
primaryOutputDirectory: "backend/cdk.out",
}),
});
getContext Function:
export const getContext = (app: App): Promise<CDKContext> => {
return new Promise(async (resolve, reject) => {
try {
const currentBranch = await gitBranch();
const environment = app.node.tryGetContext("environments").find((e: any) => e.branchName === currentBranch);
const globals = app.node.tryGetContext("globals");
return resolve({...globals, ...environment});
}
catch (error) {
console.error("error", error);
return reject();
}
})
}
Package.json dependencies:
"dependencies": {
"#aws-cdk/aws-appsync-alpha": "^2.55.1-alpha.0",
"aws-cdk-lib": "^2.58.0",
"aws-sdk": "^2.1278.0",
"constructs": "^10.1.204",
"git-branch": "^2.0.1",
"source-map-support": "^0.5.21"
}
Code Build has two options for cloning repositories:
CodePipeline Default - "AWS CodePipeline uses the default zip format for artifacts in the pipeline. Does not include git metadata about the repository"
Full Clone - "AWS CodePipeline passes metadata about the repository that allows subsequent actions to do a full git clone. Only supported for AWS CodeBuild actions."
Quotes taken from the console.
Therefore, the pipeline definition needed to add a Code Commit source prop to tell the CDK to do a full clone. CDK Docs for options here.
Updating the input:
input: CodePipelineSource.codeCommit(this.codeRepository, environment, {
codeBuildCloneOutput: true
})
codeBuildCloneOutput - "If this is set, the next CodeBuild job clones the repository (instead of CodePipeline downloading the files)." This allows for a full clone of the repository, and will remove the error.
CDK Permissions Update:
This image shows now the CodeBuild can do a GitPull:
Problem: In the AWS Code Pipeline Build Stage, I am receiving the error Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: npm ci. Reason: exit status 254
Desired Outcome: To deploy new changes to the Code Commit Repository, and for the Cloud Formation Stacks to deploy successfully, just like cdk deploy --all.
I have followed this AWS tutorial. Everything deploys successfully.
The Code Pipeline Stack:
export class CodePipelineStack extends Stack {
private readonly codePipeline: CodePipeline;
private readonly codeRepository : Repository;
constructor(scope: Construct, id: string, props: StackProps, context: CDKContext){
super(scope, id, props);
this.codeRepository = new Repository(this, "Repo-CDK", { repositoryName: "Repo-CDK", description: "Building A Repo using CDK Methodology" });
this.codePipeline = new CodePipeline(this, "pipeline", {
pipelineName: "pipeline",
selfMutation: true,
synth: new ShellStep('DeploymentStep', {
input: CodePipelineSource.codeCommit(this.codeRepository, environment),
commands: ['npm ci', 'npm run build', 'npx cdk synth'],
})
});
}
}
I also have app.synth(); at the bottom of the function that builds the stacks.
The reason that npm ci does not work is that CodeBuild uses an old version of npm.
Updating npm before running npm ci fixed the issue:
const pipeline = new pipelines.CodePipeline(this, 'Pipeline', {
synth: new pipelines.ShellStep('Synth', {
...
// Update npm before running commands
installCommands: ['npm i -g npm#latest'],
commands: [
'npm ci',
'npm run build',
'npx cdk synth',
],
}),
});
After following the solution on github, and the solution on Stack Overflow, I am still experiencing the same issue when building a code pipeline with AWS CDK.
Error:
This CDK CLI is not compatible with the CDK library used by your application. Please upgrade the CLI to the latest version.
(Cloud assembly schema version mismatch: Maximum schema version supported is 21.0.0, but found 22.0.0)
This error appears in the Code Build Stage of the Code Pipeline. Sourcing the code from Code Commit works successfully, as the first stage.
CDK Pipeline Code:
As you can see in the code below, I have the install commands of uninstalling the cdk, and then installing it again. This was the recommended solution provided by the document above. Re-ordering does not influence the outcome.
this.codePipeline = new CodePipeline(this, `${environment}-${appName}-`, {
pipelineName: `${environment}-${appName}-`,
selfMutation: true,
crossAccountKeys: false,
role: this.codePipelineRole,
synth: new ShellStep("Deployment", {
input: CodePipelineSource.codeCommit(this.codeRepository, environment, {
codeBuildCloneOutput: true
}),
installCommands: ["npm uninstall -g aws-cdk", "npm i -g npm#latest", "npm install -g aws-cdk"],
commands: [
"cd backend",
"npm ci",
"npm run build",
"npx cdk synth",
],
primaryOutputDirectory: "backend/cdk.out",
})
});
Dependencies in the package.json file:
"dependencies": {
"#aws-cdk/aws-appsync-alpha": "^2.55.1-alpha.0",
"aws-cdk-lib": "^2.58.0",
"aws-sdk": "^2.1278.0",
"constructs": "^10.1.204",
"git-branch": "^2.0.1",
"source-map-support": "^0.5.21"
}
The solution was to do without the npx in npx cdk synth. I removed it and the code worked. This was also experienced when attempting to run npx cdk synth locally.
Solution: cdk synth
Background:
I have a lerna monorepo with yarn workspaces with two packages. I am using rollup as the bundler.
packages/module1/package.json:
{
scripts: {
"watch": "rollup -c rollup.config.js --watch",
"build": "NODE_ENV=production && rollup -c rollup.config.js"
}
}
packages/module2/package.json:
{
scripts: {
"watch": "rollup -c rollup.config.js --watch",
"build": "NODE_ENV=production && rollup -c rollup.config.js"
}
}
Expected Behavior:
lerna run build will run the build scripts for each package.
lerna run watch will run the watch scripts for each package in watch mode.
Current Behavior:
lerna run build works as expected. The build script runs properly for both packages.
lerna run watch just hangs there:
lerna notice cli v3.13.1
lerna info Executing command in 2 packages: "yarn run watch"
[[just hangs here]]
I have tried lerna run --parallel watch, and this only runs once. It exits after rollup completes. In other words, it never seems to be watching.
I believe the command you are looking for is lerna exec. This will run whatever command is passed to it over every package in your Monorepo.
lerna exec --parallel -- yarn build
If each package has the same build step, you could abstract it to the top level package.json like so:
lerna exec --parallel -- rollup -c=rollup.config.js
Which will go into each package and run that rollup command.
Sources:
Adding Rollup to a Monorepo
Creating a Monorepo with Lerna & Yarn Workspaces
It will need some tweaks to enable rollup to watch in parallel for the lerna monorepo.
lerna run --parallel watch
The code above will only run for one package and block the rest of the packages, and here is the code for the inner workings of the rollup. The following code snippet is the Watcher class constructor from the rollup github code base. As you can see, the watcher actually accept an array of configs. So now you only need to write some wrapper code to incorporate all your configs into one and then run the watch from the same config for all packages.
constructor(configs: GenericConfigObject[] | GenericConfigObject) {
this.emitter = new (class extends EventEmitter implements RollupWatcher {
close: () => void;
constructor(close: () => void) {
super();
this.close = close;
// Allows more than 10 bundles to be watched without
// showing the `MaxListenersExceededWarning` to the user.
this.setMaxListeners(Infinity);
}
})(this.close.bind(this));
this.tasks = (Array.isArray(configs) ? configs : configs ? [configs] : []).map(
config => new Task(this, config)
);
this.running = true;
process.nextTick(() => this.run());
}
Postman is used as a popular testing tool for API testing, you can write bunch of unit tests using Postman and execute them as a part of your build process to perform unit testing. The following covers the Jenkins integration of Postman tests.
In order to do that you should have
Exported Postman tests as a collection
Make the APIs available at run time when the tests are performed. (Using docker or by creating a separate build pipeline.)
Node module newman can be used to execute Postman collections. Refer the following Package.json file. Here we are executing the postman collection inside the unit_tests folder using newman, also newman dependency is defined.
package.json
{
"name": "postman-newman-jenkins",
"version": "1.0.0",
"description": "My Test Project",
"directories": {
"tests": "tests"
},
"scripts": {
"newman-tests": "newman run unit_tests/my-collection.postman_collection.json --reporters cli,junit --reporter-junit-export newman.xml --insecure"
},
"author": "Test Author",
"dependencies": {
"newman": "^3.5.2"
}
}
The following is the content of the Jenkinsfile. We are using NPM to install the dependencies and execute tests.
Jenkinsfile
pipeline {
agent { label 'LinuxSlave' }
stages {
stage ('Checkout') {
steps {
checkout scm
}
}
stage('Test'){
steps {
sh 'npm install'
sh 'npm run newman-tests'
junit 'newman.xml'
}
}
}
}
you can avoid installing newman on the machine (slave/ master) and use docker
example pipeline script:
pipeline {
agent any stages {
stage('Test') {
steps {
sh 'docker run -t postman/newman_ubuntu1404 run https://www.getpostman.com/collections/8a0c9bc08f062d12dcda'
}
}
}
}
more info on docker & newman here