How to create VPC that can be shared across stacks? - aws-cdk

I am trying to wrap my head around how to create a reusable VPC that can be used across multiple stacks using AWS CDK. I want to be able to create different stack per project and then be able to import the VPC that should be assigned to the different stacks. I also want to create this using a good structure where I can deploy different stacks at different times (meaning: I do not want to deploy all stacks at once).
I have tried the following approach but this will create a new VPC per stack which is not what I want to achieve, instead I would like to create my VPC once and then if it already exists it will simply reuse the previous created.
app.ts
import cdk = require('#aws-cdk/core');
import { Stack1 } from '../lib/stack1';
import { Stack2 } from '../lib/stack2';
const app = new cdk.App();
new Stack1(app, "Stack1");
new Stack2(app, "Stack2");
stack1.ts
import cdk = require('#aws-cdk/core');
import { Configurator } from './configurators/configurator'
export class Stack1 extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const configurator = new Configurator(scope, "Stack1");
// later reuse vpc from configurator using configurator.vpc
}
}
stack2.ts
import cdk = require('#aws-cdk/core');
import { Configurator } from './configurators/configurator'
export class Stack2 extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const configurator = new Configurator(scope, "Stack2");
// later reuse vpc from configurator using configurator.vpc
}
}
configurator.ts
import cdk = require('#aws-cdk/core');
import ec2 = require("#aws-cdk/aws-ec2");
export class Configurator {
vpc: ec2.Vpc;
constructor(scope: cdk.Construct, name: string) {
this.vpc = new ec2.Vpc(scope, "MyVPC", {
maxAzs: 3
});
}
}
After doing
cdk synth
cdk deploy Stack1
cdk deploy Stack2
This will create 2 VPCs and not reusing 1 VPC as I would like. I will deploy the stacks to same account and region.
How can I change my approach in order to achieve the output I am looking for? I want to be able to deploy my stacks independently of each other.

If you intend to reuse the VPC in different stacks, I'd recommend placing it in a separate stack, since your VPC stack will have a different lifecycle than your application stacks.
Here's what I'd do. I hope you don't mind a bit of Python :)
First, define your VPC in VpcStack:
class VpcStack(core.Stack):
def __init__(self, app: core.App, id: str, **kwargs) -> None:
super().__init__(app, id, **kwargs)
aws_ec2.Vpc(self, 'MyVPC', max_azs=3)
Then look it up in another stack:
class Stack1(core.Stack):
def __init__(self, app: core.App, id: str, **kwargs) -> None:
super().__init__(app, id, **kwargs)
# Lookup the VPC named 'MyVPC' created in stack 'vpc-stack'
my_vpc = aws_ec2.Vpc.from_lookup(self, 'MyVPC', vpc_name=f'vpc-stack/MyVPC')
# You can now use the VPC in ECS cluster, etc.
And this would be your cdk_app.py:
app = core.App()
vpc = VpcStack(app, 'vpc-stack')
stack1 = Stack1(app, 'stack1')

I tried 0x32e0edfb answer and got some problem.
so I fix like this.
VPC Stack
class VpcStack(core.Stack):
def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
super().__init__(scope, id, **kwargs)
self.eks_vpc = ec2.Vpc(self, 'eks-vpc',
cidr='10.1.0.0/16',
max_azs=2
)
share VPC to other Stack
class EksClusterStack(core.Stack):
def __init__(self, scope: core.Construct, id: str, props: ec2.Vpc, **kwargs) -> None:
super().__init__(scope, id, **kwargs)
cluster = eks.Cluster(self, 'eks-control-plane',
vpc=props,
default_capacity=0
)
and then app.py file
app = core.App()
vpc_stack = VpcStack(app, 'vpc-stack')
eks_cluster_stack = EksClusterStack(app, 'eks-cluster', vpc_stack.eks_vpc)
eks_cluster_stack.add_dependency(vpc_stack)
app.synth()
from_lookup is much better used on already existing VPC.
so I choose to use share-vpcs to share VPC information.
from_lookup only does the API call once - then, the data is cached in the cdk.context.json file, which should be committed to source control
That problem was when I recreating the same VPC.
cdk.context.json didn't update to lasted version. So when I use from_lookup always get old vpc-id.
I need to use cdk context --clear command and then deploy again. cdk.context.json would get lasted version vpc-id.
Finally, it can work properly on from_lookup method.
ref:
https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-eks/test/integ.eks-kubectl.lit.ts
https://docs.aws.amazon.com/cdk/latest/guide/context.html

Related

How do I get AWS CDK stack env values for bootstrapping an environment?

AWS and other sources consider explicitly specifying the AWS account and region for each stack as best practice. I'm trying to write a CI pipeline that will bootstrap my environments. However, I'm not seeing any straight-forward way to retrieve the stack's explicit env values from here:
regions.forEach((region) =>
new DbUpdateStack(app, `${stackBaseName}-prd-${region}`, {
env: {
account: prdAccount,
region: region
},
environment_instance: 'prd',
vpc_id: undefined,
})
);
EG, base-name-prd-us-east-1 knows the region and account as defined in the code but how do I access this from the command line without doing something hacky?
I need to run cdk bootstrap with those values and I don't want to duplicate them.
The Cloud Assembly module can introspect an App's stack environments. Synth the app, then instantiate a CloudAssembly class by pointing at the cdk output directory:
import * as cx_api from '#aws-cdk/cx-api';
(() => {
const cloudAssembly = new cx_api.CloudAssembly('cdk.out');
const appEnvironments = cloudAssembly.stacks.map(stack => stack.environment);
console.log(appEnvironments);
})();
Result:
[
{
account: '123456789012',
region: 'us-east-1',
name: 'aws://123456789012/us-east-1',
},
];

AWS CDK Pipelines using with an existing codepipeline

The documentation of #aws-cdk/pipelines seems to suggest that a CDK pipeline can be added to an existing #aws-cdk/aws-codepipeline/Pipeline, using the codePipeline prop: https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_pipelines.CodePipeline.html
codePipeline? Pipeline An existing Pipeline to be reused and built upon.
However, I am not able to get this to work and am experiencing multiple errors at the cdk synth step, depending on how I try to set it up. As far as I can tell there isn't really any documentation yet to cover this scenario.
Essentially, we are trying to create a pipeline that runs something like:
clone
lint / typecheck / unit test
cdk deploy to test environment
integration tests
deploy to preprod
smoke test
manual approval
deploy to prod
I guess it's just not clear the difference between this codebuild pipeline and the cdk pipeline. Also, the naming convention of stages seems a little unclear - referencing this issue: https://github.com/aws/aws-cdk/issues/15945
See: https://github.com/ChrisSargent/cdk-issues/blob/pipelines/lib/cdk-test-stack.ts and below:
import * as cdk from "#aws-cdk/core";
import * as pipelines from "#aws-cdk/pipelines";
import * as codepipeline from "#aws-cdk/aws-codepipeline";
import * as codepipeline_actions from "#aws-cdk/aws-codepipeline-actions";
export class CdkTestStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const cdkInput = pipelines.CodePipelineSource.gitHub(
"ChrisSargent/cdk-issues",
"pipelines"
);
// Setup the code source action
const sourceOutput = new codepipeline.Artifact();
const sourceAction = new codepipeline_actions.GitHubSourceAction({
owner: "ChrisSargent",
repo: "cdk-issues",
branch: "pipelines",
actionName: "SourceAction",
output: sourceOutput,
oauthToken: cdk.SecretValue.secretsManager("git/ChrisSargent"),
});
const pipeline = new codepipeline.Pipeline(this, "Pipeline", {
stages: [
{
actions: [sourceAction],
stageName: "GitSource",
},
],
});
const cdkPipeline = new pipelines.CodePipeline(this, "CDKPipeline", {
codePipeline: pipeline,
synth: new pipelines.ShellStep("Synth", {
// Without input, we get: Error: CodeBuild action 'Synth' requires an input (and the pipeline doesn't have a Source to fall back to). Add an input or a pipeline source.
// With input, we get:Error: Validation failed with the following errors: Source actions may only occur in first stage
input: cdkInput,
commands: ["yarn install --frozen-lockfile", "npx cdk synth"],
}),
});
// Produces: Stage 'PreProd' must have at least one action
// pipeline.addStage(new MyApplication(this, "PreProd"));
// Produces: The given Stage construct ('CdkTestStack/PreProd') should contain at least one Stack
cdkPipeline.addStage(new MyApplication(this, "PreProd"));
}
}
class MyApplication extends cdk.Stage {
constructor(scope: cdk.Construct, id: string, props?: cdk.StageProps) {
super(scope, id, props);
console.log("Nothing to deploy");
}
}
Any guidance or experience with this would be much appreciated.
I'm able to achieve something similar by adding waves/stages with only pre and post steps into the CDK pipelines, sample code is listed as below, I'm amending on your original code snippet:
import * as cdk from "#aws-cdk/core";
import * as pipelines from "#aws-cdk/pipelines";
import * as codepipeline from "#aws-cdk/aws-codepipeline";
import * as codepipeline_actions from "#aws-cdk/aws-codepipeline-actions";
export class CdkTestStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const cdkInput = pipelines.CodePipelineSource.gitHub(
"ChrisSargent/cdk-issues",
"pipelines"
);
const cdkPipeline = new pipelines.CodePipeline(this, "CDKPipeline", {
selfMutation: true,
crossAccountKeys: true, //can be false if you don't need to deploy to a different account.
pipelineName,
synth: new pipelines.ShellStep("Synth", {
// Without input, we get: Error: CodeBuild action 'Synth' requires an input (and the pipeline doesn't have a Source to fall back to). Add an input or a pipeline source.
// With input, we get:Error: Validation failed with the following errors: Source actions may only occur in first stage
input: cdkInput,
commands: ["yarn install --frozen-lockfile", "npx cdk synth"],
primaryOutputDirectory: 'cdk.out'
}),
});
// add any additional test step here, they will run parallels in waves
cdkPipeline.addWave('test', {post: [provideUnitTestStep(this, 'unitTest')]});
// add a manual approve step if needed.
cdkPipeline.addWave('promotion', {post: [new ManualApprovalStep('PromoteToUat')]});
// Produces: Stage 'PreProd' must have at least one action
// pipeline.addStage(new MyApplication(this, "PreProd"));
// Produces: The given Stage construct ('CdkTestStack/PreProd') should contain at least one Stack
cdkPipeline.addStage(new MyApplication(this, "PreProd"));
}
}
class MyApplication extends cdk.Stage {
constructor(scope: cdk.Construct, id: string, props?: cdk.StageProps) {
super(scope, id, props);
console.log("Nothing to deploy");
}
}
What's noticing is that you might need to covert the way you write your Codebuild action to the new cdk CodeBuildStep. A sample unit test step could is like below:
const provideUnitTestStep = (
id: string
): cdkpipeline.CodeBuildStep => {
const props: CodeBuildStepProps = {
partialBuildSpec: codebuild.BuildSpec.fromObject({
version: '0.2',
env: {
variables: {
DEFINE_VARIBLES: 'someVariables'
}
},
phases: {
install: {
commands: [
'install some dependencies',
]
},
build: {
commands: [
'run some test!'
]
}
}
}),
commands: [],
buildEnvironment: {
buildImage: codebuild.LinuxBuildImage.STANDARD_5_0
}
};
return new cdkpipeline.CodeBuildStep(`${id}`, props);
};
It's not so trivial(and straight forward enough) to retrive the underline CodeBuild project Role, you will need to pass in rolePolicyStatements property in the CodeBuildStep props to grant extra permission needed for your test.
First of all, the error Pipeline must have at least two stages is correct.
You only got the GitHub checkout/clone command as a single stage.
For a second stage, you could use a CodeBuild project to compile/lint/unit test... as you mentioned.
However, what would you like to do with your compiled artifacts then?
Build containers to deploy them later?
If so, there are better ways with CDK of doing this (DockerImageAsset).
This also could save up your preexisting pipeline and you can use the CDK Pipeline directly.
Could you please try to set the property restartExecutionOnUpdate: true,
of your regular Pipeline, like in my following snippet?
const pipeline = new codepipeline.Pipeline(this, "Pipeline", {
restartExecutionOnUpdate: true,
stages: [
{
actions: [sourceAction],
stageName: "GitSource",
},
],
});
This is needed for the self-mutation capability of the CDK pipeline.
This happened to me when I was creating a pipeline in a stack without specifically defined account and region.
Check if you have env like this:
new CdkStack(app, 'CdkStack', {
env: {
account: awsProdAccount,
region: defaultRegion,
}
});

CDK generating empty targets for CfnCrawler

I'm using CDK Python API to define a Glue crawler, however, the CDK generated template contains empty 'Targets' block in the Crawler resource.
I've not been able to find an example to emulate. I've tried varying the definition of the targets object, but the object definition seems to be ignored by CDK.
from aws_cdk import cdk
BUCKET='poc-1-bucket43879c71-5uabw2rni0cp'
class PocStack(cdk.Stack):
def __init__(self, app: cdk.App, id: str, **kwargs) -> None:
super().__init__(app, id)
from aws_cdk import (
aws_iam as iam,
aws_glue as glue,
cdk
)
glue_role = iam.Role(
self, 'glue_role',
assumed_by=iam.ServicePrincipal('glue.amazonaws.com'),
managed_policy_arns=['arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole']
)
glue_crawler = glue.CfnCrawler(
self, 'glue_crawler',
database_name='db',
role=glue_role.role_arn,
targets={"S3Targets": [{"Path": f'{BUCKET}/path/'}]},
)
I expect the generated template to contain a valid 'targets' block with a single S3Target. However, cdk synth outputs a template with empty Targets in the AWS::Glue::Crawler resource:
gluecrawler:
Type: AWS::Glue::Crawler
Properties:
DatabaseName: db
Role:
Fn::GetAtt:
- glueroleFCCAEB57
- Arn
Targets: {}
Resolved, thanks to a clever colleague!
Changing "S3Targets" to "s3Targets", and "Path" to "path" resolved the issue. See below.
Hi Bob,
When I use typescript, the following works for me:
new glue.CfnCrawler(this, 'glue_crawler', {
databaseName: 'db',
role: glue_role.roleArn,
targets: {
s3Targets: [{ path: "path" }]
}
}
When I used Python, the following appears working too:
glue_crawler = glue.CfnCrawler(
self, 'glue_crawler',
database_name='db',
role=glue_role.role_arn,
targets={
"s3Targets": [{ "path": f'{BUCKET}/path/'}]
},
)
In Typescript, TargetsProperty is an interface with s3Targets as a property. And in
s3Targets, path is a property as well. I guess during the JSII transformation, it forces
us to use the same names in Python instead of the initial CFN resource names.
A more general way to approach this problem is to dig inside the cdk library in 2 steps:
1.
from aws_cdk import aws_glue
print(aws_glue.__file__)
(.env/lib/python3.8/site-packages/aws_cdk/aws_glue/__init__.py)
Go to that file and see how the mapping/types are defined. As of 16 Aug 2020, you find
#jsii.data_type(
jsii_type="#aws-cdk/aws-glue.CfnCrawler.TargetsProperty",
jsii_struct_bases=[],
name_mapping={
"catalog_targets": "catalogTargets",
"dynamo_db_targets": "dynamoDbTargets",
"jdbc_targets": "jdbcTargets",
"s3_targets": "s3Targets",
}
)
I found that the lowerCamelCase always work, while the pythonic snake_case does not.

'cdk destroy' is not working as intended or I am not understanding it correctly?

Here is my demo stack,
export class HelloCdkStack extends cdk.Stack {
constructor(parent: cdk.App, id: string, props?: cdk.StackProps) {
super(parent, id, props);
new s3.Bucket(this, 'MyFirstBucket', {
versioned: true,
encryption: s3.BucketEncryption.KmsManaged,
});
}
}
'cdk deploy' creates a new bucket, but when I execute 'cdk destroy' it does not delete the bucket. Am I doing anything wrong?
By default, S3 buckets are configured to be 'orphaned' when a stack is deleted. Setting removalPolicy to Destroy will physically destroy the bucket on deletion.
You can set destroy to removalPolicy, it will remove the bucket if it's empty: https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-s3.Bucket.html#removalpolicy
If you want to destroy even non-empty bucket, you should also set autoDeleteObjects property to true: https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-s3.Bucket.html#autodeleteobjects
If you need to automatically destroy a bucket with files in it, check out this CDK construct: https://www.npmjs.com/package/#mobileposse/auto-delete-bucket
If you need to automatically destroy a bucket that is expected to be empty, use the standard bucket and set removalPolicy to DESTROY. https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-s3.Bucket.html#removalpolicy
In my case the problem was, cdk was trying to fetch different accounts Credentials. Add the --verbose or -v flag to see if any exception is thrown internally.
It's a shame that the exception was not getting logged to stdout or stderr (as it should for any tool)
In python, following the getting started, you can add removal_policy=cdk.RemovalPolicy.DESTROY parameter when instantiate the s3.Bucket object, so the bucket will be delete on cdk destroy.
from aws_cdk import core as cdk
from aws_cdk import aws_s3 as s3
class HelloCdkStack(cdk.Stack):
def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
bucket = s3.Bucket(self,
"MyFirstBucket",
versioned=True,
removal_policy=cdk.RemovalPolicy.DESTROY) # delete bucket on destroy

How to setup AWS CDK app execution in AWS CodeBuild?

I want to run AWS CDK synthesis from Git repository using AWS CodeBuild - i.e. if I update the CDK app code in the repo I want CloudFormation stacks to be updated automatically. What are the best practices for setting up build role permissions?
For a GitHub repository, your CodeBuild role doesn't need additional permissions but it should have access to an oauthToken to access GitHub.
For a CodeCommit repository, create or import a codecommit.Repository object and use a CodeCommitSource object for your source parameter, and the build role permissions will be set up automatically (in particular, the permissions that will be added will be to codecommit:GitPull from the indicated repository).
See here.
You might also be interested in CDK's app-delivery package. It doesn't just create a CodeBuild project though, it uses CodePipeline to fetch, build and deploy a CDK application, so it might be more than you are looking for.
AWS released a month ago a new class to the CDK suite called pipelines that includes several utilities to ease the job of setting up self modifying pipelines. In addition, there's codepipeline-actions that includes constructs to hook your pipeline to CodeCommit, GitHub, BitBucket, etc...
Here's a complete example (verbatim from the linked blog post), using github as a source, that deploys a lambda through CodePipeline:
Create a stage with your stack
import { CfnOutput, Construct, Stage, StageProps } from '#aws-cdk/core';
import { CdkpipelinesDemoStack } from './cdkpipelines-demo-stack';
/**
* Deployable unit of web service app
*/
export class CdkpipelinesDemoStage extends Stage {
public readonly urlOutput: CfnOutput;
constructor(scope: Construct, id: string, props?: StageProps) {
super(scope, id, props);
const service = new CdkpipelinesDemoStack(this, 'WebService');
// Expose CdkpipelinesDemoStack's output one level higher
this.urlOutput = service.urlOutput;
}
}
Create a stack with your pipeline
import * as codepipeline from '#aws-cdk/aws-codepipeline';
import * as codepipeline_actions from '#aws-cdk/aws-codepipeline-actions';
import { Construct, SecretValue, Stack, StackProps } from '#aws-cdk/core';
import { CdkPipeline, SimpleSynthAction } from "#aws-cdk/pipelines";
/**
* The stack that defines the application pipeline
*/
export class CdkpipelinesDemoPipelineStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const sourceArtifact = new codepipeline.Artifact();
const cloudAssemblyArtifact = new codepipeline.Artifact();
const pipeline = new CdkPipeline(this, 'Pipeline', {
// The pipeline name
pipelineName: 'MyServicePipeline',
cloudAssemblyArtifact,
// Where the source can be found
sourceAction: new codepipeline_actions.GitHubSourceAction({
actionName: 'GitHub',
output: sourceArtifact,
oauthToken: SecretValue.secretsManager('github-token'),
owner: 'OWNER',
repo: 'REPO',
}),
// How it will be built and synthesized
synthAction: SimpleSynthAction.standardNpmSynth({
sourceArtifact,
cloudAssemblyArtifact,
// We need a build step to compile the TypeScript Lambda
buildCommand: 'npm run build'
}),
});
// This is where we add the application stages
// ...
}
}

Resources