aws cdk for Elasticache Redis Cluster - aws-cdk

I have gone through the https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_elasticache.html.
How to create an Elasticache Redis template using AWS-CDK. It would be more helpful if you share the sample code.

sorry for late response but can be usefull for others.
CDK doesn't have an High Level Construct to create a Redis Cluster, but you can create it by using the low level construct api.
For Redis Cluster types you can take a look at this: https://aws.amazon.com/it/blogs/database/work-with-cluster-mode-on-amazon-elasticache-for-redis/
I've created a single Redis (no replication) Cluster using typescript like this:
const subnetGroup = new CfnSubnetGroup(
this,
"RedisClusterPrivateSubnetGroup",
{
cacheSubnetGroupName: "privata",
subnetIds: privateSubnets.subnetIds,
description: "subnet di sviluppo privata"
}
);
const redis = new CfnCacheCluster(this, `RedisCluster`, {
engine: "redis",
cacheNodeType: "cache.t2.small",
numCacheNodes: 1,
clusterName: "redis-sviluppo",
vpcSecurityGroupIds: [vpc.defaultSecurityGroup.securityGroupId],
cacheSubnetGroupName: subnetGroup.cacheSubnetGroupName
});
redis.addDependsOn(subnetGroup);
If you need a Redis (cluster enabled) Cluster you can you replication group
const redisSubnetGroup = new CfnSubnetGroup(
this,
"RedisClusterPrivateSubnetGroup",
{
cacheSubnetGroupName: "privata",
subnetIds: privateSubnets.subnetIds,
description: "subnet di produzione privata"
}
);
const redisReplication = new CfnReplicationGroup(
this,
`RedisReplicaGroup`,
{
engine: "redis",
cacheNodeType: "cache.m5.xlarge",
replicasPerNodeGroup: 1,
numNodeGroups: 3,
automaticFailoverEnabled: true,
autoMinorVersionUpgrade: true,
replicationGroupDescription: "cluster redis di produzione",
cacheSubnetGroupName: redisSubnetGroup.cacheSubnetGroupName
}
);
redisReplication.addDependsOn(redisSubnetGroup);
Hope this help.

I just struggled for hours creating a Redis cluster mode enabled with just one shard but two nodes. If you create a CfnReplicationGroup with num_cache_clusters=2, it will create a primary & replica node.
The trick is to create a CfnReplicationGroup with num_cache_clusters=2 and set the cache_parameter_group_name="default.redis6.x.cluster.on"
Then it will create a redis cache with cluster mode enable, one shard but two nodes

Related

How to add PTR record for EC2 Instance's Private IP in CDK?

I've two private hosted zones created for populating A records and PTR records corresponding to my EC2 instance's private ip. Yes, it's the private ip that I need. This subnet is routed to our corporate data center, so we need non-cryptic hostnames and consistent reverse lookup on them within the account.
I've got the forward lookup working well, however I'm confused how exactly it should be for the reverse lookup on the IP. Assume, my CIDR is 192.168.10.0/24 where the EC2 instances will get created.
const fwdZone = new aws_route53.PrivateHostedZone(
this, "myFwdZone", {
zoneName: "example.com",
vpc: myVpc,
});
const revZone = new aws_route53.PrivateHostedZone(
this, "myRevZone", {
zoneName: "10.168.192.in-addr.arpa",
vpc: myVpc,
}
);
I'm later creating the A record by referencing the EC2 instance's privateIp property. This worked well.
const myEc2 = new aws_ec2.Instance(this, 'myEC2', {...})
new aws_route53.RecordSet(this, "fwdRecord", {
zone: fwdZone,
recordName: "myec2.example.com",
recordType: aws_route53.RecordType.A,
target: aws_route53.RecordTarget.fromIpAddresses(
myEc2.instancePrivateIp
),
});
However, when I try to create the PTR record for the same, I've got some trouble. I needed to extract the fourth octet and specify as the recordName
new aws_route53.RecordSet(this, "revRecord", {
zone: revZone,
recordName: myEc2.instancePrivateIp.split('.')[3],
recordType: aws_route53.RecordType.PTR,
target: aws_route53.RecordTarget.fromValues("myec2.example.com"),
});
The CDK synthesized CloudFormation template looks odd as well, especially the token syntax.
revRecordDEADBEEF:
Type: AWS::Route53::RecordSet
Properties:
Name: ${Token[TOKEN.10.168.192.in-addr.arpa.
Type: PTR
HostedZoneId: A12345678B00CDEFGHIJ3
ResourceRecords:
- myec2.example.com
TTL: "1800"
Is this the right way to achieve this ? If I specified the recordName as just the privateIp, then the synthesized template ends up doing something else, which I see is incorrect too.
revRecordDEADBEEF:
Type: AWS::Route53::RecordSet
Properties:
Name:
Fn::Join:
- ""
- - Fn::GetAtt:
- myEC2123A01BC
- PrivateIp
- .10.168.192.in-addr.arpa.
Type: PTR
HostedZoneId: A12345678B00CDEFGHIJ3
ResourceRecords:
- myec2.example.com
TTL: "1800"
Answering the CDK part of your question: the original error was because you were performing string manipulation on an unresolved token. Your CDK code runs before any resources are provisioned. This has to be the case, since it generates the CloudFormation template that will be submitted to CloudFormation to provision the resources. So when the code runs, the instance does not exist, and its IP address is not knowable.
CDK still allows you to access unresolved properties, returning a Token instead. You can pass this token around and it will be resolved to the actual value during deployment.
To perform string manipulation on a token, you can use CloudFormation's bult-in functions, since they run during deployment, after the token has been resolved.
Here's what it would look like:
recordName: Fn.select(0, Fn.split('.', myEc2.instancePrivateIp))
As you found out yourself, you were also selecting the wrong octet of the IP address, so the actual solution would include replacing 0 with 3 in the call.
References:
https://docs.aws.amazon.com/cdk/v2/guide/tokens.html
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib-readme.html#intrinsic-functions-and-condition-expressions

Create CfnDBCluster in non-default VPC?

I'm trying to create a serverless aurora database with the AWS CDK (1.19.0). However, it will always be created in the default VPC of the region. If I specify a vpc_security_group_id cloudformation fails because the provided security group is in the vpc created in the same stack as the aurora db.
"The DB instance and EC2 security group are in different VPCs."
Here is my code sample:
from aws_cdk import (
core,
aws_rds as rds,
aws_ec2 as ec2
)
class CdkAuroraStack(core.Stack):
def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
super().__init__(scope, id, **kwargs)
# The code that defines your stack goes here
vpc = ec2.Vpc(self, "VPC")
sg = ec2.SecurityGroup(self, "SecurityGroup",
vpc = vpc,
allow_all_outbound = True
)
cluster = rds.CfnDBCluster(self, "AuroraDB",
engine="aurora",
engine_mode="serverless",
master_username="admin",
master_user_password="password",
database_name="databasename",
vpc_security_group_ids=[
sg.security_group_id
]
)
Do I miss something and it is possible to create the CfnDbCluster in a specific vpc or is this just not possible atm?
Thanks for any help and advice. Have a nice day!
You should create a DB subnet group and include only the subnets you want Amazon RDS to launch instances into. Amazon RDS creates a DB subnet group in default VPC if none is specified.
You can use db_subnet_group_name property to specify your subnets, however it is better to use high-level constructs. In this case, there is one called DatabaseCluster.
cluster = DatabaseCluster(
scope=self,
id="AuroraDB",
engine=DatabaseClusterEngine.AURORA,
master_user=rds.Login(
username="admin",
password="Do not put passwords in your CDK code directly"
),
instance_props={
"instance_type": ec2.InstanceType.of(ec2.InstanceClass.BURSTABLE2, ec2.InstanceSize.SMALL),
"vpc_subnets": {
"subnet_type": ec2.SubnetType.PRIVATE
},
"vpc": vpc,
"security_group": sg
}
)
Do not specify password attribute for your database, CDK assigns a Secrets Manager generated password by default.
Just to note that this construct is still experimental, that means there might be a breaking change in the future.

What are the allowed values for RDS ClusterParameterGroup family?

I am trying to configure an Aurora PostgreSQL 2.3 cluster (compatible with PostgreSQL 10.7) but don't know what the rds.ClusterParameterGroup.family should be set to or if that even matters here.
The example I found used "aurora5.6" as shown below but I don't know how that corresponds to the PostgreSQL version.
const dbparams = new rds.ClusterParameterGroup(this, 'DbClusterParams', {
family: 'aurora5.6',
description: 'my parameter group',
parameters: {
character_set_database: 'utf8mb4'
}
});
// create Aurora cluster
const dbcluster = new rds.DatabaseCluster(this, 'DbCluster', {
defaultDatabaseName: 'MySampleDb',
engine: rds.DatabaseClusterEngine.AURORA_POSTGRESQL,
masterUser: {
username: 'myadmin',
},
instanceProps: {
instanceType: ec2.InstanceType.of(ec2.InstanceClass.BURSTABLE2, ec2.InstanceSize.SMALL),
vpc
},
parameterGroup: dbparams,
kmsKey,
});
The API documentation doesn't provide any details. https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-rds.ClusterParameterGroup.html
You can find out more about the ClusterParameterGroup Family when taking a look at the CloudFormation Documentation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbclusterparametergroup.html
The approach described there:
aws rds describe-db-engine-versions --query "DBEngineVersions[].DBParameterGroupFamily"
returns a complete list of possible values (with duplicates):
...
...
"aurora-mysql5.7",
"aurora-mysql5.7",
"aurora-mysql5.7",
"aurora-mysql5.7",
"docdb3.6",
"neptune1",
"neptune1",
"neptune1",
"aurora-postgresql9.6",
"aurora-postgresql9.6",
"aurora-postgresql9.6",
"aurora-postgresql9.6",
...
...

What is the best way to use the AWS CDK to get the CIDR of the VPC?

I've started using the AWS CDK to stand up a new VPC, but I am struggling when trying to query other existing VPCs and their CIDR ranges - this is to ensure that my new VPC does not overlap with existing CIDR ranges. The return string is not something I can understand. Could you provide an example on how to query for a list of CIDR ranges in subnets?
Thanks.
If you are trying to reference an existing VPC into your CDK stack, you should use the VpcNetwork.import static method which doesn't require you to specify the CIDR blocks of the VPC.
You will need other information specified in VpcNetworkRefProps, which shouldn't be too hard to obtain from the AWS Console or the AWS CLI:
Something like:
const externalVpc = VpcNetwork.import(this, 'ExternalVpc', {
vpcId: 'vpc-bd5656d4',
availabilityZones: [ 'us-east1a', 'us-east-1b' ],
publicSubnetIds: [ 'subnet-1111aaaa', 'subnet-2222bbbb' ],
privateSubnetIds: [ 'subnet-8368fbce', 'subnet-8368abcc' ],
});
We are looking at making this easier (see #506)

How can a docker service know about all other containers of the same service?

I'm working on a file sync Docker microservice. Basically I will have a file-sync service that is global to the swarm (one on each node). Each container in the service needs to peer with all the other containers on different nodes. Files will be distributed across the nodes, not a complete duplicate copy. Some files will reside on only certain nodes. I want to be able to selectively copy a subset of the files from one node to another.
How can I get a list of the endpoints of all the other containers so the microservice can peer with the them? This needs to happen programmatically.
On a related note, I'm wondering if a file-sync microservice is the best route for the solution I'm working on.
Basically I have some videos a user has uploaded. I want to be able to encode them into different formats. I was planning on having the video encoding node have the file-sync service pull the files, encode the videos, and then use the file-sync to push the encoded files back to the same server. I know I can use some kind of object store but that isn't available to me with bare metal dedicated servers and I'd rather not deal with OpenStack if I don't need to.
Thanks to #johnharris85 for the above suggestion. For anyone else that is interested I created a snippet that can be used in node.
https://gist.github.com/brennancheung/62d2abe16569e600d2be5e9495c85331
const dns = require('dns')
function lookup (serviceName) {
const tasks = `tasks.${serviceName}`
return new Promise((resolve, reject) => {
dns.lookup(tasks, { all: true }, (err, addresses, family) => {
if (err) {
return reject(err)
}
const filtered = addresses.filter(address => address.family === 4)
const ips = filtered.map(x => x.address)
resolve(ips)
})
})
}
async function main () {
const result = await lookup('hello')
console.log(result)
}
main()

Resources