Context:
Projen is an awesome tool to generate and manage (JSII-built) AWS CDK projects.
Background:
Previously I have managed CDK dependencies with RenovateBot's group:aws-cdkMonorepo preset. This will result in RenovateBot creating a single Github Pull Request for AWS CDK depedency updates.
Question:
With Projen, one controls the CDK version in .projenrc.js:
const { AwsCdkConstructLibrary } = require('projen');
const project = new AwsCdkConstructLibrary({
authorName: "Example",
authorAddress: "contact#example.com",
cdkVersion: "1.64.0",
name: "#example/project",
repository: "https://github.com/example/project.git",
});
project.synth();
So how can one manage that cdkVersion value with tooling such as DependaBot or RenovateBot?
Since keeping one's CDK constructs up-to-date with current CDK version is critial and with multiple CDK constructs doing it by hand will be painful.
The central version management depends on your requirement. If you are using a centralized construct library it is a must-have.
For managing the dependencies in centrally in a single configuration, you need to add the following snippet in the .projenrc.js
cdkDependecies:[
'#aws-cdk/core'
]
Now, whenever you run projen the cdk app would be managed centrally and use the latest version.
Related
I am learning AWS Cloud Development Kit (CDK).
As part of this learning, I am trying to understand how I am supposed to correctly handle production and development environment.
I know AWS CDK provides the environment parameter to allow deploying stacks to specific account.
But then, how to have specific options for development versus production stacks ? It does not seem to be provided by default by AWS CDK or am I missing/misunderstanding something ?
A very simple example could be that I want a S3 bucket called my-s3-bucket-dev for my development account and one named my-s3-bucket-prod for my production account. But then how to have e.g. a variable stage correctly handled in AWS CDK ?
I know I can add parameters in the cdk.json file but again, I don't know how to correctly use this file to depend upon the deployed stack i.e. production vs development.
Thanks for the support
Welcome to AWS CDK.
Enjoy the ride. ;)
Actually, there is no semantic (in your case the stage) in an account itself.
This has nothing to do with CDK or Cloud Formation.
You need to take care of this.
You're right, that you could use the CDK context in the cdk.json.
There's no schema enforcement in the context, except for some internally used variables by CDK.
You could define your dev and prod objects within.
There are other ways of defining the context.
Here is an example, what it could look like:
{
"app": "node app",
// usually there's some internal definition for your CDK project
"context": {
"dev": {
"accountId" : "prod_account",
"accountRegion" : "us-east-1",
"name": "dev",
"resourceConfig":
{
// here you could differentiate the config per AWS resource-type
// e.g. dev has lower hardware specs
}
},
"prod": {
"accountId" : "prod_account",
"accountRegion" : "us-east-1",
"name": "prod",
"resourceConfig":
{
// here you could differentiate the config per AWS resource-type
// prod has higher hardware specs or more cluster nodes
}
}
}
}
With this being defined, you need to run your CDK application with the -c flag to specify which configuration object (dev or prod), you want to have.
For instance, you could run it with cdk synth -c stage=prod.
This sets the stage variable in your context and makes it available.
When it was successful, you can re-access the context again and fetch the appropriate config object.
const app = new cdk.App();
const stage = app.node.tryGetContext('stage');
// the following step is only needed, if you have a different config per account
const stageConfig = app.node.tryGetContext(stage );
// ... do some validation and pass the config to the stacks as constructor argument
As I said, the context is one way of doing this.
However, there are drawbacks to it.
It's JSON and no code.
What I prefer is to have TypeScript types per resource configuration (e.g. S3) and wire them all together as a plain object.
The object maps the account/region information and the corresponding resource configurations.
Is it somehow possible to get a list of the stacks that another stack depends on using the AWS CDK CLI? For example, given a list of stacks that looks something like:
const app = new App();
const alphaStack = new Stack(app);
const betaStack = new Stack(app);
betaStack.addDependency(alphaStack);
const gammaStack = new Stack(app);
gammaStack.addDependency(gammaStack);
const deltaStack = new Stack(app);
deltaStack.addDependency(betaStack);
deltaStack.addDependency(gammaStack);
I'd like to run a command that could give me output similar to the following:
$ cdk list-deps alpha-stack # no result
$ cdk list-deps beta-stack
alpha-stack
$ cdk list-deps gamma-stack
alpha-stack
$ cdk list-deps delta-stack
beta-stack
gamma-stack
Specifically, I'd like to be able to run this before I deploy my stacks.
Run cdk synth STACK_NAME
and you should find the dependencies from cdk.out/manifest.json
❯ jq '.artifacts.STACK_NAME.dependencies' cdk.out/manifest.json
[
"STACK_NAME.assets"
...
]
In case this helps anyone in the future; the following seems to solve the problem that I wanted to solve:
// the stack that we're interested in finding deps for
const stackName = "...";
// assuming app is as defined in the question
const { stacks } = app.synth();
stacks
.find(({ stackName }) => stackName === searchForStackName)
?.dependencies.forEach((dep) => console.log(dep.id));
Caveats:
It's obviously not built-in to the CLI, and requires exposing app from your CDK definitions.
This does require calling synth on the app. Initially my understanding was that this creates some artifacts within AWS, which I wanted to avoid; but it seems that's not actually the case.
I'm not sure how reliable/stable dep.id is from this snippet/answer. Although it has so far been robust enough for my purposes, dependencies returns a list of CloudArtifact which I'm not certain if it will always represent a Stack.
I know this isn't quite as robust as what you're looking for but here is a useful command to find the stacks that depend on a particular export:
aws cloudformation list-imports --export-name EXPORT_NAME
You can also run
cdk destroy stackName
It will ask you for confirmation of the dependencies necesary to destroy your stack.
This is a dangerous operation and should be done with care because it can destroy your stacks.
I have a situation where I want to import my graph data to database.I am having janusgraph(latest version) running with cassandra(version 3) and elasticsearch(version 6.6.0) using Docker.I have been suggested to use gryo format.So I have tried this command
graph.io(IoCore.gryo()).reader().create().readGraph(ToInputStream.from("my_graph.kryo"), graph);
but ended up with an error
No such property: ToInputStream for class: Script4
The documentation I am following is here.Please take a look and put me in a right procedure. Thanks in advance!
ToInputStream is not a function of Gremlin or JanusGraph. I believe that it is only a function of IBM Compose so unless you are running JanusGraph on that specific platform, this command will not work.
Versions of JanusGraph that utilize TinkerPop 3.4.x will support the io() step and this is the preferred manner in which to load gryo (as well as graphson and graphml) files.
Graph graph = ... // setup JanusGraph instance
GraphTraversalSource g = traversal().withGraph(graph); // might use withRemote() here instead depending on how you are connecting I suppose
g.io("graph.kryo").read().iterate()
Note that if you are connecting remotely - it seems you are sending scripts to the Docker instance given your error - then be sure that that "graph.kryo" file path is accessible to Docker. That's what's nice about ToInputStream from Compose as it allows you to access remote sources.
I'm using cakebuid as my build tool for TFS 2017 Update 2 and trying to implement the traditional Git Flow. In this flow, there are a few automatic merges that happen every time changes get into master, those changes need to be propagated to the develop branch.
Using cake I can run a PowerShell script or use LibGit2Sharp to accomplish the automatic merge for the best case scenarios. But, what about when the merge has conflicts? Do I need to fail the whole build because the merge process fail?
We have certainly something to deal with merges in TFS, this is no other than the Pull Request.
Question
Is there any tool or add-in for cake that allows me to create
Pull Request during the execution of a build step?
I don't think there is any add-in available for you to create a pull request but since you can run PowerShell, you can easily use the TFS rest api to create pull request
https://www.visualstudio.com/en-us/docs/integrate/api/git/pull-requests/pull-requests
It was recently announced that there is a VSTS CLI:
https://blogs.msdn.microsoft.com/devops/2017/11/15/introducing-the-new-cli-for-vsts/
Which includes the ability to create a pull request:
https://learn.microsoft.com/en-gb/cli/vsts/get-started?view=vsts-cli-latest#create-a-pull-request
I don't think it would be particularly hard to create a Cake Addin which wraps this tool, and exposes the functionality through a set of addin's.
In the mean time, you could shell out to this tool using the Process aliases that currently exist in Cake.
Finally, I spend sometimes creating the package:
Nuget: https://www.nuget.org/packages/Cake.Tfs.AutoMerge
GitHub: https://github.com/mabreuortega/Cake.Tfs
The way you can use it now is similar to this:
Task("Merge")
.Does(c => {
CreateAutoMergePullRequest(
new AutoMergeSettings
{
// Required
CollectionUri = "https://{instance}/{collection-name}",
ProjectName = "project-name",
RepositoryName = "repository-name",
SourceBranch = "refs/heads/release/1.0.0",
TargetBranch = "refs/heads/develop",
Title = "[Auto Merge from Release]",
// Optional
Description = "Brief description of the changes about to get merge",
// Control
DeleteSourceBranch = false,
SquashMerge = false,
OverridePolicies = true,
AutoComplete = true,
AutoApprove = true
});
});
Any suggestions, please use the GitHub issue tracker.
Hope this help!
I am doing automation in my company. We are a C# workshop.
Currently I am working on automated build. NANT is flow control tool. While NANT is not actively developed (last binary released on June 2012 and github repo is not active), MSBuild is better. Therefore, I prefer MSBuild but retiring NANT is still questionable - what is the cost?
I have come up with some pros and cons, but I know collective intelligence is better. Thanks for your help!
Update:
I have read the question, but the second answer rises a concern for me. On build machine there are multiple .NET frameworks, will it be troublesome?
MSBuild
Pros:
Commercial support
Community is growing
Intergrated with VS and TFS
Keep pace with .Net
Cons:
Rewrite current script
Not familiar by people
NANT
Pros:
Already in use
Familiar by people
Cons:
Not updated for a long time (since 2012)
Community is not active
Lack of new .Net support
We wrote FlubuCore (rewrite of Flubu). It's an open source C# library for building projects and executing deployment scripts using C# code.
Main advantages of flubu that I see are:
.Net Core support.
Easy to learn and to use because you write build script entirely in C#.
Fluent interface and intelisense.
Quite a lot of built in tasks (compile, running tests, managing iis, creating deploy package, publishing nuget packages, executing powershell scripts...)
Write your own custom c# code in script and execute it..
Run any external program or command in script with RunProgramTask.
Reference any .net library or c# source code file in buildscript. Now also available option to reference nuget package in build script.
Write tests, debug your build script..
Use flubu tasks in any other .net application.
Web api is available for flubu. Useful for automated deployments remotely.
Write your own flubu tasks and extend flubu fluent interface with them.
You can find flubu on nuget:
Search for FlubuCore.Runner if u need it for .net project
Search for dotnet-flubu if u need it for.net core project
Example of how flubu is used in .net:
protected override void ConfigureBuildProperties(IBuildPropertiesContext context) {
context.Properties.Set(BuildProps.NUnitConsolePath,
# "packages\NUnit.ConsoleRunner.3.6.0\tools\nunit3-console.exe");
context.Properties.Set(BuildProps.ProductId, "FlubuExample");
context.Properties.Set(BuildProps.ProductName, "FlubuExample");
context.Properties.Set(BuildProps.SolutionFileName, "FlubuExample.sln");
context.Properties.Set(BuildProps.BuildConfiguration, "Release");
}
protected override void ConfigureTargets(ITaskContext session) {
var loadSolution = session.CreateTarget("load.solution")
.SetAsHidden()
.AddTask(x => x.LoadSolutionTask());
var updateVersion = session.CreateTarget("update.version")
.DependsOn(loadSolution)
.SetAsHidden()
.Do(TargetFetchBuildVersion);
session.CreateTarget("generate.commonassinfo")
.SetDescription("Generates common assembly info")
.DependsOn(updateVersion)
.TaskExtensions().GenerateCommonAssemblyInfo()
var compile = session.CreateTarget("compile")
.SetDescription("Compiles the solution.")
.AddTask(x => x.CompileSolutionTask())
.DependsOn("generate.commonassinfo");
var unitTest = session.CreateTarget("unit.tests")
.SetDescription("Runs unit tests")
.DependsOn(loadSolution)
.AddTask(x => x.NUnitTaskForNunitV3("FlubuExample.Tests"));
session.CreateTarget("abc").AddTask(x => x.RunProgramTask(# "packages\LibZ.Tool\1.2.0\tools\libz.exe"));
session.CreateTarget("Rebuild")
.SetDescription("Rebuilds the solution.")
.SetAsDefault()
.DependsOn(compile, unitTest);
}
//// Some custom code
public static void TargetFetchBuildVersion(ITaskContext context) {
var version = context.Tasks().FetchBuildVersionFromFileTask().Execute(context);
int svnRevisionNumber = 0; //in real scenario you would fetch revision number from subversion.
int buildNumber = 0; // in real scenario you would fetch build version from build server.
version = new Version(version.Major, version.Minor, buildNumber, svnRevisionNumber);
context.Properties.Set(BuildProps.BuildVersion, version);
}
Example of how flubu is used in .net core
public class MyBuildScript : DefaultBuildScript
{
protected override void ConfigureBuildProperties(IBuildPropertiesContext context)
{
context.Properties.Set(BuildProps.CompanyName, "Flubu");
context.Properties.Set(BuildProps.CompanyCopyright, "Copyright (C) 2010-2016 Flubu");
context.Properties.Set(BuildProps.ProductId, "FlubuExample");
context.Properties.Set(BuildProps.ProductName, "FlubuExample");
context.Properties.Set(BuildProps.SolutionFileName, "FlubuExample.sln");
context.Properties.Set(BuildProps.BuildConfiguration, "Release");
}
protected override void ConfigureTargets(ITaskContext context)
{
var buildVersion = context.CreateTarget("buildVersion")
.SetAsHidden()
.SetDescription("Fetches flubu version from FlubuExample.ProjectVersion.txt file.")
.AddTask(x => x.FetchBuildVersionFromFileTask());
var compile = context
.CreateTarget("compile")
.SetDescription("Compiles the VS solution and sets version to FlubuExample.csproj")
.AddCoreTask(x => x.UpdateNetCoreVersionTask("FlubuExample/FlubuExample.csproj"))
.AddCoreTask(x => x.Restore())
.AddCoreTask(x => x.Build())
.DependsOn(buildVersion);
var package = context
.CreateTarget("Package")
.CoreTaskExtensions()
.DotnetPublish("FlubuExample")
.CreateZipPackageFromProjects("FlubuExample", "netstandard2.0", "FlubuExample")
.BackToTarget();
//// Can be used instead of CreateZipPackageFromProject. See MVC_NET4.61 project for full example of PackageTask
//// context.CreateTarget("Package2").AddTask(x =>
x.PackageTask("FlubuExample"));
var test = context.CreateTarget("test")
.AddCoreTaskAsync(x => x.Test().Project("FlubuExample.Tests"))
.AddCoreTaskAsync(x => x.Test().Project("FlubuExample.Tests2"));
context.CreateTarget("Rebuild")
.SetAsDefault()
.DependsOn(compile, test, package);
}
}
Detailed presentation and documentation can be found here:
https://github.com/flubu-core/flubu.core
You can find full examples here:
https://github.com/flubu-core/examples
Thanks for all answers. We have decided to use Cake since we are a C# workshop.
There is a property nant.settings.currentframework which is used to set target framework in case you have multiple .net framework
<property name="nant.settings.currentframework" value="net-2.0" />
As per .92 build:
nant.settings.currentframework The current target framework, eg. 'net-1.0'.
nant.settings.currentframework.description Deprecated. Description of the current target framework.
nant.settings.currentframework.frameworkdirectory Deprecated. The framework directory of the current target framework.
nant.settings.currentframework.sdkdirectory Deprecated. The framework SDK directory of the current target framework.
nant.settings.currentframework.frameworkassemblydirectory Deprecated. The framework assembly directory of the current target framework.
nant.settings.currentframework.runtimeengine Deprecated. The runtime engine of the current target framework if used eg. mono.exe.