I am building with Terraform a Cloud Scheduler job that will hit a service deployed in Cloud Run. Because the service and the scheduler are deployed in different pipelines, it is possible that the service does not exist yet when running the scheduler pipeline. This is why I am using data "google_cloud_run_service" to retrieve the service and then control whether it exists in the scheduler block. It is here that I don't find the right syntax to use in count.
data "google_cloud_run_service" "run-service" {
name = "serv-${var.project_env}"
location = var.region
}
resource "google_cloud_scheduler_job" "job" {
count = length(data.google_cloud_run_service.run-service.status)
name = "snap-job-${var.project_env}"
description = "Call write API on my service"
schedule = "* * * * *"
time_zone = "Etc/UTC"
http_target {
http_method = "GET"
uri = "${data.google_cloud_run_service.run-service.status[0].url}/write"
oidc_token {
service_account_email = google_service_account.sa_scheduler.email
}
}
depends_on = [google_app_engine_application.app]
}
The above length(data.google_cloud_run_service.run-service.status) control will not make any effect, and Terraform tries to create the scheduler even though there is no service defined.
I have also tried other variations with similar result such as length(data.google_cloud_run_service.run-service.status[0]) > 0 ? 1 : 0.
Other options that I tried will not work either with different errors:
data.google_cloud_run_service.run-service ? 1 : 0: data.google_cloud_run_service.run-service is object with 9 attributes;The condition expression must be of type bool
data.google_cloud_run_service.run-service.status[0].url ? 1 : 0: data.google_cloud_run_service.run-service is object with 9 attributes;The condition expression must be of type bool
Related
We have a new terraform script that is pushing a docker image to an AWS Lambda. The script works well and correctly connects the fresh image to the Lambda. I can confirm this by checking the Image URL as shown in the AWS console for the Lambda and it is the newly pushed+connected image. However when testing the lambda it is clearly running the prior code. It seems like the Lambda has been updated but the running in-memory instances didnt get the message.
Question: is there a way to force the in-memory Lambdas to be cycled to the new image?
Here is our TF code for the Lambda:
resource "aws_lambda_function" "my_lambda" {
function_name = "MyLambda_${var.environment}"
role = data.aws_iam_role.iam_for_lambda.arn
image_uri = "${data.aws_ecr_repository.my_image.repository_url}:latest"
memory_size = 512
timeout = 300
architectures = ["x86_64"]
package_type = "Image"
environment {variables = {stage = var.environment, commit_hash=var.commit_hash}}
}
After more searching I found some discussions (here) that mention the source_code_hash option in terraform for the Lambda creation block (docs here). Its mostly used with a SHA hash of the zip file used for pushing code from an S3 bucket, but in our case we are using a container/image so there is not really a file to get a hash from. However, it turns out that it is just a string that Lambda checks for changes. So we added the following:
resource "aws_lambda_function" "my_lambda" {
function_name = "MyLambda_${var.environment}"
role = data.aws_iam_role.iam_for_lambda.arn
image_uri = "${data.aws_ecr_repository.my_image.repository_url}:latest"
memory_size = 512
timeout = 300
architectures = ["x86_64"]
package_type = "Image"
environment {variables = {stage = var.environment, commit_hash=var.commit_hash}}
source_code_hash = var.commit_hash << New line
}
And we use a bitbucket pipeline to inject the git hash into the terraform apply operation. This fix allowed the Lambda to correctly update the running version.
Alternatively, if you don't want to depend on bitbucket for this, you can add a data source for the ECR image:
data "aws_ecr_image" "repo_image" {
repository_name = "repo-name"
image_tag = "tag"
}
And then use its id as a source code hash like this:
source_code_hash = trimprefix(data.aws_ecr_image.repo_image.id, "sha256:")
Let's say I've a module which creates some resources that represent a default server. Now I want to inherit this default server to customize it into different directions.
At the Manager node which inherits the DockerNode I want to run docker swarm init and get the join token. On all Worker nodes I want to join with the token.
So in my main.tf where I use the DockerNode I have defined the nodes like this:
module "manager" {
source = "./modules/swarm-node"
node_network = {
network_id = hcloud_network.swarm-network.id
ip = "10.0.1.10"
}
node_name_prefix = "swarm-manager"
server = var.server
docker_compose_version = var.docker_compose_version
volume_size = var.volume_size
volume_filesystem = var.volume_filesystem
ssh_keys = [hcloud_ssh_key.ssh-key-a.name, hcloud_ssh_key.ssh-key-b.name]
depends_on = [
hcloud_network_subnet.swarm-network-nodes
]
}
module "worker" {
count = 2
source = "./modules/swarm-node"
node_index = count.index
node_network = {
network_id = hcloud_network.swarm-network.id
ip = "10.0.1.10${count.index}"
}
node_name_prefix = "swarm-worker"
server = var.server
docker_compose_version = var.docker_compose_version
volume_size = var.volume_size
volume_filesystem = var.volume_filesystem
ssh_keys = [hcloud_ssh_key.ssh-key-a.name, hcloud_ssh_key.ssh-key-b.name]
depends_on = [
hcloud_network_subnet.swarm-network-nodes,
module.manager
]
}
How to run docker swarm init and return the join token on the server resource inside of module.manager?
How to join the swarm with each worker?
I've researched this for quite a while:
Some solutions expose the Docker Daemon over TCP and access it from the worker to get the token. I don't like to expose the Docker Daemon unnecessarily.
Some solutions copy the base module (in my case DockerNode) just to modify one or two lines. I like to to follow DRY.
Some solutions have an additional shell script, which read the .tfstate and SSH into each machine to do further customization. I would like to use Terraform for this with all it's benefits.
When instantiating the vpc object within a stack using the CDK. There is a parameter max_azs which supposedly defaults to 3. However, when I create a vpc no matter what I set that number to, I only ever get 2 AZs.
from aws_cdk import (
core,
aws_ec2 as ec2
)
app = core.App()
subnets = []
subnets.append(ec2.SubnetConfiguration(name = "public", subnet_type = ec2.SubnetType.PUBLIC, cidr_mask = 20))
subnets.append(ec2.SubnetConfiguration(name = "private", subnet_type = ec2.SubnetType.PRIVATE, cidr_mask = 20))
subnets.append(ec2.SubnetConfiguration(name = "isolated", subnet_type = ec2.SubnetType.ISOLATED, cidr_mask = 20))
vpc = ec2.Vpc(app, "MyVpc", subnet_configuration = subnets, max_azs = 3)
print(vpc.availability_zones)
app.synth()
I would expect to see 3 azs used here, but actually only ever get 2. Even if i set the value to 99, which should mean all azs.
Ah yes I came across the same issue myself. What solved it for me was specifying the region and account when creating the stack.
The following examples are for typescript but I'm sure you can write the corresponding python.
new MyStack(app, 'MyStack', {
env: {
region: 'us-east-1',
account: '1234567890',
}
});
In the case of typescript you need to rebuild and synth before you deploy.
$ npm run build
$ cdk synth
I am trying to capture the record which has been created on running of gatling simulation test.
My scenario is that
read json data from csv and publish to kafka which is consumed by microservice and store data into couchbase,
since kafka publishing message in asyn mode so there is no way we can get to know that how many record got created in data base.
is there any way in gatling to get data from couchabse and assert so that if record in couchbase not equal to the request then simulation should fail ?
val scn = scenario("Order test sceanrio")
.feed(csv("TestOrder.csv").circular)
.exec(ProducerBuilder[Array[Byte], Array[Byte]]())
setUp(scn.inject(atOnceUsers(count))).protocols(kafkaProtocol)
//.assertion(getCouchbaseOrderCount == count) // not supported by
gatling
I Have resolved this issue by using tearDown in the simulation.
Below is the tearDown code for gatling,
after {
println("**************** Asserting scenario *****************")
assert(orderCount() == count)
}
def orderCount(): Int = {
val cluster = openConnection
val bucket: Bucket = cluster.openBucket("Order", "password");
println(bucket)
val query = "SELECT meta().id FROM `Order`"
Thread.sleep(1000)
val orderCount: Int = bucket.query(N1qlQuery.simple(query)).allRows().size();
println(" Order Count :: " + orderCount)
orderCount
}
I have been trying to find a lightweight method in a Groovy scriptler script to list all currently running jobs of any type. The only method I have found to be reliable is this:
start = System.currentTimeMillis()
def jobsFound = []
def buildingJobs = Jenkins.instance.getAllItems(Job.class).findAll {
it.isBuilding()
}
buildingJobs.each { job->
allRuns = job._getRuns()
allRuns.each { item->
if (!item.isBuilding()) { return } // This job is not building
jobsFound.push(item.getUrl())
}
}
timespent = (System.currentTimeMillis() - start ) / 1000
println "Time: ${timespent} seconds"
println "{jobsFound.size} jobs"
// RESULTS:
// Time: 2.015 seconds. 15 jobs
The problem is that the above enumerates ALL currently running jobs - we have thousands! - then enumerate all builds of each of those running jobs (some jobs have as many as 300 builds). The above can take as long as FIVE minutes to complete depending on how many jobs are currently building.
A much more efficient method is to enumerate the active executors
but this method MISSES the pipeline (aka Workflow) jobs running on the master:
start = System.currentTimeMillis()
def busyExecutors = Jenkins.instance.computers.collect {
c -> c.executors.findAll { it.isBusy() }
}.flatten()
def jobsFound = []
busyExecutors.each { e ->
job = e.getCurrentExecutable()
jobsFound.push(job.getUrl())
}
timespent = (System.currentTimeMillis() - start ) / 1000
println "Time: ${timespent} seconds. ${jobsFound.size} jobs"
// RESULTS:
// Time: 0.005 seconds. 12 jobs
Sure enough the discrepancy between the two counts are pipeline jobs running on the master.
I guess my question boils down to:
Is there a way to efficiently enumerate all jobs running on the master?
Clearly Jenkins does not include the master in computers, and although there is a MasterComputer class, its not clear how to get it or the OffByOneExecutors on which flyweight (pipeline) jobs run.
Not directly with Jenkins types/objects but via Jenkins' Remote access API derived from From Jenkins, how do I get a list of the currently running jobs in JSON?:
http://localhost:8080/api/xml?&tree=jobs[builds[*]]&xpath=/hudson/job/build[building="true"]&wrapper=builds
results in:
<builds>
<build _class="hudson.model.FreeStyleBuild">
<action _class="hudson.model.CauseAction"/>
<action/>
<action/>
<building>true</building>
<displayName>#10</displayName>
<duration>0</duration>
<estimatedDuration>3617</estimatedDuration>
<executor/>
<fullDisplayName>Freestyle-Project #10</fullDisplayName>
<id>10</id>
<keepLog>false</keepLog>
<number>10</number>
<queueId>2</queueId>
<timestamp>1499611190781</timestamp>
<url>http://localhost:8080/job/Freestyle-Project/10/</url>
<builtOn/>
<changeSet _class="hudson.scm.EmptyChangeLogSet"/>
</build>
</builds>
Unfortunately <builtOn/>, which, I guess, is supposed to refer to the node, isn't supplied in my Jenkins v2.60.1 (yet?).
With:
http://localhost:8080/api/json?pretty=true
you get:
...
"nodeDescription" : "the master Jenkins node",
...
"jobs" : [
{
"_class" : "hudson.model.FreeStyleProject",
"name" : "Freestyle-Project",
"url" : "http://xmg:8080/job/Freestyle-Project/",
"color" : "aborted_anime"
}
]
...
which you can filter then with:
nodeDescription.equals('the master Jenkins node') && color.endsWith('_anime').