How to write the JQ expression for dynamic partitioning from Kinesis console in CDK Kinesis configuration parameters? - aws-cdk

Question

I have a JQ expression for dynamic partitioning in the Kinesis firehose as follows:
Key name JQ expression
time .time| strptime("%Y-%m-%dT%H:%M:%SZ") | mktime | strftime("%Y%m%d")
This was written for the below time format:
"time": "2020-01-29T17:26:50Z"
The above JQ expression gives me time format output as 20200129 which I use as my partition date while storing records from Kinesis.
How should I write it in the config parameters while writing Kinesis firehose code in AWS CDK?
s3_destination_conf = ds.ExtendedS3DestinationConfigurationProperty(
bucket_arn=output_bucket.bucket_arn,
dynamic_partitioning_configuration=ds.DynamicPartitioningConfigurationProperty(
enabled=True
),
processing_configuration=ds.ProcessingConfigurationProperty(
enabled=True,
processors=[
ds.ProcessorProperty(
type="MetadataExtraction",
parameters=[
ds.ProcessorParameterProperty(
parameter_name="JsonParsingEngine",
parameter_value="JQ-1.6",
),
ds.ProcessorParameterProperty(
parameter_name="MetadataExtractionQuery",
==== Issue line => parameter_value="{time: .time| strptime('%Y-%m-%dT%H:%M:%SZ') | mktime | strftime('%Y%m%d')}",
),
],
)
],
),
role_arn=role.role_arn,
buffering_hints=ds.BufferingHintsProperty(
interval_in_seconds=params.KinesisFirehose.BUFFER_INTERVAL_SEC,
size_in_m_bs=params.KinesisFirehose.BUFFER_SIZE_MB,
),
I tried playing a bit with the double quotes and single quotes in the JQ string(parameter_value="{time: .time| strptime('%Y-%m-%dT%H:%M:%SZ') | mktime | strftime('%Y%m%d')}") but it didn't help.

Answer 1

Try the following:
parameter_value="{time: .time| strptime(\'%Y-%m-%dT%H:%M:%SZ\') | mktime | strftime(\'%Y%m%d\')}"
You can also have a look on the string escape characters in python here.