given an avro record schema for RecordA as
"type":"record"...
"fields": [ { "name": "typeB", "type":"RecordB"} ...
and RecordB avro record as
"type":"record"
"fields": [ {"name": "mysqoopfield", "type":"int" } ...
is there a mechanism in sqoop-export by which I can export the value of mysqoopfield to RDBMS with a table with just column mysqoofield ?
have tried
sqoop-export --connect ... --driver ... --export /path/to/avro --columns typeB.mysqoopfield | typeB[mysqoopfield]
which don't work. the . is translated to a '_' by the AvroExportMapper. Just ondering if this is supported before digging further ?
Doesn't also seem like a custom mapper implemenation can be provided to sqoop ?
Related
Most of my code is represented by ECMA Script libraries that have import and export rather than require statements.
Is there a suggested way to represent such libraries using rules_dotnet?
I am looking for something like this:
load("#build_bazel_rules_nodejs//:index.bzl", "ecmascript_library")
ecmascript_library(
name = "foo",
srcs = [
"foo.js"
],
)
ecmascript_library(
name = "bar",
srcs = [
"bar.js"
],
deps = [
":foo",
],
)
Where Bazel will ensure the search paths for rules consuming bar can find foo.
I would then like to use my library definitions to create bundles for Node.js etc.
How can I accomplish this?
I'm trying to create a statemachine with a BatchSubmitJob in AWS CDK with dynamic environment variables in the BatchContainerOverrides. I was thinking about something like this:
container_overrides = sfn_tasks.BatchContainerOverrides(
environment={
"TEST.$": "$.dynamic_from_payload"
}
)
return sfn_tasks.BatchSubmitJob(self.scope,
id="id",
job_name="name",
job_definition_arn="arn",
job_queue_arn="arn",
container_overrides=container_overrides,
payload=sfn.TaskInput.from_object({
"dynamic_from_payload.$": "$.input.some_variable"
}))
However, upon deployment, CDK will add "Name" and "Value" to the statemachine definition, but Value is now static. This is part of the statemachine definition as seen in the console:
"Environment": [
{
"Name": "TEST.$",
"Value": "$.dynamic_from_payload"
}
]
But I need to have it like this:
"Environment": [
{
"Name": "TEST",
"Value.$": "$.dynamic_from_payload"
}
]
I also tried using "Ref::", as done here for the command parameters: AWS Step and Batch Dynamic Command. But this doesn't work either.
I also looked into escape hatches, overwriting the CloudFormation template. But I don't think that is applicable here, since the generated statemachine definition string is basically one large string.
I can think of two solutions, both of which don't make me happy: override the statemachine definition string with escape hatches with a copy in which "Value" is replaced on certain conditions (probably with regex) OR put a lambda in the statemachine that will create and trigger the batch job and a lambda that will poll if the job is finished.
Long story short: Does anyone have an idea of how to use dynamic environment variables with a BatchSubmitJob in CDK?
You can use the aws_cdk.aws_stepfunctions.JsonPath class:
container_overrides = sfn_tasks.BatchContainerOverrides(
environment={
"TEST": sfn.JsonPath.string_at("$.dynamic_from_payload")
}
)
Solved thanks to K. Galens!
I ended up with a Pass state with intrinsic functions to format the value and the aws_cdk.aws_stepfunctions.JsonPath for the BatchSubmitJob.
So something like this:
sfn.Pass(scope
id="id",
result_path="$.result",
parameters={"dynamic_from_payload.$": "States.Format('static_sub_part/{}',$.dynamic_sub_part)"} )
...
container_overrides = sfn_tasks.BatchContainerOverrides(
environment={
"TEST": sfn.JsonPath.string_at("$.result.dynamic_from_payload")
}
)
From the PR that implemented empty_dirs, it seems there's support for defining dir owners (with the names argument) and mode into the add_empty_dir method of TarFile class.
But the container_image rule (and container_layer) supports only mode.
This works:
container_image(
name = "with_empty_dirs",
empty_dirs = [
"etc",
"foo",
"bar",
],
mode = "0o777",
)
But this returns an error: "ERROR: (...) no such attribute 'names' in 'container_image_' rule":
container_image(
name = "with_empty_dirs",
empty_dirs = [
"etc",
"foo",
"bar",
],
names = "nginx",
)
Do we need to “write a customized container_image” if we want to add support for owner of empty_dirs?
In a BUILD file, the attribute you're looking for is ownername. See the pkg_tar reference documentation documentation for more details. Also, I don't think you can pass it directly to container_image, you have to create a separate pkg_tar first. Like this:
pkg_tar(
name = "with_empty_dirs_tar",
empty_dirs = [
"etc",
"foo",
"bar",
],
ownername = "nginx.nginx",
)
container_image(
name = "with_empty_dirs",
tars = [":with_empty_dirs_tar"],
)
In general, container_image has a subset of pkg_tar as direct attributes to make simple forms of adding files, but for complex use cases you should create a pkg_tar yourself for full access to all of its features for adding/arranging files and setting their ownership.
The names you see in that PR is a variable in a Python helper tool which the BUILD file rules use as part of the implementation. There's a layer of abstraction between what you write in a BUILD file and that Python code.
I want to check if a variable is defined in OPA policy.
> subject
1 error occurred: 1:1: rego_unsafe_var_error: var subject is unsafe
Is there a function to check if a variable is defined
You can use show command to show active module definition.
> show
package repl
b = 45534543
a = 3423
sites = [
{"name": "prod"},
{"name": "smoke1"},
{"name": "dev"},
]
You will find the list of defined variables here.
I'd like to switch to XML syntax in Sublime Text 2 using some key binding, for example Ctrl+Shift+X.
There is a command for that, I can successfully execute it from console:
view.set_syntax_file("Packages/XML/XML.tmLanguage")
I tried this binding, but it doesn't work:
{ "keys": ["ctrl+shift+x"], "command": "set_syntax_file", "args" : {"syntax_file" : "Packages/XML/XML.tmLanguage" }}
Here API reference for the set_syntax_file command can be found.
Any ideas?
Try this:
{ "keys": ["ctrl+shift+x"], "command": "set_file_type", "args" : {"syntax" : "Packages/XML/XML.tmLanguage" } }
set_syntax_file is an API command, so to use it I created a simple plug-in.