Related
Unable to Error creating a kafka message to producer - Expected start-union. Got VALUE_STRING
{
"namespace": "de.morris.audit",
"type": "record",
"name": "AuditDataChangemorris",
"fields": [
{"name": "employeeID", "type": "string"},
{"name": "employeeNumber", "type": ["null", "string"], "default": null},
{"name": "serialNumbers", "type": [ "null", {"type": "array", "items": "string"}]},
{"name": "correlationId", "type": "string"},
{"name": "timestamp", "type": "long", "logicalType": "timestamp-millis"},
{"name": "employmentscreening","type":{"type": "enum", "name": "employmentscreening", "symbols": ["NO","YES"]}},
{"name": "vouchercodes","type": ["null",
{
"type": "array",
"items": {
"name": "Vouchercodes",
"type": "record",
"fields": [
{"name": "voucherName","type": ["null","string"], "default": null},
{"name": "authocode","type": ["null","string"], "default": null}
]
}
}], "default": null}
]
}
when i was trying to create a sample data in json format based on the above avsc for kafka consumer i am getting the below error upon testing
{
"employeeID": "qtete46524",
"employeeNumber": {
"string": "custnumber9813"
},
"serialNumbers": {
"type": "array",
"items": ["363536623","5846373733"]
},
"correlationId": "corr-656532443",
"timestamp": 1476538955719,
"employmentscreening": "NO",
"vouchercodes": [
{
"voucherName": "skygo",
"authocode": "A238472ASD"
}
]
}
getting the below error when i got when i ran the dataflow job in gcp
Error message from worker: java.lang.RuntimeException: java.io.IOException: Insert failed: [{"errors":[{"debugInfo":"","location":"serialnumbers","message":"Array specified for non-repeated field: serialnumbers.","reason":"invalid"}],"index":0}]**
how to create correct sample data based on the above schema ?
Read the spec
The value of a union is encoded in JSON as follows:
if its type is null, then it is encoded as a JSON null;
otherwise it is encoded as a JSON object with one name/value pair whose name is the type’s name and whose value is the recursively encoded value
So, here's the data it expects.
{
"employeeID": "qtete46524",
"employeeNumber": {
"string": "custnumber9813"
},
"serialNumbers": {"array": [
"serialNumbers3521"
]},
"correlationId": "corr-656532443",
"timestamp": 1476538955719,
"employmentscreening": "NO",
"vouchercodes": {"array": [
{
"voucherName": {"string": "skygo"},
"authocode": {"string": "A238472ASD"}
}
]}
}
With this schema
{
"namespace": "de.morris.audit",
"type": "record",
"name": "AuditDataChangemorris",
"fields": [
{
"name": "employeeID",
"type": "string"
},
{
"name": "employeeNumber",
"type": [
"null",
"string"
],
"default": null
},
{
"name": "serialNumbers",
"type": [
"null",
{
"type": "array",
"items": "string"
}
]
},
{
"name": "correlationId",
"type": "string"
},
{
"name": "timestamp",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
}
},
{
"name": "employmentscreening",
"type": {
"type": "enum",
"name": "employmentscreening",
"symbols": [
"NO",
"YES"
]
}
},
{
"name": "vouchercodes",
"type": [
"null",
{
"type": "array",
"items": {
"name": "Vouchercodes",
"type": "record",
"fields": [
{
"name": "voucherName",
"type": [
"null",
"string"
],
"default": null
},
{
"name": "authocode",
"type": [
"null",
"string"
],
"default": null
}
]
}
}
],
"default": null
}
]
}
Here's an example of producing and consuming to Kafka
$ jq -rc < /tmp/data.json | kafka-avro-console-producer --topic foobar --property value.schema="$(jq -rc < /tmp/data.avsc)" --bootstrap-server localhost:9092 --sync
$ kafka-avro-console-consumer --topic foobar --from-beginning --bootstrap-server localhost:9092 | jq
{
"employeeID": "qtete46524",
"employeeNumber": {
"string": "custnumber9813"
},
"serialNumbers": {
"array": [
"serialNumbers3521"
]
},
"correlationId": "corr-656532443",
"timestamp": 1476538955719,
"employmentscreening": "NO",
"vouchercodes": {
"array": [
{
"voucherName": {
"string": "skygo"
},
"authocode": {
"string": "A238472ASD"
}
}
]
}
}
^CProcessed a total of 1 messages
I want to convert a grib2 file to a geojson with the following format:
{
"type": "FeatureCollection",
"features": [
{ "type": "Feature", "properties": { "ID": 0, "sigwaveht": 1.000000 }, "geometry": { "type": "LineString", "coordinates": [ [ 20.5, 77.559374979743737 ], [ 20.756756711040964, 77.5 ], [ 21.0, 77.426829270065582 ], [ 21.5, 77.426829270065582 ] ] } },
{ "type": "Feature", "properties": { "ID": 1, "sigwaveht": 1.000000 }, "geometry": { "type": "LineString", "coordinates": [ [ 17.5, 76.879518074163784 ], [ 18.0, 76.840000001907356 ], [ 18.555555592348554, 77.0 ], [ 18.555555592348554, 77.5 ] ] } },
{ "type": "Feature", "properties": { "ID": 2, "sigwaveht": 1.000000 }, "geometry": { "type": "LineString", "coordinates": [ [ 28.5, 76.732142838136269 ], [ 29.0, 76.634146323734484 ], [ 29.937500058207661, 77.0 ], [ 29.937500058207661, 77.5 ] ] } },
I can accomplish this by using ogr2ogr2 to convert a shape file to a geojson in this format but what can I do to convert a grib2 to a geoJSON of this format?
You can't convert a GRIB, which is a raster format, to GeoJSON, which is a vector format.
What do you expect to achieve? Vector data composed of points where each point is one of the pixels of the raster format?
If this is what you want, you will probably have to code it yourself, I don't think there are any standard tools to do this. Just make a loop over the raster data pixels and write one point feature for every pixel.
Is it possible to add an access policy via a conditional statement? Basically, if environment == production I don't want to add the registration.
I have the following in my template however I don't want the application called foobarApplicationId to be added if the environment is production. Can I do this inline or do I need a seperate template? Will setting foobarApplicationId to be an empty string work?
{
"name": "[variables('keyVault-name')]",
"type": "Microsoft.KeyVault/vaults",
"apiVersion": "2016-10-01",
"location": "[resourceGroup().location]",
"properties": {
"tenantId": "[subscription().tenantId]",
"sku": {
"family": "A",
"name": "standard"
},
"accessPolicies": [
{
"tenantId": "[subscription().tenantId]",
"objectId": "[parameters('keyVaultOwner')]",
"permissions": {
"keys": [
"all"
],
"secrets": [
"all"
],
"certificates": [
"all"
],
"storage": [
]
}
},
{
"tenantId": "[subscription().tenantId]",
"objectId": "[parameters('foobarApplicationId')]",
"permissions": {
"keys": [
"get",
"wrapKey",
"unwrapKey",
"sign",
"verify",
"list"
],
"secrets": [
"get",
"list"
],
"certificates": [
"get",
"list"
],
"storage": [
]
}
},
"condition" within "accessPolicies" doesn't seem to have any effect for me. It doesn't cause any validation or deployment error, but the access policies got added even when the condition evaluated to false.
I found the following trick works better: Use an if clause for your "objectId" and "permissions", such that if the condition is false, you would assign an empty set of permissions to the empty GUID, effectively becoming a no-op.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"variables": {
"keyVaultNoPermissions": { },
"keyVaultAppReadPermissions": {
"keys": [ "get", "wrapKey", "unwrapKey", "sign", "verify", "list" ],
"secrets": [ "get", "list" ],
"certificates": [ "get", "list" ]
}
},
"resources": [
// ...
{
"type": "Microsoft.KeyVault/vaults/accessPolicies",
"apiVersion": "2016-10-01",
"name": "[concat(parameters('keyVaultName'), '/add')]",
"location": "[resourceGroup().location]",
"dependsOn": [
"[parameters('keyVaultName')]"
],
"properties": {
"accessPolicies": [
{
"tenantId": "[subscription().tenantId]",
"objectId": "[if(not(equals(parameters('environment'), 'PROD')), parameters('foobarApplicationId'), '00000000-0000-0000-0000-000000000000')]",
"permissions": "[if(not(equals(parameters('environment'), 'PROD')), variables('keyVaultAppReadPermissions'), variables('keyVaultNoPermissions'))]"
}
]
}
}
]
}
It would be in the individual access policy adding a condition section that will take an environment parameter like:
{
"condition": "[not(equals(parameters('environment'),'PROD'))]"
"tenantId": "[subscription().tenantId]",
"objectId": "[parameters('foobarApplicationId')]",
"permissions": {
"keys": [
"get",
"wrapKey",
"unwrapKey",
"sign",
"verify",
"list"
],
"secrets": [
"get",
"list"
],
"certificates": [
"get",
"list"
],
"storage": [
]
}
}
Add the access policies separately, conditionally. You can see an explanation here.
{
"resources": [{
"name": "[variables('keyVault-name')]",
"type": "Microsoft.KeyVault/vaults",
"apiVersion": "2016-10-01",
"location": "[resourceGroup().location]",
"properties": {
"tenantId": "[subscription().tenantId]",
"sku": {
"family": "A",
"name": "standard"
},
"accessPolicies": [{
"tenantId": "[subscription().tenantId]",
"objectId": "[parameters('keyVaultOwner')]",
"permissions": {
"keys": [
"all"
],
"secrets": [
"all"
],
"certificates": [
"all"
],
"storage": [
]
}
}
]
}
}, {
"name": "[concat(variables('keyVault-name'), '/add')]",
"type": "Microsoft.KeyVault/vaults",
"apiVersion": "2016-10-01",
"condition": "[not(startsWith(parameters('environmentName'), 'PROD'))]",
"location": "[resourceGroup().location]",
"properties": {
"tenantId": "[subscription().tenantId]",
"sku": {
"family": "A",
"name": "standard"
},
"accessPolicies": [{
"tenantId": "[subscription().tenantId]",
"objectId": "[parameters('foobarApplicationId')]",
"permissions": {
"keys": [
"get",
"wrapKey",
"unwrapKey",
"sign",
"verify",
"list"
],
"secrets": [
"get",
"list"
],
"certificates": [
"get",
"list"
],
"storage": [
]
}
}
]
}
}
]
}
I'm having an issue parsing some of the JSON data being retrieved from Oxford Dictionaries into my application.
I am printing the JSON response into my console which confirms that I am successfully getting the data needed.
Verification that JSON data is being retrieved:
There are two main folders initially when I get the JSON data back. "results", which contains the information I need: the word definition and other information
and "metadata".
JSON format that I am getting back from Oxford Dictionary:
I am able to parse the information contained in the "metadata" folder, and have printed it into the console to confirm.
Verification that I am able to parse and print the metadata:
I however, can't seem to parse any of the data contained in the 'Results' folder, which is what I actually need. The definition, the language, the word ID, et cetera.
What am I doing wrong here??
Example of issue with parsing "results" data:
Another example of faulty attempt to parse 'results':
Plain text copy of my JSON response:
{
"metadata": {
"provider": "Oxford University Press"
},
"results": [
{
"id": "ace",
"language": "en",
"lexicalEntries": [
{
"entries": [
{
"etymologies": [
"Middle English (denoting the ‘one’ on dice): via Old French from Latin as ‘unity, a unit’"
],
"grammaticalFeatures": [
{
"text": "Singular",
"type": "Number"
}
],
"homographNumber": "100",
"senses": [
{
"definitions": [
"a playing card with a single spot on it, ranked as the highest card in its suit in most card games"
],
"domains": [
"Cards"
],
"examples": [
{
"registers": [
"figurative"
],
"text": "life had started dealing him aces again"
},
{
"text": "the ace of diamonds"
}
],
"id": "m_en_gbus0005680.006",
"short_definitions": [
"playing card with single spot on it, ranked as highest card in its suit in most card games"
]
},
{
"definitions": [
"a person who excels at a particular sport or other activity"
],
"domains": [
"Sport"
],
"examples": [
{
"text": "a motorcycle ace"
}
],
"id": "m_en_gbus0005680.010",
"registers": [
"informal"
],
"short_definitions": [
"person who excels at particular sport or other activity"
],
"subsenses": [
{
"definitions": [
"a pilot who has shot down many enemy aircraft"
],
"domains": [
"Air Force"
],
"examples": [
{
"text": "a Battle of Britain ace"
}
],
"id": "m_en_gbus0005680.011",
"short_definitions": [
"pilot who has shot down many enemy aircraft"
]
}
],
"thesaurusLinks": [
{
"entry_id": "ace",
"sense_id": "t_en_gb0000173.001"
}
]
},
{
"definitions": [
"(in tennis and similar games) a service that an opponent is unable to return and thus wins a point"
],
"domains": [
"Tennis"
],
"examples": [
{
"text": "Nadal banged down eight aces in the set"
}
],
"id": "m_en_gbus0005680.013",
"short_definitions": [
"(in tennis and similar games) service that opponent is unable to return and thus wins point"
],
"subsenses": [
{
"definitions": [
"a hole in one"
],
"domains": [
"Golf"
],
"examples": [
{
"text": "his hole in one at the 15th was Senior's second ace as a professional"
}
],
"id": "m_en_gbus0005680.014",
"registers": [
"informal"
],
"short_definitions": [
"hole in one"
]
}
]
}
]
},
{
"etymologies": [
"early 21st century: abbreviation of asexual, with alteration of spelling on the model of ace"
],
"grammaticalFeatures": [
{
"text": "Singular",
"type": "Number"
}
],
"homographNumber": "200",
"senses": [
{
"definitions": [
"a person who has no sexual feelings or desires"
],
"domains": [
"Sex"
],
"examples": [
{
"text": "both asexual, they have managed to connect with other aces offline"
}
],
"id": "m_en_gbus1190638.004",
"short_definitions": [
"asexual person"
]
}
]
}
],
"language": "en",
"lexicalCategory": "Noun",
"pronunciations": [
{
"audioFile": "http://audio.oxforddictionaries.com/en/mp3/ace_1_gb_1_abbr.mp3",
"dialects": [
"British English"
],
"phoneticNotation": "IPA",
"phoneticSpelling": "eɪs"
}
],
"text": "ace"
},
{
"entries": [
{
"grammaticalFeatures": [
{
"text": "Positive",
"type": "Degree"
}
],
"homographNumber": "101",
"senses": [
{
"definitions": [
"very good"
],
"examples": [
{
"text": "Ace! You've done it!"
},
{
"text": "an ace swimmer"
}
],
"id": "m_en_gbus0005680.016",
"registers": [
"informal"
],
"short_definitions": [
"very good"
],
"thesaurusLinks": [
{
"entry_id": "ace",
"sense_id": "t_en_gb0000173.002"
}
]
}
]
},
{
"grammaticalFeatures": [
{
"text": "Positive",
"type": "Degree"
}
],
"homographNumber": "201",
"senses": [
{
"definitions": [
"(of a person) having no sexual feelings or desires; asexual"
],
"domains": [
"Sex"
],
"examples": [
{
"text": "I didn't realize that I was ace for a long time"
}
],
"id": "m_en_gbus1190638.006",
"short_definitions": [
"asexual"
]
}
]
}
],
"language": "en",
"lexicalCategory": "Adjective",
"pronunciations": [
{
"audioFile": "http://audio.oxforddictionaries.com/en/mp3/ace_1_gb_1_abbr.mp3",
"dialects": [
"British English"
],
"phoneticNotation": "IPA",
"phoneticSpelling": "eɪs"
}
],
"text": "ace"
},
{
"entries": [
{
"grammaticalFeatures": [
{
"text": "Transitive",
"type": "Subcategorization"
},
{
"text": "Present",
"type": "Tense"
}
],
"homographNumber": "102",
"senses": [
{
"definitions": [
"(in tennis and similar games) serve an ace against (an opponent)"
],
"domains": [
"Tennis"
],
"examples": [
{
"text": "he can ace opponents with serves of no more than 62 mph"
}
],
"id": "m_en_gbus0005680.020",
"registers": [
"informal"
],
"short_definitions": [
"(in tennis and similar games) serve ace against"
],
"subsenses": [
{
"definitions": [
"score an ace on (a hole) or with (a shot)"
],
"domains": [
"Golf"
],
"examples": [
{
"text": "there was a prize for the first player to ace the hole"
}
],
"id": "m_en_gbus0005680.026",
"short_definitions": [
"score ace on hole or with"
]
}
]
},
{
"definitions": [
"achieve high marks in (a test or exam)"
],
"examples": [
{
"text": "I aced my grammar test"
}
],
"id": "m_en_gbus0005680.028",
"regions": [
"North American"
],
"registers": [
"informal"
],
"short_definitions": [
"achieve high marks in"
],
"subsenses": [
{
"definitions": [
"outdo someone in a competitive situation"
],
"examples": [
{
"text": "the magazine won an award, acing out its rivals"
}
],
"id": "m_en_gbus0005680.029",
"notes": [
{
"text": "\"ace someone out\"",
"type": "wordFormNote"
}
],
"short_definitions": [
"outdo someone in competitive situation"
]
}
]
}
]
}
],
"language": "en",
"lexicalCategory": "Verb",
"pronunciations": [
{
"audioFile": "http://audio.oxforddictionaries.com/en/mp3/ace_1_gb_1_abbr.mp3",
"dialects": [
"British English"
],
"phoneticNotation": "IPA",
"phoneticSpelling": "eɪs"
}
],
"text": "ace"
}
],
"type": "headword",
"word": "ace"
}
]
}
Retrieve results like this:
let results = json["results"][0]["id"]
Or
if let resultObj = json["results"].first {
let id = resultObj["id"]
}
id key is in the dictionary which is at zero index of results array.
It sounds super simple, but I can't get how can I use geojson, not topojson, for my polygons.
that's my current attempt:
"data": [
{
"name": "nabs",
"url": "both_boundaries.geojson",
"format": {"type": "json"},
"transform": [
{
"type": "geopath", "projection": "mercator",
"scale": 74, "center": [-73.99,40.72]
}
]
}
]
You have to parse the features using property within your format:
"format": {"type": "json", "property":"features"},
Full example spec:
{"$schema": "https://vega.github.io/schema/vega/v3.0.json",
"width": 500,
"height": 600,
"autosize": "none",
"signals": [
{
"name": "translate0",
"update": "width / 2"
},
{
"name": "translate1",
"update": "height / 2"
}
],
"projections": [
{
"name": "projection",
"size": {"signal": "[width, height]"},
"fit": {"signal": "data('netherlands')"}
}
],
"data": [
{
"name": "netherlands",
"url": "https://raw.githubusercontent.com/mattijn/datasets/master/NL_outline_geo.json",
"format": {
"type": "json",
"property": "features"
}
}
],
"marks": [
{
"type": "shape",
"from": {
"data": "netherlands"
},
"encode": {
"update": {
"strokeWidth": {
"value": 0.5
},
"stroke": {
"value": "darkblue"
},
"fill": {
"value": "lightblue"
},
"fillOpacity": {
"value": 0.5
}
},
"hover": {
"fill": {
"value": "#66C2A5"
},
"strokeWidth": {
"value": 2
},
"stroke": {
"value": "#FC8D62"
}
}
},
"transform": [
{
"type": "geoshape",
"projection": "projection"
}
]
}
]
}