Azure Search - Query Collection of GeographyPoint - odata

I am trying to query a collection GeographyPoint within a given range from my index. I am using Azure Search.
My index contains a list of GeographyPoint
public class Parent
{
public List<Child> Coordinates {get; set;}
}
public class Child
{
public GeographyPoint Coordinates { get; set; }
}
I know if there was a single property named location on the index you could query it with something like:
$filter=geo.distance(location, geography'POINT(-82.51571 31.89063)') le 0.1524
but how do you query a collection? I've tried using the any/all filters like:
$filter=geo.distance(IndexedCompany/Offices/any(c: c/PhysicalAddress/Coordinates), geography'POINT(-82.51571 31.89063)') le 0.1524
but I get the error: "Invalid expression: The Any/All query expression must evaluate to a single boolean value.\r\nParameter name: $filter"
Any help on how to query a collection of GeographyPoint would be much appreciated.

The issue is in how you've constructed your filter. Let's break it down using types, since that's the easiest way to see where things went wrong.
$filter=geo.distance(IndexedCompany/Offices/any(c: c/PhysicalAddress/Coordinates), geography'POINT(-82.51571 31.89063)') le 0.1524
Starting from the outside in, the top-most expression in a filter must be Boolean, and that checks out here because you're comparing geo.distance (which returns Edm.Double) with a literal distance using le. So that part is fine.
Next, let's look at geo.distance. It takes two points of type Edm.GeographyPoint and returns an Edm.Double. The second parameter is a literal point, so that's fine, but the first parameter is not. Let's look at that next.
The first parameter to geo.distance here is the expression IndexedCompany/Offices/any(c: c/PhysicalAddress/Coordinates). Although it's really an operator, you can think of any as a kind of function that returns a Boolean. That would be a problem, because geo.distance is expecting a point, not a Boolean. That's our first clue to what's wrong here, but it doesn't actually explain the error message you're seeing. Let's dig further to understand that.
Assume for the moment that the expression IndexedCompany/Offices/any(c: c/PhysicalAddress/Coordinates) was your entire filter. You'd still get the same error message: "Invalid expression: The Any/All query expression must evaluate to a single boolean value." To see why, consider what any and all expect in terms of types. Each one iterates over a collection of some type, applying a predicate in the form of a lambda expression to each element. That lambda expression in this case is c/PhysicalAddress/Coordinates, which (I'm assuming) is an Edm.GeographyPoint, not a Boolean.
Taking a step back, it helps to think about the cardinalities involved here. Each document contains a company, which has many offices, which each has a geo-point location. I'm assuming the goal is to match the whole document in case any of those co-ordinates match the filter. In that case, you really just need to re-order the parts of your query so that the types and cardinalities line up:
$filter=IndexedCompany/Offices/any(office: geo.distance(office/PhysicalAddress/Coordinates, geography'POINT(-82.51571 31.89063)') le 0.1524)
The filter is still a Boolean expression because that's the type of any. The lambda passed to any takes an Office, applies geo.distance to its geo-location, and compares the distance to the desired value. Now everything type checks and the iteration is happening the right place.
Example: search=*
"value": [
{
"#search.score": 1,
"id": "myid1",
"pts": [
{
"type": "Point",
"coordinates": [ 0, 0 ],
"crs": { "type": "name", "properties": { "name": "EPSG:4326" } }
},
{
"type": "Point",
"coordinates": [ 0, 1 ],
"crs": { "type": "name", "properties": { "name": "EPSG:4326" } }
}
]
},
{
"#search.score": 1,
"id": "myid2",
"pts": [
{
"type": "Point",
"coordinates": [ 0, 10 ],
"crs": { "type": "name", "properties": { "name": "EPSG:4326" } }
},
{
"type": "Point",
"coordinates": [ 0, 11 ],
"crs": { "type": "name", "properties": { "name": "EPSG:4326" } }
}
]
}
]
Example $filter=pts/any(pt: geo.distance(pt, geography'POINT(0 2)') le 111.3192394008)
"value": [
{
"#search.score": 1,
"id": "myid1",
"pts": [
{
"type": "Point",
"coordinates": [ 0, 0 ],
"crs": { "type": "name", "properties": { "name": "EPSG:4326" } }
},
{
"type": "Point",
"coordinates": [ 0, 1 ],
"crs": { "type": "name", "properties": { "name": "EPSG:4326" } }
}
]
}
]

Related

Avro Records with Unknown Key Names and Quantity but Known Values

I am looking to convert JSON to Avro without altering the shape of the data
A nested field in the JSON contains a variable number of keys, which are never known in advance. The record that branches off of each of these unknown nodes however is of known, well-defined shape.
An example of this input data is as shown below
{
"customers": {
"paul_ince": {
"name": "Mr Paul Ince",
"age": 54
},
"kim_kardashian": {
"name": "Ms Kim Kardashian",
"age": 41
},
"elon_musk": {
"name": "Elon Musk, Technoking of Tesla",
"age": 50
}
}
Now it would be more avro friendly of course to have customers lead to an array of the form
{
"customers": [
{
"customer_name": "paul_ince",
"name": "Mr Paul Ince",
"age": 54
},
{
...
}
]
}
But this would violate my constraint that the shape of the input data be unchanged.
This problem seems to manifest itself frequently if I rely on data from external sources or scrapes, or preexisting data that was never created for Avro.
In my head, the schema would look something like the below,
{
"fields": [
{
"name": "customers",
"type": {
"type": "record",
"name": "customers",
"fields": [
{
"name": $customer_name,
"type": {
"type": "record",
"name": $customer_name,
"fields": [
{
"name": "name",
"type": "string",
},
{
"name": "age",
"type": "int"
}
]
}
}
]
}
}
]
}
where $customer_name is an assignment value, defined on read. Just asking the question it feels like this violates fundamental avro but I must use Avro and I strongly desire to maintain the input shape of the data. It would be highly impractical to modify this, not least given how frequently this problem appears and how large and varied the data I need to transfer from JSON to Avro is

Building an OpenAPI response, including oneOf, and maybe allOf

I am trying to build up a response from a variety of schema components using OpenAPI 3. There are basically three parts to the response:
A shared component that other endpoints use (i.e. success/failure flags). - #/components/schemas/core_response_schema inside allOf.
Properties that all responses on this endpoint use (i.e., user_id) - the properties component of the below.
One of several schemas that will vary depending on the type of user. - the oneOf component.
I've determined that I have to use allOf to be able to mix properties (item 2) and the core response (item 1), though this feels wrong as there's only one item. I tried a $ref, but it didn't work.
The below successfully passes three different OpenAPI linting tools, but in the example it builds, Swagger UI does not show the item 2 things (properties), and does show all of the item 3 things (should be oneOf).
"responses": {
"200": {
"description": "Operation successfully executed.",
"content": {
"application/json": {
"schema": {
"properties": {
"user_id": {
"$ref": "#/components/schemas/user_id"
},
"results": {
"type": "array",
"items": {
"$ref": "#/components/schemas/result_user_by_id"
}
}
},
"type": "object",
"allOf": [
{
"$ref": "#/components/schemas/core_response_schema"
}
],
"oneOf": [
{
"$ref": "#/components/schemas/user_type_a"
},
{
"$ref": "#/components/schemas/user_type_b"
},
{
"$ref": "#/components/schemas/user_type_c"
}
]
}
}
}
}
},
"components": {
"schemas": {
"core_response_schema": {
"properties": {
"success": {
"description": "A flag indicating whether the request was successfully completed or not.",
"type": "boolean"
},
"num_results": {
"description": "The number of results for this request",
"type": "integer"
}
},
"type": "object"
},
"user_id": {
"description": "Unique 10 character `user_id`.",
"type": "string",
"maxLength": 10,
"minLength": 10,
"example": "a1b2c3d4e5"
},
}
}
And example payloads for two users. Type A and B (it's a contrived example).
User Type A:
{
"success": true,
"num_results": 1,
"user_id": "c1b00cb714",
"results": [{
"user_type": "a",
"group_id": "e7a99e3769",
"name": null,
"title": null,
... (and so on until we get to the stuff that's unique to this type of user) ...
"favourite_artworks": [
"sunflowers",
"landscapes"
],
"artwork_urls": [
"http://sunflowers.example"
]
}
]
}
User Type B:
{
"success": true,
"num_results": 1,
"user_id": "c1b00cb715",
"results": [{
"user_type": "B",
"group_id": "e7a99e3769",
"name": null,
"title": null,
... (and so on until we get to the stuff that's unique to this type of user) ...
"supported_charities": [
"UN Foundations"
],
"charity_urls": [
"http://www.un.int"
],
}
]
}
What's the correct way to merge together different schemas and properties in OpenAPI? Is this right and Swagger UI just can't handle it?
And how do you mix a schema with properties without having to use allOf?
This suggests it's possible: Swagger Schema: oneOf, anyOf, allOf valid at the same time?
After further investigation, I've determined this is a bug in swagger-ui - https://github.com/swagger-api/swagger-ui/issues/3803 - they simply don't support oneOf (or anyOf) currently.
As far as at least three different linting tools are concerned, a mixture of anyOf, oneOf, and allOf can be used together in the same schema.
Redoc appears to have similar problems - https://github.com/Rebilly/ReDoc/issues/641

In Watson Discovery, limiting "return"ed fields to aggregation values

For the Discovery REST api, the argument/parameter "return" controls which fields are returned.
So if I pass these arguments to the API
{
"query": named_sector,
"count": "10",
"filter": filter_dates,
"aggregation" : "term(docSentiment.type,count:3)"
}
my_query = discovery.query(my_disc_environment_id, my_disc_collection_id, qopts)
print(json.dumps(my_query, indent=2))
I get the following:
{
"matching_results": 14779,
"aggregations": [
{
"type": "term",
"field": "docSentiment.type",
"count": 3,
"results": [
{
"key": "positive",
"matching_results": 4212
},
{
"key": "negative",
"matching_results": 3259
},
{
"key": "neutral",
"matching_results": 152
}
]
}
],
"results": [
{
"id": "6389715fe7e7f711e0bc09d4f1236639",
"score": 1.3689895,
"yyyymm": "201704",
"url": "https://seekingalpha.com/article/4060446-valuation-dashboard-consumer-discretionary-update",
"enrichedTitle": null,
"host": "seekingalpha.com",
"text": "Valuation Dashboard: Consumer Discretionary - Update\n\nSummary\n\nValuation metrics in Consumer Discretionary.\n\nEvolution since last month.\n\nA list of stocks loo ....
and thousands of more lines. How do I restrict the output to the aggregations section? Is this an issue of me better handling the JSON structure that is returned?
thanks
If you change the count argument to 0, the returned JSON will only contain the aggregations.
Also, if you're using the Discovery web tooling, you can enter 0 for the "Number of results to return (Count)" field.
More details and an example can be found here: https://www.ibm.com/watson/developercloud/doc/discovery/using.html#building-aggregations

Can a json response can be partially paginate?

I'm wondering if a json can be partially paginate.
For example
{
"data": [{
"type": "articles",
"id": "1",
"attributes": {
"title": "JSON API paints my bikeshed!",
"body": "The shortest article. Ever."
}
}],
"included": [
{
"type": "people",
"id": 42,
"attributes": {
"name": "John"
}
},
{
...annnd 80000 others
}
}
]
}
Where included have soo many elements (80.000 for examples) than maybe we need pagination?
But if it's paginate and we go on the next page only included elements will change, the json will still return the data.articles.
Is it a correct behavior ?
First proposal :
{
"data": [{
"type": "articles",
"id": "1",
"attributes": {
"title": "JSON API paints my bikeshed!",
"body": "The shortest article. Ever."
},
"relationships": {
"users": {
"link": "https://website.com/api/v1/articles/1/users.json"
}
}
}]
}
To be compliant with the JSON API spec, your compound document must obey the full linkage requirement. Any included resources MUST be identified via relationship data.
In your example, you could fulfill this by adding a data member under the users relationship. You could then link to every included person.
If the relationship data is a partial set, you can use pagination links within the relationship object.

Problems extracting values from geojson

I'm struggling to create an array of the values for D01. This is the first feature in my geojson variable:
county = [{
"type": "FeatureCollection",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:EPSG::4269" } },
"features": [
{ "type": "Feature", "properties": { "NAMELSAD10": "St. Clair County", "D01": 5.650500, "D02": 0.504000, "D03": 0.495000, "D04": 48.100000, "D05": 0.199000, "D06": 0.965000, "D07": 0.038000, "D08": 0.031000, "D09": 0.211000 }, "geometry": { "type": "Polygon", "coordinates": [
//so many coordinate pairs
] } } ] ]
To clarify, I want [5.650500, ...]
From this post I used:
help = county.features.map(function(f){
return f.properties.D01;
})
which gives the error: Cannot read property "map" of undefined.
Thinking some of my values might be null, I wrote this:
var help = new Array();
try{
help = county.features.map(function(f){
return f.properties.D01;
})
}catch(e){}
console.log(help);
which results in [], the blank 'help' array being written.
It seems I'm not accessing what I want to here. Can someone please point me in the right direction?
Your county seems an Array.
try this:
county[0].features.map(function(f){
return f.properties.D01;
})

Resources