I have this dataset:
[
{
"date": "2020-06-07",
"data": {
"publish": true,
"publish:title": "Hello world from tempo!"
}
},
{
"date": "2020-06-07",
"data": null
},
{
"date": "2020-06-07",
"data": null
},
{
"date": "2020-06-07",
"data": {
"sleep:start": 9.5,
"sleep:duration": 10
}
},
{
"date": "2020-06-06",
"data": null
},
{
"date": "2020-06-06",
"data": null
}
]
I want to select the date and data["sleep:start"] values for each row using Alasql. The documentation suggests using obj->property->subproperty for that purpose, so my Alasql query should look a bit like this:
SELECT date, data->sleep:start FROM ? ORDER BY date DESC LIMIT 6
Unfortunately, this doesn't work, and quoting sleep:start doesn't either. Is there a way to do what I'm looking for using Alasql?
Nevermind, it works with this: SELECT date, data->[sleep:duration] FROM ? ORDER BY date DESC LIMIT 6
ref https://github.com/agershun/alasql/issues/1201#issuecomment-639171925
Related
Pulling data into Google Data Studio from a Google Sheet with dates stored in yyyy-mm-dd format. The dates look correct and calculate correctly with formulas and adjustments everywhere except in a Gantt chart using the Vega-Lite Community Visualization, which shows the date in a long-number format (e.g. 20210520), and is unable to display the data when using "type": "temporal" or using "timeUnit": "utcyearmonthdatehours".
I've ran various tests, including...
Changing the date format for the date columns to plain text, yyyyddmm, yymmdd, yyyy/mm/dd formats.
Replace the current date columns with new columns using the alternate formats in point 1 (above).
Changing the date field formats directly in Google Data Studio to the formats in point 1 (above).
Creating a secondary set of date columns in plain-text using an Arrayformula and Text() function to reformat the actual dates to plain-text.
So far, options 2 & 4 are the only way I've been able to get the gantt to render correctly, reading the data in date format. But option 2 renders the other charts in GDS as unusable, as the other charts cannot translate the plain-text to usable dates.
Option 4 does work, but isn't the ideal route, given the redundant data. I'd prefer to have just 1 column for the Start Date and another for the End Date, rather than 2 columns for both. Feels like I may be missing something obvious here. Is there a way to either properly format the dates in Google Sheets to work properly with both the GDS date fields and Vega-Lite, or is there a way to properly parse the date data in Vega-Lite without needing to use a second set of plain-text columns?
Report replicating the issue: Project Tracking (debug report)
Edit: below is the code for the Vega-lite visualizations using the date fields from Google Sheets, which Vega-lite is not interpreting as dates.
Without timeunit or temporal field type:
{
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
"description": "A bar chart with highlighting on hover and selecting on click. (Inspired by Tableau's interaction style.)",
"config": {
"background": null,
"view": {
"stroke": "transparent"
}
},
"layer": [
{
"layer": [
{
"params": [
{
"name": "grid",
"select": "interval",
"bind": "scales"
}
],
"mark": {
"type": "bar",
"cursor": "pointer",
"tooltip": true,
"point": true,
"cornerRadiusEnd": 5,
"opacity": 0.8
},
"encoding": {
"color": {
"field": "$dimension3",
"title": "$dimension3.name"
}
}
}
],
"encoding": {
"x": {
"field": "$dimension0",
"axis": {
"title": null,
"grid": true
}
},
"y": {
"field": "$dimension1",
"title": "$dimension1.name",
"type": "nominal",
"sort": "x",
"axis": {
"title": null,
"grid": true,
"tickBand": "extent"
}
},
"x2": {
"field": "$dimension2"
},
"yOffset": {
"field": "$dimension3"
}
}
}
]
}
With timeunit and field type temporal:
{
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
"description": "A bar chart with highlighting on hover and selecting on click. (Inspired by Tableau's interaction style.)",
"config": {
"background": null,
"view": {
"stroke": "transparent"
}
},
"layer": [
{
"layer": [
{
"params": [
{
"name": "grid",
"select": "interval",
"bind": "scales"
}
],
"mark": {
"type": "bar",
"cursor": "pointer",
"tooltip": true,
"point": true,
"cornerRadiusEnd": 5,
"opacity": 0.8
},
"encoding": {
"color": {
"field": "$dimension3",
"title": "$dimension3.name"
}
}
}
],
"encoding": {
"x": {
"field": "$dimension0",
"type": "temporal",
"timeUnit": "utcyearmonthdatehours",
"axis": {
"title": null,
"grid": true
}
},
"y": {
"field": "$dimension1",
"title": "$dimension1.name",
"type": "nominal",
"sort": "x",
"axis": {
"title": null,
"grid": true,
"tickBand": "extent"
}
},
"x2": {
"field": "$dimension2"
},
"yOffset": {
"field": "$dimension3"
}
}
}
]
}
We have an Microsoft Search instance for crawling one custom app : https://learn.microsoft.com/en-us/microsoftsearch/connectors-overview
Query & display is working as expected but aggregation provides wrong results
query JSON : https://graph.microsoft.com/v1.0/search/query
select title + submitter and aggregation on submitter
"fields": [
"title",
"submitter"
],
"aggregations": [
{
"field": "submitter",
"size": 1,
"bucketDefinition": {
"sortBy": "keyAsString",
"isDescending": true,
"minimumCount": 0
}
}
]
JSON response
submitter property is correctly returned with Firstname Lastname on row 0 but aggregate is lowercase and middle space trimmed firstnamelastname
"hitsContainers": [
{
"total": 1,
"moreResultsAvailable": false,
"hits": [
{
"hitId": "xxxx",
"contentSource": "ConnectionId",
"rank": 1,
"summary": "New service / <c0>business</c0> <c0>model</c0> <c0>design</c0> <ddd/>",
"resource": {
"#odata.type": "#microsoft.graph.externalConnectors.externalItem",
"properties": {
"title": "New service / business model design",
"submitter": "Firstname Lastname"
}
}
}
],
"aggregations": [
{
"field": "submitter",
"buckets": [
{
"key": "firstnamelastname",
"count": 1,
"aggregationFilterToken": "\"ǂǂ696c736573706f656c73747261\""
}
]
}
]
}
]
reproducible in Microsoft Graph Explorer (a bit obfuscated)
result with space
aggregation concatenated in lowercase
Rootcause has been identified as submitter property wasn't created with flag refinable
{
"name": "submitter",
"type": "String",
"isSearchable": "true",
"isQueryable": "true",
"isRetrievable": "true"
"isRefinable": "false"
}
as a consequence, output was incorrect.
testing with refinable = true provides correct aggregation value (1 = non refinable, 2 = refinable).
small note : refinable properties can't be searchable
I am trying to use the filter in the expanded entity using the Odata query like
https://labstest.science.com/DEV/odata/ROOM_REQUEST?$expand=REQ_COLONYROOM($select=Name;$filter=Name eq 'RB05')
This works partially. It brings all the ROOM_REQUEST with Null for the REQ_COLONYROOM where Name is not equal to RB05
{
"#odata.context": "https://labstest.science.com/DEV/odata/$metadata#ROOM_REQUEST(REQ_COLONYROOM(Name))",
"value": [
{
"Id": 18399308,
"Name": "M1",
"Barcode": "M1",
"REQ_COLONYROOM": null
},
{
"Id": 18403071,
"Name": "M2",
"Barcode": "M2",
"REQ_COLONYROOM": {
"#odata.id": "https://labstest.science.com/odata/COLONY_ROOM('MC19')",
"Name": "RB05"
}
},
{
"Id": 18403109,
"Name": "M3",
"Barcode": "M3",
"REQ_COLONYROOM": null
}]}
Though M1 and M3 has different Colony than RB05 it returns Null with the odata call
Did you try something like this:
https://labstest.science.com/DEV/odata/ROOM_REQUEST?$expand=REQ_COLONYROOM&$select=REQ_COLONYROOM/Name&$filter=REQ_COLONYROOM/Name eq 'RB05'
I think it would be enough not having SELECT:
https://labstest.science.com/DEV/odata/ROOM_REQUEST?$expand=REQ_COLONYROOM&$filter=REQ_COLONYROOM/Name eq 'RB05'
For the Discovery REST api, the argument/parameter "return" controls which fields are returned.
So if I pass these arguments to the API
{
"query": named_sector,
"count": "10",
"filter": filter_dates,
"aggregation" : "term(docSentiment.type,count:3)"
}
my_query = discovery.query(my_disc_environment_id, my_disc_collection_id, qopts)
print(json.dumps(my_query, indent=2))
I get the following:
{
"matching_results": 14779,
"aggregations": [
{
"type": "term",
"field": "docSentiment.type",
"count": 3,
"results": [
{
"key": "positive",
"matching_results": 4212
},
{
"key": "negative",
"matching_results": 3259
},
{
"key": "neutral",
"matching_results": 152
}
]
}
],
"results": [
{
"id": "6389715fe7e7f711e0bc09d4f1236639",
"score": 1.3689895,
"yyyymm": "201704",
"url": "https://seekingalpha.com/article/4060446-valuation-dashboard-consumer-discretionary-update",
"enrichedTitle": null,
"host": "seekingalpha.com",
"text": "Valuation Dashboard: Consumer Discretionary - Update\n\nSummary\n\nValuation metrics in Consumer Discretionary.\n\nEvolution since last month.\n\nA list of stocks loo ....
and thousands of more lines. How do I restrict the output to the aggregations section? Is this an issue of me better handling the JSON structure that is returned?
thanks
If you change the count argument to 0, the returned JSON will only contain the aggregations.
Also, if you're using the Discovery web tooling, you can enter 0 for the "Number of results to return (Count)" field.
More details and an example can be found here: https://www.ibm.com/watson/developercloud/doc/discovery/using.html#building-aggregations
I have a group of records being returned by an Active Record query. I need to somehow iterate over these to build an array of hashes that sort of looks like:
[
{ '15/01/2013': 3 },
{ '24/051/2012': 1 },
{ '08/07/2015': 22 }
]
The records returned will be in the form of having 3 records for the first, 1 record for the second and 22 records for the 3rd. I basically need a way to transform that into a simple hash like above.
Here's an example of my DB output:
[
{
"item_id": 21728,
"id": 31731,
"notes": "sample",
"date": "2014-09-12",
"created_at": "2015-08-13T15:16:06.074Z",
"updated_at": "2015-08-13T15:16:06.074Z"
},
{
"item_id": 21728,
"id": 31732,
"notes": "sample",
"date": "2014-09-12",
"created_at": "2015-08-13T15:16:06.104Z",
"updated_at": "2015-08-13T15:16:06.104Z"
},
{
"item_id": 21728,
"id": 31733,
"notes": "sample",
"date": "2014-09-13",
"created_at": "2015-08-13T15:16:06.112Z",
"updated_at": "2015-08-13T15:16:06.112Z"
}
]
Let's say your DB results are stored in an array called results.
results.group_by(&:date).map{|k,v| {k => v.count}}
This, first groups your DB results according to their dates and then counts how many objects correspond to each date.