I’m looking at -and trying to retrieve- data for a specific search query on the full archive (premium) search API (V1) but I’m getting a weird "hole" of about 9 days of data from 7 to 16 January. Results for the rest of January up until now are apparently ok.
Parameters passed to the search endpoint are:
'query' => '<a longish query string about 750 characters>',
'fromDate' => '202301070000',
'toDate' => '202301140830',
'maxResults' => '500'
but apparently the data is missing from the count endpoint as well, since this is what I’m getting with a bucket=day granularity (this covers all of Jan up to now):
{
"results": [{
"timePeriod": "202301010000",
"count": 525
},
>>>>> ...EVERYTHING FINE UP TO HERE <<<<<<
{
"timePeriod": "202301070000",
"count": 15 <--- THIS IS A PARTIAL RESULT
}, {
"timePeriod": "202301080000",
"count": 0
}, {
"timePeriod": "202301090000",
"count": 0
}, {
"timePeriod": "202301100000",
"count": 0
}, {
"timePeriod": "202301110000",
"count": 0
}, {
"timePeriod": "202301120000",
"count": 0
}, {
"timePeriod": "202301130000",
"count": 0
}, {
"timePeriod": "202301140000",
"count": 0
}, {
"timePeriod": "202301150000",
"count": 0
}, {
"timePeriod": "202301160000",
"count": 195 <--- ALSO A PARTIAL RESULT
},
{
"timePeriod": "202301170000",
"count": 682
},
>>>>> ...FINE FROM HERE ON <<<<<<
],
"totalCount": 10456,
"requestParameters": {
"bucket": "day",
"fromDate": "202301010000",
"toDate": "202301241720"
}
}
for your enjoyment here is a chart of what I'm (not) getting.
I'm a bit weirded out -- also, premium API access is all but free and is paid upfront, you know.
From further research and consultation on the Twitter dev fora, this is platform-wide. There may be possibly be a reindexing in a few days but as of now all searches return no results in the 2nd week of January 2023.
Related
I'm working on creating set of widgets for dashboard via code using
REST api https://learn.microsoft.com/en-us/rest/api/azure/devops/dashboard/widgets/create?view=azure-devops-server-rest-5.0
So, I was able to create the widgets programatically but struggling while configuring the widget to point to certain team and details.
Example: creating Burndown widget using Analytics extention (https://marketplace.visualstudio.com/items?itemName=ms.vss-analytics&ssr=false#overview).
I was able to create widget with contributionId and details.
However, can't set the settings to do aggregation of story points over certain sprints.
Any help regarding this is appreciated.
Here is the json passed in Create request. Need help on "settings" part.
{
"id": "",
"eTag": "7",
"name": "Burndown",
"position": {
"row": 1,
"column": 4
},
"size": {
"rowSpan": 2,
"columnSpan": 3
},
"settings": "",
"settingsVersion": {
"major": 1,
"minor": 0,
"patch": 0
},
"artifactId": "",
"url": "urlvalue",
"_links": {
"self": {
"href": "urlvalue"
},
"group": {
"href": "urlvalue"
},
"dashboard": {
"href": "urlvalue"
}
},
"isEnabled": true,
"contentUri": null,
"contributionId": "ms.vss-dashboards-web.Microsoft.VisualStudioOnline.Dashboards.BurndownWidget",
"typeId": "Microsoft.VisualStudioOnline.Dashboards.BurndownWidget",
"configurationContributionId": "ms.vss-dashboards-web.Microsoft.VisualStudioOnline.Dashboards.BurndownWidget.Configuration",
"configurationContributionRelativeId": "Microsoft.VisualStudioOnline.Dashboards.BurndownWidget.Configuration",
"isNameConfigurable": true,
"loadingImageUrl": "urlvalue",
"lightboxOptions": {
"width": 900,
"height": 700,
"resizable": true
}
}
Below are two samples about the settings parts.
1.Set the certain date as the end date.
"settings":"{
\"teams\":[{
\"projectId\":\"projectId\",
\"teamId\":\"teamId\"
}],
\"aggregation\":{
\"identifier\":0,
\"settings\":\"\"
},
\"completedWorkEnabled\":false,
\"fieldFilters\":[],
\"stackByWorkItemTypeEnabled\":false,
\"burndownTrendlineEnabled\":true,
\"workItemTypeFilter\":{
\"identifier\":\"BacklogCategory\",
\"settings\":\"Microsoft.RequirementCategory\"
},
\"includeBugsForRequirementCategory\":false,
\"timePeriodConfiguration\":{
\"startDate\":\"2019-10-14\",
\"samplingConfiguration\":{
\"identifier\":0,
\"settings\":{
\"endDate\":\"2019-10-15\",
\"lastDayOfWeek\":5,
\"sampleInterval\":0
}
}
},
\"totalScopeTrendlineEnabled\":true
}",
2. Set the certain Iteration as the end date.
"settings":"{
\"teams\":[{
\"projectId\":\"projectId\",
\"teamId\":\"teamId\"
}],
\"aggregation\":{
\"identifier\":0,
\"settings\":\"\"
}
\"completedWorkEnabled\":false,
\"fieldFilters\":[],
\"stackByWorkItemTypeEnabled\":false,
\"burndownTrendlineEnabled\":true,
\"workItemTypeFilter\":{
\"identifier\":\"BacklogCategory\",
\"settings\":\"Microsoft.RequirementCategory\"
},
\"includeBugsForRequirementCategory\":false,
\"timePeriodConfiguration\":{
\"startDate\":\"2019-10-14\",
\"samplingConfiguration\":{
\"identifier\":1,
\"settings\":[
\"iterationId\"
]
}
},
\"totalScopeTrendlineEnabled\":true
}",
For the Discovery REST api, the argument/parameter "return" controls which fields are returned.
So if I pass these arguments to the API
{
"query": named_sector,
"count": "10",
"filter": filter_dates,
"aggregation" : "term(docSentiment.type,count:3)"
}
my_query = discovery.query(my_disc_environment_id, my_disc_collection_id, qopts)
print(json.dumps(my_query, indent=2))
I get the following:
{
"matching_results": 14779,
"aggregations": [
{
"type": "term",
"field": "docSentiment.type",
"count": 3,
"results": [
{
"key": "positive",
"matching_results": 4212
},
{
"key": "negative",
"matching_results": 3259
},
{
"key": "neutral",
"matching_results": 152
}
]
}
],
"results": [
{
"id": "6389715fe7e7f711e0bc09d4f1236639",
"score": 1.3689895,
"yyyymm": "201704",
"url": "https://seekingalpha.com/article/4060446-valuation-dashboard-consumer-discretionary-update",
"enrichedTitle": null,
"host": "seekingalpha.com",
"text": "Valuation Dashboard: Consumer Discretionary - Update\n\nSummary\n\nValuation metrics in Consumer Discretionary.\n\nEvolution since last month.\n\nA list of stocks loo ....
and thousands of more lines. How do I restrict the output to the aggregations section? Is this an issue of me better handling the JSON structure that is returned?
thanks
If you change the count argument to 0, the returned JSON will only contain the aggregations.
Also, if you're using the Discovery web tooling, you can enter 0 for the "Number of results to return (Count)" field.
More details and an example can be found here: https://www.ibm.com/watson/developercloud/doc/discovery/using.html#building-aggregations
I have model Shop each has relation with Timetable which could contain something like:
shop_id: 1, day: 5, open_hour: 7, open_minutes: 0, close_hour: 13, close_minute: 30
shop_id: 1, day: 5, open_hour: 14, open_minutes: 30, close_hour: 18, close_minute: 00
of course Timetable could have more elegant format, but question is next: how with elasticsearch(tire) could I find Shop which is open?
all Idea will be apreciated! Thanks!
Found solution:
create separate index for each day (sunday, monday, ..)
for each day build full array of minutes from Timetable:
((open_hour * 60 + open_minute)..(close_hour * 60 + close_minute)).to_a
add filter to search:
filter :term, current_day_name => (current_hour * 60 + current_minutes)
this solution works as well, but it looks cumbersome, because if Shop works 8-h hours per day I have created array with size: 8 * 60 = 480 (which is converted to string as indexed field), so thats why this question is still open, and maybe someone will find better solution
Tire part for #Andrei Stefan answer:
indexes :open_hours, type: :nested do
indexes :open, type: 'integer'
indexes :close, type: 'integer'
end
open_hours_query = Tire::Search::Query.new do
filtered do
query { all }
filter :range, "open_hours.open" => { lte: current_time }
filter :range, "open_hours.close" => { gte: current_time }
end
end
filter :nested, { path: 'open_hours', query: open_hours_query.to_hash }
I would consider doing it like the following:
The opening and closing hours are integer values of an array of nested objects in Elasticsearch:
Example: shop opening at 07:00 and closing at 13:30 and then opening at 14:30 and closing at 18:00 in day 1 would be translated to this in ES:
"shop_name": "Shop 1",
"open_hours": [
{ "open": 420, "close": 810 },
{ "open": 870, "close": 1080 }
]
Each day in the week (1 -> 7) represents a value (to be added to the number of minutes):
Day 1 = addition 0
Day 2 = addition 2000
Day 3 = addition 4000
...
Day 7 = addition 10000
So, for each day there is an increment of 2000 because each day contains at most 1440 minutes (24 hours * 60 minutes) and to be able to differentiate one day from a single number these numbers don't have to intersect.
So, the example above with the shop opening at 07:00 would be translated for Day 4 for example to this:
"shop_name": "Shop 1",
"open_hours": [
{ "open": 6420, "close": 6810 },
{ "open": 6870, "close": 7080 }
]
When querying these documents, that point of the day you want to search needs to obey the same rules as above. For example, if you want to see if in Day 4 at 13:45 the "Shop 1" is opened you would search for a (6000 + 13*60 + 45 = 6825) minute.
The mapping for everything above in Elasticsearch would be this:
{
"mappings": {
"shop" : {
"properties": {
"shop_name" : { "type" : "string" },
"open_hours" : {
"type" : "nested",
"properties": {
"open" : { "type" : "integer" },
"close": { "type" : "integer" }
}
}
}
}
}
}
Test data:
POST /shops/shop/_bulk
{"index":{}}
{"shop_name":"Shop 1","open_hours":[{"open":420,"close":810},{"open":870,"close":1080}]}
{"index":{}}
{"shop_name":"Shop 2","open_hours":[{"open":0,"close":500},{"open":1000,"close":1440}]}
{"index":{}}
{"shop_name":"Shop 3","open_hours":[{"open":0,"close":10},{"open":70,"close":450},{"open":900,"close":1050}]}
{"index":{}}
{"shop_name":"Shop 4","open_hours":[{"open":2000,"close":2480}]}
{"index":{}}
{"shop_name":"Shop 5","open_hours":[{"open":2220,"close":2480},{"open":2580,"close":3000},{"open":3100,"close":3440}]}
{"index":{}}
{"shop_name":"Shop 6","open_hours":[{"open":6000,"close":6010},{"open":6700,"close":6900}]}
Querying for shops opened in Day 2 at minute #2400 of the day (06:40):
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "open_hours",
"query": {
"bool": {
"must": [
{
"filtered": {
"filter": {
"range": {
"open_hours.open": {
"lte": 2400
}}}}},
{
"filtered": {
"filter": {
"range": {
"open_hours.close": {
"gte": 2400
}}}}}
]
}}}}
]
}}}
Would output Shop 4 and Shop 5:
"shop_name": "Shop 4",
"open_hours": [
{
"open": 2000,
"close": 2480
}
]
"shop_name": "Shop 5",
"open_hours": [
{
"open": 2220,
"close": 2480
},
{
"open": 2580,
"close": 3000
},
{
"open": 3100,
"close": 3440
}
]
LATER EDIT: since Elasticsearch has come a looong way since I added this reply and many things changed since then, a filtered filter (in the context of the bool must I used) can be replaced by a bool filter or even a simple must. Also, the string doesn't exist in 6.x anymore, so you can use text if you somehow need to search by shop name using analyzers, or keyword ("shop_name" : { "type" : "text" },):
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "open_hours",
"query": {
"bool": {
"filter": [
{
"range": {
"open_hours.open": {
"lte": 2400
}
}
},
{
"range": {
"open_hours.close": {
"gte": 2400
}
}
}
]
}
}
}
}
]
}
}
}
I've been given this hash:
{
"item": {
"icon": "http://services.runescape.com/m=itemdb_rs/4332_obj_sprite.gif?id=4798",
"icon_large": "http://services.runescape.com/m=itemdb_rs/4332_obj_big.gif?id=4798",
"id": 4798,
"type": "Ammo",
"typeIcon": "http://www.runescape.com/img/categories/Ammo",
"name": "Adamant brutal",
"description": "Blunt adamantite arrow...ouch",
"current": {
"trend": "neutral",
"price": 227
},
"today": {
"trend": "neutral",
"price": 0
},
"day30": {
"trend": "positive",
"change": "+1.0%"
},
"day90": {
"trend": "positive",
"change": "+1.0%"
},
"day180": {
"trend": "positive",
"change": "+2.0%"
},
"members": "true"
}
}
I obtain the current price like this:
class GpperxpController < ApplicationController
def index
end
def cooking
require 'open-uri'
#sharkid = '385'
#sharkurl = "http://services.runescape.com/m=itemdb_rs/api/catalogue/detail.json?item=#{#sharkid}"
#sharkpage = Nokogiri::HTML(open(#sharkurl))
#sharkinfo = JSON.parse(#sharkpage.text)
#sharkinfo = #sharkinfo['item']['current']['price']
end
end
<%= #sharkinfo %> in my view returns 227. However, I want to perform some math operations on it, which is why I must use .to_i. Only problem is when I append .to_i, the value changes to 1. Why is that?
Price in the given json (http://services.runescape.com/m=itemdb_rs/api/catalogue/detail.json?item=385) contains ,.
... "current":{"trend":"neutral","price":"1,844"},...
^
Remove , before call String#to_i.
"1,844".to_i
# => 1
"1,844".gsub(',', '').to_i
# => 1844
Just running irb, and putting your JSON response in a variable, I had no problem getting the response to be 227, either by pulling the price out as text and then converting to an integer or by pulling the price out as an integer in one fell swoop.
So my initial code looked like:
json_text = '''
{
"item": {
"icon": "http://services.runescape.com/m=itemdb_rs/4332_obj_sprite.gif?id=4798",
"icon_large": "http://services.runescape.com/m=itemdb_rs/4332_obj_big.gif?id=4798",
"id": 4798,
"type": "Ammo",
"typeIcon": "http://www.runescape.com/img/categories/Ammo",
"name": "Adamant brutal",
"description": "Blunt adamantite arrow...ouch",
"current": {
"trend": "neutral",
"price": 227
},
"today": {
"trend": "neutral",
"price": 0
},
"day30": {
"trend": "positive",
"change": "+1.0%"
},
"day90": {
"trend": "positive",
"change": "+1.0%"
},
"day180": {
"trend": "positive",
"change": "+2.0%"
},
"members": "true"
}
'''
require 'json'
si = JSON.parse(json_text)
And then either of the following:
p = si['item']['current']['price']
price = p.to_i
or
price = si['item']['current']['price'].to_i
put the value of 227 in my price variable.
Something I would avoid if I were you though, is using the same variable name for different things. If what you want to have is the integer price in #sharkinfo, then you would do well to have a temporary name (without the # symbol) to put the price as text in, then assign the integer value to the desired variable.
Try this and see if it helps. I'll try to monitor this for a bit to see if you get anywhere. Also, at the point you pull the text out of the JSON, I believe this ceases to be a JSON problem any longer. Finally, you might include what version of ruby and what platform (Windows/Mac/Linux/etc) you are using.
I've been really struggling getting a simple scenario running with EmberJS and Rails.
Here's what I have (combined JS):
App = Ember.Application.create
LOG_TRANSITIONS: true
App.Post = DS.Model.extend
title: DS.attr 'string'
description: DS.attr 'string'
App.StreamRoute = Ember.Route.extend
setupController: (controller, model) ->
controller.set 'posts', model
model: -> #store.find('post')
App.Router.map ->
#.route 'stream', path: '/'
Here's the template content:
{{#each posts}}
{{title}}
{{/each}}
Here's the /posts JSON for Post.all (perhaps this is wrong?):
{
"posts": [
{
"posts": {
"created_at": "2013-08-15T23:48:54+01:00",
"description": "A few months ago I helped develop these posters for research that our UX team had gathered to create personas for our customers to show who they are and who actually uses our product.",
"id": 7,
"likes_count": 1,
"slug": "16ErQ",
"thumb": {
"url": "\/posts\/1\/16ErQ\/man.png",
"medium": {
"url": "\/posts\/1\/16ErQ\/medium_man.png"
}
},
"title": "Persona Project",
"updated_at": "2013-08-15T23:48:54+01:00",
"user_id": 1,
"views_count": 0
}
},
{
"posts": {
"created_at": "2013-08-16T15:47:03+01:00",
"description": "Just a little something.",
"id": 8,
"likes_count": 0,
"slug": "VYIvn",
"thumb": {
"url": "\/posts\/2\/VYIvn\/face.jpg",
"medium": {
"url": "\/posts\/2\/VYIvn\/medium_face.jpg"
}
},
"title": "Face",
"updated_at": "2013-08-16T15:47:03+01:00",
"user_id": 2,
"views_count": 0
}
},
{
"posts": {
"created_at": "2013-08-16T17:03:10+01:00",
"description": "Some people say, he's still running.",
"id": 9,
"likes_count": 2,
"slug": "hQBnt",
"thumb": {
"url": "\/posts\/1\/hQBnt\/run.jpg",
"medium": {
"url": "\/posts\/1\/hQBnt\/medium_run.jpg"
}
},
"title": "Run, Forest, run.",
"updated_at": "2013-08-23T23:44:19+01:00",
"user_id": 1,
"views_count": 0
}
}
]
}
I thought this would be fine, but it doesn't quite work, when I run it, I get 3 post results (which is how many there are) but the columns all contain null values: http://c.daryl.im/RTzO
As you can see, I also have that error. Any ideas?
Your JSON looks wrong. It should be something like this:
{
"posts":[
{
"created_at":"2013-08-15T23:48:54+01:00",
"description":"A few months ago I helped develop these posters for research that our UX team had gathered to create personas for our customers to show who they are and who actually uses our product.",
"id":7,
"likes_count":1,
"slug":"16ErQ",
"thumb":{
"url":"/posts/1/16ErQ/man.png",
"medium":{
"url":"/posts/1/16ErQ/medium_man.png"
}
},
"title":"Persona Project",
"updated_at":"2013-08-15T23:48:54+01:00",
"user_id":1,
"views_count":0
},
...
]}
Basically you have nested posts and need to remove one layer.