Rails - How to add pagination to Fastjson api? - ruby-on-rails

The default result of rendering FastJsonApi gem serialized_json like below:
render json: FlashcardSerializer.new(flashcards).serialized_json
would be something like this:
{
"data": [
{
"id": "1",
"type": "flashcard",
"attributes": {
"question": "why?",
"answer": "pretty good",
"slug": null
}
},
{
"id": "2",
"type": "flashcard",
"attributes": {
"question": "What is 0",
"answer": "it is 0",
"slug": null
}
}
]
}
I would rather add some extra information especially for pagination and I like the result to be something like this:
{
"data": [
{
"id": "1",
"type": "flashcard",
"attributes": {
"question": "why?",
"answer": "pretty good",
"slug": null
}
},
{
"id": "2",
"type": "flashcard",
"attributes": {
"question": "What is 0",
"answer": "it is 0",
"slug": null
}
},
"count":100,
"page":1,
]
}
I am aware of other available gems that manage pagination in API, and I know how to do it without Fastjson. The main issue here is that is there any way to get the aforementioned result from this gem without changing a lot in the code. Thanks

The desired document would be invalid according to the JSON API specification. You would need to include next and previous links in a link section. The current and total_count would belong in the meta section.
{
"data": [
{
"id": "1",
"type": "flashcard",
"attributes": {
"question": "why?",
"answer": "pretty good",
"slug": null
}
},
{
"id": "2",
"type": "flashcard",
"attributes": {
"question": "What is 0",
"answer": "it is 0",
"slug": null
}
},
]
"meta": {
"page": { "current": 1, "total": 100 }
},
"links": {
"prev": "/example-data?page[before]=yyy&page[size]=1",
"next": "/example-data?page[after]=yyy&page[size]=1"
},
}
Have a look at the JSON API specification before you continue designing the API.
You can pass these information into the serializer as an options argument
class FlashcardsController < ApplicationController
def index
render json: FlashcardSerializer.new(
flashcards, { links: {}, meta: { page: { current: 1 } }
).serialized_json
end
end
How you generate the data depends what you use to paginate.
If you design a new API, I would also recommend to use cursor based pagination rather than offset pagination because of it's limitations.
https://github.com/Netflix/fast_jsonapi#compound-document
https://github.com/Netflix/fast_jsonapi/blob/master/spec/lib/object_serializer_spec.rb#L8-L32

Related

How to differ Site Page and Wiki Page in MS Graph or SharePoint REST API

I created a SharePoint Page in SitePages library using the new SharePoint online experience. The page was created and I tried to fetch the page properties using MS Graph endpoint:
https://graph.microsoft.com/v1.0/drives/{drive-id}/root:/site_page.aspx?expand=listItem
The response I got:
{
...
"name": "site_page.aspx",
...
"file": {
"mimeType": "application/xml",
"hashes": {
"quickXorHash": ""
}
},
...
"listItem": {
...
"id": "4",
...
"contentType": {
"id": "0x0101009D1CB255DA76424F860D91F20E6C411800020BAE24978F3545AFD24007B325ACF9"
},
"fields": {
"FileLeafRef": "site_page.aspx",
"Title": "site_page",
"LinkTitle": "site_page",
"id": "4",
"ContentType": "Site Page",
...
}
}
}
Then I added Wiki Page content to this library to support Wiki pages in my site.
I created a new Wiki Page and fetched the page properties again using MS Graph endpoint:
https://graph.microsoft.com/v1.0/drives/{drive-id}/root:/wiki_page.aspx?expand=listItem
The response I got:
{
...
"name": "wiki_page.aspx",
...
"file": {
"mimeType": "application/xml"
},
...
"listItem": {
...
"id": "5",
...
"contentType": {
"id": "0x0101009D1CB255DA76424F860D91F20E6C411800020BAE24978F3545AFD24007B325ACF9"
},
"fields": {
"FileLeafRef": "wiki_page.aspx",
"Title": "wiki_page",
"LinkTitle": "wiki_page",
"id": "5",
"ContentType": "Site Page",
"WikiField" : "..."
...
}
}
}
As you can see both items have the same Content Type. How that is possible?
The only difference being that the wiki_page Item doesn't include 'hashes' property and does include 'WikiField' property (the place where the wiki page content is stored).
Is there another way to distinguish page types?
You could try this endpoint: GET /sites/{site-id}/lists/{list-id}/items?expand=fields(select= ContentType)
My test result:
{
"#odata.etag": "\"b3b04ace-40cd-4847-a3d8-678bc658216d,2\"",
"createdDateTime": "2020-04-22T05:34:06Z",
"eTag": "\"b3b04ace-40cd-4847-a3d8-678bc658216d,2\"",
"id": "11",
"lastModifiedDateTime": "2020-04-22T05:34:06Z",
"webUrl": "https://xxxx.sharepoint.com/sites/dev/SitePages/wikipage.aspx",
"createdBy": {
"user": {
"email": "amos#xxxx.onmicrosoft.com",
"id": "fc1e9add-6f9c-4b95-83e1-a022441681d7",
"displayName": "test"
}
},
"lastModifiedBy": {
"user": {
"email": "amos#xxxx.onmicrosoft.com",
"id": "fc1e9add-6f9c-4b95-83e1-a022441681d7",
"displayName": "test"
}
},
"parentReference": {
"id": "79e13173-d5ee-4a17-a081-5c94d148f905",
"siteId": "xxxx.sharepoint.com,b57886ef-4c2a-4d56-ad29-27266638ac3b,b62d1450-8e6f-4be7-84a3-f6600fd6cc14"
},
"contentType": {
"id": "0x01010800511BB12BD4FB664A89516226DBDDF1FB"
},
"fields#odata.context": "https://graph.microsoft.com/v1.0/$metadata#sites('siteid')/lists('63307f0b-bc1c-4372-bfea-6352ed57a0ff')/items('11')/fields/$entity",
"fields": {
"#odata.etag": "\"b3b04ace-40cd-4847-a3d8-678bc658216d,2\"",
"ContentType": "Wiki Page"
}
},
{
"#odata.etag": "\"70f12845-7646-4b2b-85bd-4a8074c105a0,1\"",
"createdDateTime": "2020-04-22T05:35:54Z",
"eTag": "\"70f12845-7646-4b2b-85bd-4a8074c105a0,1\"",
"id": "12",
"lastModifiedDateTime": "2020-04-22T05:35:54Z",
"webUrl": "https://xxxx.sharepoint.com/sites/dev/SitePages/webpartpage.aspx",
"createdBy": {
"user": {
"email": "amos#xxxx.onmicrosoft.com",
"id": "fc1e9add-6f9c-4b95-83e1-a022441681d7",
"displayName": "test"
}
},
"lastModifiedBy": {
"user": {
"email": "amos#xxxx.onmicrosoft.com",
"id": "fc1e9add-6f9c-4b95-83e1-a022441681d7",
"displayName": "test"
}
},
"parentReference": {
"id": "79e13173-d5ee-4a17-a081-5c94d148f905",
"siteId": "xxxx.sharepoint.com,b57886ef-4c2a-4d56-ad29-27266638ac3b,b62d1450-8e6f-4be7-84a3-f6600fd6cc14"
},
"contentType": {
"id": "0x0101090100FAC6DAD225005749BE7D6124B50B156E"
},
"fields#odata.context": "https://graph.microsoft.com/v1.0/$metadata#sites('siteid')/lists('63307f0b-bc1c-4372-bfea-6352ed57a0ff')/items('12')/fields/$entity",
"fields": {
"#odata.etag": "\"70f12845-7646-4b2b-85bd-4a8074c105a0,1\"",
"ContentType": "Web Part Page"
}
}
You'd better check the type of your page.
Updatded:
"fields#odata.context": "https://graph.microsoft.com/v1.0/$metadata#sites('siteid')/lists('63307f0b-bc1c-4372-bfea-6352ed57a0ff')/items('13')/fields/$entity"
"fields": {
"#odata.etag": "\"18361706-416b-4a71-8d31-bce87c1a57e4,3\"",
"ContentType": "Site Page"
}

Defining Array of Objects in Swagger Documentation

I'm using swagger for quite a bit now, we have started documenting our code using it, in one place there's an API response which returns multiple objects in the included block.
Example:
{
"data": {
"id": "1",
"type": "schoolPositions",
"attributes": {
"description": "teases the students",
"mustHaves": "principle"
},
"relationships": {
"schoolLocation": {
"data": {
"id": "72",
"type": "schoolLocations"
}
},
"schoolCompensation": {
"data": {
"id": "75",
"type": "schoolCompensations"
}
},
"jobSpecs": {
"data": [
{
"id": "82",
"type": "schoolAttachments"
}
]
}
}
},
"included": [
{
"id": "72",
"type": "schoolLocations",
"attributes": {
"city": "Berhampore",
"state": "West Bengal",
"postalCode": "742101",
"country": "India",
"globalRegionId": 30,
"regionId": 683
}
},
{
"id": "75",
"type": "schoolCompensations",
"attributes": {
"salary": "",
"bonus": "",
"equity": "",
"currencyId": null,
"equityType": "percent",
"salaryDescription": null
}
},
{
"id": "82",
"type": "schoolAttachments",
"attributes": {
"attachmentType": "JobSpecificationAttachmentType",
"fileFileName": "vs.jpg",
"fileContentType": "image/jpeg",
"fileFileSize": 2410039,
"fileUpdatedAt": "2018-12-12T07:06:38Z",
"downloadUrl": "001-vs.jpg?1544598398",
"klass": "SchoolAttachments"
}
}
]
I have wasted an entire day on the internet and documentation trying to document the included part, but I'm going wrong somewhere
response 200 do
key :description, 'School Data'
schema do
property :data do
key :type, :array
items do
key :'$ref', :School
end
end
property :included do
key :type, :array
items do
key :'$ref', :SchoolLocations
key :'$ref', :SchoolCompensations
key :'$ref', :SchoolAttachments
end
end
end
end
This shows only the SchoolAttachments in the included part.
I have tried using allOff but it doesn't work.

Creating an avro schema for an array with multiple record types?

I am creating an avro schema for a JSON payload that appear to have an array of multiple objects. I'm not sure exactly how to represent this in the schema. The key in question is content:
{
"id": "channel-id",
"name": "My Channel with a New Title",
"description": "Herpy me derpy merpus herpsum ner berp berps derp ter tee",
"privacyLevel": "<private|org>",
"planId": "some-plan-id",
"owner": "a-user-handle",
"curators": [
"user-handle-1",
"user-handle-2"
],
"members": 5,
"content": [
{
"id": "docker",
"slug": "docker",
"index": 1,
"type": "path"
},
{
"id": "such-linkage",
"slug": "such-linkage",
"index": 2,
"type": "external-link",
"details": {
"url": "http://some-dank-link.com",
"title": "My Dank Link",
"contentType": "External Link",
"level": "Beginner",
"duration": "PT34293H33M9S"
}
},
{
"id": "21f1e812-b10a-40df-8b52-3a1d05fc215c",
"slug": "windows-azure-storage-in-depth",
"index": 3,
"type": "course"
},
{
"id": "7c346c05-6416-42dd-80b2-d5e758de7926",
"slug": "7c346c05-6416-42dd-80b2-d5e758de7926",
"index": 4,
"type": "project"
}
],
"imageUrls": ["https://url/to/an/image", "https://url/to/another/image"],
"analyticsEnabled": true,
"orgDiscoverable": false,
"createdDate": "2015-12-31T01:23:45+00:00",
"archiveDate": "2015-12-31T01:23:45+00:00",
"messagePublishedAt": "2015-12-31T01:23:45+00:00"
}
If you are asking if it is possible create an array with different kind of records, it is. Avro support this through union. it would looks like .
{
"name": "myRecord",
"type":"record",
"fields":[
{
"name":"myArrayWithMultiplesTypes",
"type":{
"type": "array",
"items":[
{
"name":"typeOne",
"type":"record",
"fields":[
{"name":"name", "type":"string"}
]
},
{
"name":"typeTwo",
"type":"record",
"fields":[
{"name":"id", "type":"int"}
]
}
]
}
}
]
}
If you already have the records defined previously, then it could look like this:
{
"name": "mulitplePossibleTypes",
"type": [
"null",
{
"type": "array",
"items": [
"com.xyz.kola.cloud.events.itemmanager.Part",
"com.xyz.kola.cloud.events.itemmanager.Document",
"com.xyz.kola.cloud.events.itemmanager.DigitalModel",
"com.xyz.kola.cloud.events.itemmanager.Interface"
]
}
]
},

How to make Elasticsearch sort/prefer hits with exactly matching strings first

I'm using default analyzers and indexing. So let's say I have this simple mapping:
"question": {
"properties": {
"title": {
"type": "string"
},
"answer": {
"properties": {
"text": {
"type": "string"
}
}
}
}
}
(that was an example. sorry if it has typos)
Now, I perform the following search.
GET _search
{
"query": {
"query_string": {
"query": "yes correct",
"fields": ["answer.text"]
}
}
}
The results will score a text value like "yes correct." (doc id value 1) higher than simply "yes correct" (without a period, doc id value 181). Both hits have the same score value, but the hits array lists the one with the smaller doc id first. I understand that the default index option includes sorting by doc id, so how do I exclude that one attribute and still use the rest of the default options?
I'm not setting any custom analyzers, so everything is using default values for Elasticsearch 2.0.
This is probably a use case for Dis Max Query
A query that generates the union of documents produced by its
subqueries, and that scores each document with the maximum score for
that document as produced by any subquery, plus a tie breaking
increment for any additional matching subqueries.
So following that, you need to make your answer score as an exact match and give it highest boost. You'll have to use a custom analyzer for that. That'd be your mappings:
PUT /test
{
"settings": {
"analysis": {
"analyzer": {
"my_keyword": {
"type": "custom",
"tokenizer": "keyword",
"filter": [
"asciifolding",
"lowercase"
]
}
}
}
},
"mappings": {
"question": {
"properties": {
"title": {
"type": "string"
},
"answer": {
"type": "object",
"properties": {
"text": {
"type": "string",
"analyzer": "my_keyword",
"fields": {
"stemmed": {
"type": "string",
"analyzer": "standard"
}
}
}
}
}
}
}
}
}
Your test data:
PUT /test/question/1
{
"title": "title nr1",
"answer": [
{
"text": "yes correct."
}
]
}
PUT /test/question/2
{
"title": "title nr2",
"answer": [
{
"text": "yes correct"
}
]
}
Now when you're querying for "yes correct." using such query:
POST /test/_search
{
"query": {
"dis_max": {
"tie_breaker": 0.7,
"boost": 1.2,
"queries": [
{
"match": {
"answer.text": {
"query": "yes correct.",
"type": "phrase"
}
}
},
{
"match": {
"answer.text.stemmed": {
"query": "yes correct.",
"operator": "and"
}
}
}
]
}
}
}
You get this output:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0.37919715,
"hits": [
{
"_index": "test",
"_type": "question",
"_id": "1",
"_score": 0.37919715,
"_source": {
"title": "title nr1",
"answer": [
{
"text": "yes correct."
}
]
}
},
{
"_index": "test",
"_type": "question",
"_id": "2",
"_score": 0.11261705,
"_source": {
"title": "title nr2",
"answer": [
{
"text": "yes correct"
}
]
}
}
]
}
}
If you run very same query without trailing dot, which then becomes "yes correct", you're getting this result:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0.37919715,
"hits": [
{
"_index": "test",
"_type": "question",
"_id": "2",
"_score": 0.37919715,
"_source": {
"title": "title nr2",
"answer": [
{
"text": "yes correct"
}
]
}
},
{
"_index": "test",
"_type": "question",
"_id": "1",
"_score": 0.11261705,
"_source": {
"title": "title nr1",
"answer": [
{
"text": "yes correct."
}
]
}
}
]
}
}
Hopefully this is what you're looking for.
By the way, I'd recommend to always use Match query when performing text search. Taken from documentation:
Comparison to query_string / field The match family of queries
does not go through a "query parsing" process. It does not support
field name prefixes, wildcard characters, or other "advanced"
features. For this reason, chances of it failing are very small / non
existent, and it provides an excellent behavior when it comes to just
analyze and run that text as a query behavior (which is usually what a
text search box does). Also, the phrase_prefix type can provide a
great "as you type" behavior to automatically load search results.
Elasticsearch or rather Lucene scoring does not take into account the relative positioning of the tokens. It utlizes 3 different criterias to do the same
Term frequency - Frequency at which the search terms is present in
the document
Inverse document frequency - Number of occurrence of the search term
in the entire database. The more the occurance , the more the common
is the search term and less the importance it has in search
Field length normalization - Number of tokens present in the target
field.
You can learn more about it here.

SurveyMonkey: Where is the text of my respondent's response?

I'm new to the SurveyMonkey API and it hasn't been too difficult to get payloads back from API calls, but right now I'm trying to get back what responses a specific respondent gave.
I have a survey which has two respondents, the first question on the survey asks the user to enter three pieces of information: Their Name, an ID and today's date.
So, if I do a call to get_survey_details, I can see the questions just fine. For example
obj.pages[0].questions[0].answers[0].answerid: "xxxxxxxx" //some long ID
obj.pages[0].questions[0].answers[0].text: "Enter Your Name"
obj.pages[0].questions[0].answers[0].type: "row"
There's a couple more pieces of information in that object, like whether the question is visible, etc., but these seem to be the pertinent pieces to the question I have.
So! I make another call to get_responses using the same survey_id and respondent_id (there's only two so actually I get them both).
In the resulting payload I get an array of 2 objects (one to hold each respondents responses). So I look in the first (obj[0]) and I see an array of questions and the respondent id. Fine. I look in the questions array and I see one object for each question and in each of those an answers object.
so that's:
obj[0].questions[0].answers[0].col: "yyyyyy" //some long ID
obj[0].questions[0].answers[0].row: "nnnnnn" //some other long ID
No response text. just this row/col business.
At this point, I'm super-confused (which is like regular confused, but with a cape). Where the heck are the respondents actual responses?
What the heck does "row" and "column" reference? Do I have to do some other API call with the row and/or column in order to get the text of the respondent's response?
I've looked through the documentation (and will continue to do so after posting this) and through stackoverflow to see if anyone else has asked this before. There was one question that came close, but really they were just forgetting to pair 'get_responses' with 'get_survey_details'. I'm doing that, but am still lost as ever. And I don't see any documentation really explaining in detail how this row/column concept works for mapping responses to the text of the response. :/
I know this is a really long-winded question, but I'm just so confused as to how to actually get responses out of this API. :(
Thanks for reading.
The text for a given response should come through under the "text" key. e.g. for a survey that only consists of an essay style question:
{
"status": 0,
"data": [
{
"respondent_id": "123456",
"questions": [
{
"answers": [
{
"text": "This is an essay style answer.",
"row": "0"
}
],
"question_id": "78910"
}
]
}
]
}
"row" and "col" literally reference the row and column of an answer - e.g. in a matrix question, there will be a list of rows for different questions ("what did you think of the hotel?") and ratings ("bad, okay, great") - and each answer is a combination of these. For a regular multiple choice question there will be multiple rows and only one column.
Calling "get_responses" with the correct respondent_id should provide you with the text response that you want. It's only the fixed details of the answer stored in the survey itself you should have to look up (provided in get_survey_details).
Using GET : /surveys/{survey_id}/details, we can get the corresponding question Ids along with the answer Ids.
{
"pages": [
{
"href": "https://api.surveymonkey.net/v3/surveys/87263608/pages/260492760",
"description": "",
"questions": [
{
"sorting": null,
"family": "matrix",
"subtype": "rating",
"required": {
"text": "This question requires an answer.",
"amount": "0",
"type": "all"
},
"answers": {
"rows": [
{
"visible": true,
"text": "",
"position": 1,
"id": "10788526669"
}
],
"choices": [
{
"description": "Not at all likely",
"weight": -100,
"id": "10788526670",
"visible": true,
"is_na": false,
"text": "Not at all likely - 0",
"position": 1
},
{
"description": "",
"weight": -100,
"id": "10788526671",
"visible": true,
"is_na": false,
"text": "1",
"position": 2
},
{
"description": "",
"weight": -100,
"id": "10788526672",
"visible": true,
"is_na": false,
"text": "2",
"position": 3
},
{
"description": "",
"weight": -100,
"id": "10788526673",
"visible": true,
"is_na": false,
"text": "3",
"position": 4
},
{
"description": "",
"weight": -100,
"id": "10788526674",
"visible": true,
"is_na": false,
"text": "4",
"position": 5
},
{
"description": "",
"weight": -100,
"id": "10788526675",
"visible": true,
"is_na": false,
"text": "5",
"position": 6
},
{
"description": "",
"weight": -100,
"id": "10788526676",
"visible": true,
"is_na": false,
"text": "6",
"position": 7
},
{
"description": "",
"weight": 0,
"id": "10788526677",
"visible": true,
"is_na": false,
"text": "7",
"position": 8
},
{
"description": "",
"weight": 0,
"id": "10788526678",
"visible": true,
"is_na": false,
"text": "8",
"position": 9
},
{
"description": "",
"weight": 100,
"id": "10788526679",
"visible": true,
"is_na": false,
"text": "9",
"position": 10
},
{
"description": "Extremely likely",
"weight": 100,
"id": "10788526680",
"visible": true,
"is_na": false,
"text": "Extremely likely - 10",
"position": 11
}
]
},
"visible": true,
"href": "https://api.surveymonkey.net/v3/surveys/87263608/pages/260492760/questions/1044924866",
"headings": [
{
"heading": "How likely is it that you would recommend XYZ to a friend or colleague?"
}
],
"position": 1,
"validation": null,
"id": "1044924866",
"forced_ranking": false
},
{
"sorting": null,
"family": "single_choice",
"subtype": "vertical",
"required": null,
"answers": {
"choices": [
{
"visible": true,
"text": "High Interest",
"position": 1,
"id": "10788529403"
},
{
"visible": true,
"text": "Long process",
"position": 2,
"id": "10788529404"
},
{
"visible": true,
"text": "Low XYZ Amount",
"position": 3,
"id": "10788529405"
},
{
"visible": true,
"text": "Lot of Documents",
"position": 4,
"id": "10788529406"
},
{
"visible": true,
"text": "Bad customer service",
"position": 5,
"id": "10788529407"
}
]
},
"visible": true,
"href": "https://api.surveymonkey.net/v3/surveys/87263608/pages/260492760/questions/1044925207",
"headings": [
{
"heading": "What is the most important issue which we need to address for overall a better service?"
}
],
"position": 2,
"validation": null,
"id": "1044925207",
"forced_ranking": false
}
],
"title": "",
"position": 1,
"id": "260492760",
"question_count": 2
}
],
}
We can use these ids to decipher the answer we get after fetching responses using get response API(Bulk or each respondent).
For eg:,
If my survey has two questions, like
Then after fetching the responses we get a json like this:
{
"total_time": 34,
"href": "https://api.surveymonkey.net/v3/collectors/94630092/responses/5120000552",
"custom_variables": {},
"ip_address": "182.76.20.30",
"id": "5120000552",
"logic_path": {},
"date_modified": "2016-12-01T11:01:11+00:00",
"response_status": "completed",
"custom_value": "LAI100023",
"analyze_url": "http://www.surveymonkey.com/analyze/browse/EvaBWWcU9K1XTH_2FFFBTfFul4ge94MwVWvBk0eAFDJ3c_3D?respondent_id=5120000552",
"pages": [
{
"id": "260492760",
"questions": [
{
"id": "1044924866",
"answers": [
{
"choice_id": "10788526677",
"row_id": "10788526669"
}
]
},
{
"id": "1044925207",
"answers": [
{
"choice_id": "10788529404"
}
]
}
]
}
],
"page_path": [],
"recipient_id": "2743199128",
"collector_id": "94630092",
"date_created": "2016-12-01T11:00:37+00:00",
"survey_id": "87263608",
"collection_mode": "default",
"edit_url": "http://www.surveymonkey.com/r/?sm=SfTljxZSoBFvaRUeGSI6L813qctjfG_2FDCVcqCks7CDc4TcJC_2BNHqmPYD7NNTcvST",
"metadata": {
"contact": {
"first_name": {
"type": "string",
"value": "John"
},
"last_name": {
"type": "string",
"value": "Doe"
},
"email": {
"type": "string",
"value": "neeta#xyz.com"
}
}
}
}
We can map the questions and answers using their IDs in this response with the ids we got from survey details. For open ended text questions, we get direct typed responses.

Resources