I am processing messages. Each message corresponds to a coordinate, and MOST of the coordinates are in a specific JSON form. Problem is, some naughty messages have the wrong format (a different format, where a whole list of coordinates comes in one message).
Ideally, I would like to process both types of message, but the built-in Parse function for JsonProvider seems to succeed even when the incoming data is in the wrong format. How can I know in advance (before attempting to parse) what format my datum is in?
My two data models look something like this:
type ModelA = JsonProvider<"""{
"id": "13222",
"timestamp": 1499329186332.0,
"latitude": 12.125419,
"longitude": 15.054884
}""">
type ModelB = JsonProvider<"""{
"data": {
"positions": [
{
"id": "13223",
"timestamp": 1499329186332.0,
"latitude": 12.125419,
"longitude": 15.054884
},
{
"id": "13223",
"timestamp": 1499329186332.0,
"latitude": 12.125419,
"longitude": 15.054884
}]
}
}""">
Related
Cosider an API which reply is always of this structure:
{
"pagination": {
"limit": int,
"offset": int,
"count": int,
"total": int
},
"data": [
{...some obj...}
]
}
So payloads differ only in structure of data objects.
Ideally I'd like to tell F# that all types built from samples have some common part - pagination info, so I can have one generic method which reads all pages.
Is it possible, or do I have to extract pagination object and data array separately with two type providers? I see the benefit of having one provider per response body as it supports reading data from the stream.
I would define two different provided types, one for parsing the pagination data and one for parsing the actual data, i.e. something like this:
type Pagination = JsonProvider<"""{
"pagination": { "limit": 1, "offset": 2,
"count": 3, "total": 4 }
}""">
type OneDataType = JsonProvider<"""{
"data": [ {"a": 1} ]
}""">
If you want to avoid parsing the same JSON file twice (e.g. by calling Pagination.Parse and OneDataType.Parse on the same string), you can actually just parse the data once and then pass the parsed JsonValue to the other type:
let odt = OneDataType.Load("/some/file")
let pg = Pagination.Root(odt.JsonValue)
pg.Pagination.Count
If you wanted to do this with a single provided type, then you could define multiple different fields for the multiple different types of data - but you'd have to name those differently. You'd then need to do some fiddling to read the data correctly. I would not do this, because I find it confusing, but it would look something like this:
type AnyData = JsonProvider<"""{
"pagination": { "limit": 1, "offset": 2,
"count": 3, "total": 4 },
"data": [],
"one_data_type": [ {"a":1} ],
"another_data_type": [ {"b":"xx" }]
}""">
let a = AnyData.Load("/some/file")
// Access pagination data
a.Pagination
// Access data as if it was OneDataType
let oneData = [| for d in a.Data ->
AnyData.OneDataType(d.JsonValue) |]
I'm using the FSharp.Data.TypeProvider but I have an optional property that is not rendered as such.
I have this example JSON:
[
{
"error": {
"message": "Error validating access token: Session has expired on Friday, 24-Jul-20 16:00:00 PDT. The current time is Friday, 24-Jul-20 16:06:14 PDT.",
"type": "OAuthException",
"code": 190,
"error_subcode": 463,
"fbtrace_id": "A0yQG8l8ggauD5PMbYSnEyA"
}
},
{
"error": {
"message": "Error validating access token: Session has expired on Friday, 24-Jul-20 16:00:00 PDT. The current time is Friday, 24-Jul-20 16:06:14 PDT.",
"type": "OAuthException",
"code": 190,
"error_subcode": 463,
"fbtrace_id": "A0yQG8l8ggauD5PMbYSnEyA"
}
},
{
"data": [
{
"id": "17841511906082530"
}
]
}
]
The field error is read as Optional<Error> meanwhile the field data is Datum [], It should be Optional<Data>, this error seems to have something to do with the array, if I change the value of data to another type, like a number, it correctly infers the type.
Maybe is some cache that's wrong? How could I reset the TypeProvider cache?
OR am I doing something else wrong?
This issue is also reported in the github project: https://github.com/fsharp/FSharp.Data/issues/1322
That's expected behavior.
In your JSON string, data is an array, an empty array indicates None, thus the field Data is read as array. That makes sense.
The fact that each element of that array belongs to type Datum also makes sense because Datum is the single form of Data (plural form). Indeed, if you rename data to dogs you will see it is read as Dog [].
Update: I don’t think we can tell the type provider to infer the field data as Option<Datum[]>. To check whether it is actually an empty array or not existed, we need to examine the JsonValue field:
match node.JsonValue with
| JsonValue.Record [|("data", _)|] ->
// yep, we have field data here
| _ ->
// nope, field data is missed
This piece of code:
open FSharp.Data
type ColorProvider = JsonProvider<"""
[
{
"color": "Red",
"code": 15
},
{
"color": "Green"
}
]
""", SampleIsList=true>
let value = ColorProvider.Root(color = "Blue", code = None)
printf "%A" value
Produces this JSON:
{
"color": "Blue",
"code": null
}
I'm passing the json to an external service which doesn't handle nulls. Either it must be an integer or the whole property must be missing.
Is there a way to hide the code property when it's null?
You are using FSharp's object printing feature for generating JSON. It produces JSON-like output, but in my opinion it is not meant for that task. I'd suggest using a JSON library for that, personally I like using https://github.com/Microsoft/fsharplu/wiki/fsharplu.json for that task, as it produces beautiful JSON for Discriminate Unions.
I have following REST API response:
"items":
[
{
"empid": "1234",
"name": "Santosh",
"hiredby": "Mark",
"date": "2017-01-31,00:19:41 PST",
},
{
"empid": "5678",
"name": "Kumar",
"hiredby": "Bob",
"date": "2017-01-31,08:30:31 PST"
}
]
My query is : - How do i get empid based on querying name as Kumar.
For example: I need to find "Kumar" name and get his empid. (that is, search by name and get his empid as response) I'm able to get the response and store it in Response object. but, from response object how can i traverse and query to get the required value.
Also,
I tried by retrieving as:
String name = get(REST_ENDPOINT).then().body("items.name",hasItems("Kumar")).extract().path("items.empid").toString();
when i print the response i get collection of the empid like [1234,5678], where as my expectation is to get only 5678.
Do I need to parse via JSONArray and JSONObject and iterate the response?
Please suggest.
You can use something like this
response1.jsonPath().getList("collect { it.credentials.findAll { it.credentialType == 'Ban User Name'}.credentialId }.flatten()")
I am using Parse and trying to upload a csv file which contains GeoPoints, but when I try to upload them the only options I see is to label it as a string, number, or boolean. Is it possible to import Geopoints or files?
Here's how to do it using JSON import:
{ "results": [
{
"name":"Kfar Saba",
"location": {"__type": "GeoPoint", "latitude": 32.18406, "longitude": 34.91731}
},
{
"name":"Ar'ara BaNegev",
"location": {"__type": "GeoPoint", "latitude": 32.5, "longitude": 35.1}
}
}]
To import GeoPoints, you'll need to use JSON as your file format. See https://parse.com/docs/data#importing for details.