I am trying to use the Json type provider to create types and iterate over a list. But, I also need to access the serialised json of each item.
Here is what I have
#r "nuget: FSharp.Data"
#r "nuget: System.Text.Json"
open FSharp.Data
open System.Text.Json
type Users = JsonProvider<"""{"users":[{"id":1, "name": "John"}, {"id":2, "name": "Paul"}]}""">
let data = Users.GetSample().Users
let printProduct (id:int, userJson: string) =
printfn "%i %s" id userJson
let returnIdandJson() =
data |> Seq.map(fun x-> (printProduct (x.Id, JsonSerializer.Serialize x)))
returnIdandJson()
This prints
1 {"JsonValue":{"Tag":3,"IsString":false,"IsNumber":false,"IsFloat":false,"IsRecord":true,"IsArray":false,"IsBoolean":false,"IsNull":false,"_Pri:false,"IsNull":false,"_Print":"{\r\n \u0022id\u0022: 1,\r\n \u0022name\u0022: \u0022John\u0022\r\n}"}}
2 {"JsonValue":{"Tag":3,"IsString":false,"IsNumber":false,"IsFloat":false,"IsRecord":true,"IsArray":false,"IsBoolean":false,"IsNull":false,"_Pri:false,"IsNull":false,"_Print":"{\r\n \u0022id\u0022: 2,\r\n \u0022name\u0022: \u0022Paul\u0022\r\n}"}}
What I want is
1 {"id":1, "name": "John"}
2 {"id":2, "name": "Paul"}
Is there a way I can access and serialise the inner list items from JsonValue?
Thanks
You can get access to the original JSON via the .JsonValue property that exists on every object returned by the provider. This property returns type JsonValue, which you can programmatically analyze if you wanted (e.g. calling its Properties() method or some such), and it also has a handly ToString implementation that just returns the serialized JSON:
let printProduct (id:int, userJson: string) =
printfn "%i %s" id userJson
let returnIdandJson() =
data |> Seq.map(fun x-> (printProduct (x.Id, string x.JsonValue)))
Related
Trying to learn F# and got stuck when trying to find a better approach of converting a csv file to a json array where each row + header is a json object in that array.
After some trial and error I finally caved and went for an ugly approach with mutable list and map. Are there any better ways this can be implemented?
let csvFileToJsonList (csvFile: FSharp.Data.CsvFile) =
let mutable tempList = List.empty<Map<string,string>>
let heads =
match csvFile.Headers with
| Some h -> h
| None -> [|"Missing"|] // what to do here?
let nbrOfColumns = csvFile.NumberOfColumns
for row in csvFile.Rows do
let columns = row.Columns
let mutable tempMap = Map.empty<string,string>
for i = 0 to nbrOfColumns-1 do
tempMap <- tempMap.Add(heads.[i], columns.[i])
tempList <- tempMap :: tempList
System.Text.Json.JsonSerializer.Serialize(tempList)
This outputs the following which is the goal:
[
{
"Header1": "Row1Val1",
"Header2": "Row1Val2",
"Header3": "Row1Val3",
"Header4": "Row1Val4",
"Header5": "Row1Val5"
},
{
"Header1": "Row2Val1",
"Header2": "Row2Val2",
"Header3": "Row2Val3",
"Header4": "Row2Val4",
"Header5": "Row2Val5"
}
]
This is about as simple as I could make it, although a longer version might be more readable for you:
let csvFileToJsonList (csvFile: FSharp.Data.CsvFile) =
let heads = csvFile.Headers |> Option.defaultValue [||]
csvFile.Rows
|> Seq.map (fun row -> Seq.zip heads row.Columns |> Map)
|> System.Text.Json.JsonSerializer.Serialize
This produces the output in the original order, which I'm assuming is preferable (your solution reverses the order).
This also assumes some headers exist, otherwise the output will be empty objects.
Description: For each row use Seq.zip to produce a sequence of header-value tuples. Pass that to the Map constructor to create a map, providing a sequence of maps, which can be serialized.
Note that using dict instead of Map might be a bit faster.
You also could use CsvProvider to create a typed object (Row)
open FSharp.Data
type Persons =
CsvProvider<"David,Raab,19.02.1983",
Schema="First (string), Last (string), BirthDay(string)",
HasHeaders=true>
let parseCsv (reader:System.IO.TextReader) = [
let data = Persons.Load reader
for row in data.Rows do
Map [
("First", row.First)
("Last", row.Last)
("Birthday", row.BirthDay)
]
]
Returning a List of a map instead of Json, but i guess you will know how to change it to Serialze the data.
#r "nuget: Microsoft.OData.Edm"
open Microsoft.OData.Edm
open System.Xml
let reader = XmlReader.Create("metadata.xml")
let success, edmModel, errors = Csdl.CsdlReader.TryParse(reader)
let container = edmModel.EntityContainer
let elements = edmModel.SchemaElements |> Seq.cast<IEdmSchemaElement> |> Array.ofSeq
for elem in elements do
printfn $"1 {elem.Name} : {elem.SchemaElementKind} : {elem.Location()}"
let entitySets = container.EntitySets() |> Seq.cast<IEdmEntitySet> |> Array.ofSeq
for elem in entitySets do
let cElems = elem.Container.Elements |> Seq.cast<IEdmEntityContainerElement> |> Array.ofSeq
printfn $" 2 {elem.Name} : {elem.ContainerElementKind}"
for celem in cElems do
printfn $" 3 {celem.Name} : {celem.ContainerElementKind}"
This is the extent to which I can walk an odata entity model graph with Microsoft.OData.Edm.
The results of the level 2 and 3 prints are the same. I want to access the keys and properties of the entity types.
I can always switch to walking the XML graph, but using a maintained odata library seems like the right thing to do.
I think the trick here is to downcast the elements to IEdmEntityType where possible:
let entTypes =
edmModel.SchemaElements
|> Seq.choose (function
| :? IEdmEntityType as entType -> Some entType
| _ -> None)
for entType in entTypes do
printfn "%s" entType.Name
for prop in entType.DeclaredProperties do
printfn " %s %s" prop.Name (prop.Type.ShortQualifiedName())
Output will look something like:
Product
ID Int32
Name String
Description String
ReleaseDate DateTimeOffset
DiscontinuedDate DateTimeOffset
Rating Int16
Price Double
Categories
Supplier ODataDemo.Supplier
ProductDetail ODataDemo.ProductDetail
FeaturedProduct
Advertisement ODataDemo.Advertisement
ProductDetail
ProductID Int32
Details String
Product ODataDemo.Product
I am very new to F#, and I'm trying to simply get an array of structs where the struct is the filename and the filecontents is an array of lines in the file. I have an error I don't really understand on the indicated line, and I haven't been able to find the correct syntax or approach to do this.
let readFileContents filePath =
File.ReadAllLines(filePath)
let makeFileStruct fileName =
new FileContents(fileName, fileName |> readFileContents)
let fileTemplates path =
Directory.GetFiles(path, "*.template")
|> Array.map(fun x -> Path.GetFileName(x))
|> Array.iter(fun x -> makeFileStruct(x)) <--- error: This expression was expected to have type 'unit'
[<EntryPoint>]
let main argv =
printfn "Testing getting files"
argv.[0]
|> fileTemplates
|> JsonConvert.SerializeObject
|> printfn "some stuff %s"
0
I'm trying to simply get an array of structs where the struct is the filename
I don't think this is a good idea becuase in order to get the predictable memory usage of a struct you'd need some combination of a very strict filename length limit and/or a very memory-inefficient struct. Unless the struct contains an non-struct object.
I have an error I don't really understand on the indicated line
Array.iter executes a function for each element of the array. So the inner function makeFileStruct would need to return a unit in order to execute it. You are looking for Array.map which creates an array from the outputs.
It is my first question on SO...so do not judge strictly =)
Usually all my questions techout in chat rooms (believe me, a lot of them =)).
Recently, we are talking about the RosettaCode. And I wondered to complement some of the tasks code to F#
One of them is JSON.
One of the possible solutions is the use of "F# Data: JSON Parser". So my question is linked with it.
This code works well:
open FSharp.Data
open FSharp.Data.JsonExtensions
type Person = {ID: int; Name:string}
let json = """[ { "ID": 1, "Name": "First" }, { "ID": 2, "Name": "Second" }]"""
json |> printfn "%s"
match JsonValue.Parse(json) with
| JsonValue.Array(x) ->
x |> Array.map(fun x -> {ID = System.Int32.Parse((x?ID).ToString()); Name = (string x?Name)})
| _ -> failwith "fail json"
|> Array.iter(fun x -> printfn "%i %s" x.ID x.Name)
Print:
[ { "ID": 1, "Name": "First" }, { "ID": 2, "Name": "Second" }]
1 "First"
2 "Second"
But it
{ID = System.Int32.Parse((x?ID).ToString()); Name = (string x?Name)}
doesn't look good.
This I read about JsonExtensions,
but when I use
{ID = (x?ID.AsInteger()) ; Name = (x?Name.AsString()) }
I get compile errors:
The field, constructor or "AsInteger" is not defined
The field, constructor or "AsString" is not defined
Strangely, thing is that I see accessibility through "open FSharp.Data.JsonExtensions"
So, question: How to use JsonExtensions?
I tried to reproduce this using a minimal example, but I do not get the error - can you try the following minimal sample?
#r "...../FSharp.Data.dll"
open FSharp.Data.JsonExtensions
open FSharp.Data
JsonValue.Parse("A").AsArray()
|> Array.map (fun a -> a?ID.AsInteger())
I do not get auto-completion on a?ID. (which is a limitation of the editor), but it compiles fine.
The only reason why I think this could be not working is if you had another open declaration that would import another implementation of the ? operator that is not returning JsonValue.
The JsonValue API is certainly not as nice as just using the type provider - so if you can, I'd probably go for the type provider instead (the low-level API is good if you need to iterate over everything in JSON recursively).
I wrote this script to plot the historical financial data:
open FSharp.Data
#load "C:\Users\Nick\Documents\Visual Studio 2013\Projects\TryFsharp\packages\FSharp.Charting.0.90.9\FSharp.Charting.fsx"
open FSharp.Charting
open System
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s="+nasdaqcode
let company = CsvFile.Load(url)
let companyPrices = [ for r in company.Rows -> r.GetColumn "Date", r.GetColumn "Close" ]
(companyPrices
|> List.sort
|> Chart.Line).WithTitle(nasdaqcode, InsideArea=false)
plotprice "MSFT"
plotprice "ORCL"
plotprice "GOOG"
plotprice "NTES"
This works well.
Question:
Some of the data starts from the year 1986, some from 2000. I would like to plot the data from year 2000 to 2015. How to select this time period?
Is it possible to display the time when the mouse hovers over the chart?
If you are accessing Yahoo data, then it's better to use the CsvProvider rather than using CsvFile from F# Data. You can find more about the type provider here. Sadly, the naming in the standard F# Data library and on TryFSharp.org is different, so this is a bit confusing.
The CSV type provider will automatically infer the types:
open FSharp.Data
open FSharp.Charting
open System
// Generate type based on a sample
type Stocks = CsvProvider<"http://ichart.finance.yahoo.com/table.csv?s=FB">
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s=" + nasdaqcode
let company = Stocks.Load(url)
// Now you can access the columns in a statically-typed way
// and the types of the columns are inferred from the sample
let companyPrices = [ for r in company.Rows -> r.Date, r.Close ]
// If you want to do filtering, you can now use the `r.Date` property
let companyPrices =
[ for r in company.Rows do
if r.Date > DateTime(2010, 1, 1) && r.Date < DateTime(2011, 1, 1) then
yield r.Date, r.Close ]
// Charting as before
companyPrices |> (...)
I'm not sure if the F# Charting library has a way for showing the price based on mouse pointer location - it is based on standard .NET Windows Forms charting controls, so you could have a look at the documentation for the underlying library.
1) GetColumn gets a string. You need to first convert it to DateTime and simply compare it. i.e.
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s="+nasdaqcode
let company = CsvFile.Load(url)
let companyPrices = [ for r in company.Rows -> DateTime.Parse(r.GetColumn "Date"), r.GetColumn "Close" ]
(companyPrices
|> List.filter (fun (date, _) -> date > DateTime(2000, 1, 1))
|> List.sort
|> Chart.Line).WithTitle(nasdaqcode, InsideArea=false)
2) You can try with adding labels (not sure how to do on hover though...)
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s="+nasdaqcode
let company = CsvFile.Load(url)
let companyPrices = [ for r in company.Rows -> DateTime.Parse(r.GetColumn "Date"), r.GetColumn "Close" ]
(companyPrices
|> List.filter (fun (date, _) -> date > DateTime(2000, 1, 1))
|> List.sort
|> fun data -> Chart.Line(data, Labels=(Seq.map (fst >> string) data))).WithTitle(nasdaqcode, InsideArea=false)