Playing around with F# Type Providers and have not got too far. I have the following json file (called PriceDemand.json):
[
{
"intervalDate": "2018-01-22T00:00:00+11:00",
"regionId": "NSW1",
"rrp": 114.17,
"totalDemand": 12338.04
},
{
"intervalDate": "2018-01-22T00:00:00+11:00",
"regionId": "NSW1",
"rrp": 113.41,
"totalDemand": 12334.98
}
]
I've written the following code to process it:
open FSharp.Data
open System
type PriceDemand = JsonProvider<"PriceDemand.json">
let data = PriceDemand.Parse("PriceDemand.json")
[<EntryPoint>]
let main argv =
data |> Seq.iter (fun v -> printf "%s" v.RegionId)
Console.ReadLine() |> ignore
0 // return an integer exit code
I have intellisense for the PriceDemand type, but the following TypeInitializationExceptionexception is being thrown:
Invalid JSON starting at character 0,
snippet =
---- PriceDemand
----- json =
------ PriceDemand.json
Any ideas what I am doing wrong?
You're calling .Parse where you should be calling .Load. The string "PriceDemand.json" is being parsed as JSON, which is not valid. If you change the call to let data = PriceDemand.Load("PriceDemand.json"), it should work fine.
Related
Why can't someJsonType parse 9.433536880271462E-4 to decimal?
Parsing the value myself works fine let c = decimal "9.433536880271462E-4"
#r "nuget: FSharp.Data, 3.3.3"
type someJsonType = FSharp.Data.JsonProvider<"""
{
"point": 45.5
}
""">
let a = someJsonType.Parse "{ \"point\": 12.5 }"
let b = someJsonType.Parse "{ \"point\": 9.433536880271462E-4 }"
let c = someJsonType.Parse "{ \"point\": 0.0009433536880271462 }"
printfn "%f" a.Point // OK
printfn "%f" b.Point // exception
printfn "%f" c.Point // OK
let d = decimal "9.433536880271462E-4" // OK
Exception:
System.Exception: Expecting a Decimal at '/point', got 0.0009433536880271462
at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThenFail#1433.Invoke(String message) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\printf.fs:line 1433
at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr callersRetAddrSlot, IntPtr callTarget, IntPtr retVal)
at Microsoft.FSharp.Core.OptimizedClosures.Invoke#3298-1.Invoke(T2 u, T3 v) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\prim-types.fs:line 3298
at <StartupCode$FSI_0072>.$FSI_0072.main#()
Stopped due to error
The data type is inferred as decimal, but b is float hence the exception. This is a quirk that means you have to modify the sample json to ensure the correct type is inferred. So, if you want to infer float then you have to modify your sample:
#r "nuget: FSharp.Data, 3.3.3"
type someJsonType = FSharp.Data.JsonProvider<"""
{
"point": 9.433536880271462E-4
}
""">
let a = someJsonType.Parse "{ \"point\": 12.5 }"
let b = someJsonType.Parse "{ \"point\": 9.433536880271462E-4 }"
let c = someJsonType.Parse "{ \"point\": 0.0009433536880271462 }"
printfn "%f" a.Point // val a.Point is float with get
printfn "%f" b.Point // OK
printfn "%f" c.Point // OK
If you want it to be decimal you can cast this after loading json data into your local object. This is what you are actually doing with let d = decimal "9.433536880271462E-4" // OK
(Similarly if the json schema is meant to be providing floats or decimals but the sample only has what appears to be an int, an int would be inferred, so again you have to modify the sample to ensure the correct type is inferred)
The JsonProvider can only work with what is at hand in the sample json, hence the importance of the sample helping the inference.
(Note: We don't always have control of the sample json, if coming from a live third party site, so I think some form of type annotation might be useful to assist but as far as I know this does not exist yet.)
I am very new to F#, and I'm trying to simply get an array of structs where the struct is the filename and the filecontents is an array of lines in the file. I have an error I don't really understand on the indicated line, and I haven't been able to find the correct syntax or approach to do this.
let readFileContents filePath =
File.ReadAllLines(filePath)
let makeFileStruct fileName =
new FileContents(fileName, fileName |> readFileContents)
let fileTemplates path =
Directory.GetFiles(path, "*.template")
|> Array.map(fun x -> Path.GetFileName(x))
|> Array.iter(fun x -> makeFileStruct(x)) <--- error: This expression was expected to have type 'unit'
[<EntryPoint>]
let main argv =
printfn "Testing getting files"
argv.[0]
|> fileTemplates
|> JsonConvert.SerializeObject
|> printfn "some stuff %s"
0
I'm trying to simply get an array of structs where the struct is the filename
I don't think this is a good idea becuase in order to get the predictable memory usage of a struct you'd need some combination of a very strict filename length limit and/or a very memory-inefficient struct. Unless the struct contains an non-struct object.
I have an error I don't really understand on the indicated line
Array.iter executes a function for each element of the array. So the inner function makeFileStruct would need to return a unit in order to execute it. You are looking for Array.map which creates an array from the outputs.
I've got an application that I've built in SAFE-Stack using websockets, more or less following the approach here: https://github.com/CompositionalIT/safe-sockets
It works fine but the Elmish debugger doesn't like the type of WsSender in this example:
type ConnectionState =
| DisconnectedFromServer
| ConnectedToServer of WsSender
| Connecting
member this.IsConnected =
match this with
| ConnectedToServer _ -> true
| DisconnectedFromServer | Connecting -> false
and WsSender = Msg -> Unit
giving the following error message in the Browser Console:
Can anyone tell me how to go about fixing this issue? (Assuming it's fixable and that I've diagnosed the problem correctly.) Thanks.
you see this error because of Elmish.Debugger using Thoth.Json to serialize your Msg/Model to a JSON format.
The type WsSender can't be represented in a JSON format because it is a function. So Thoth.Json is asking you to explain how it should encode this type.
You can do that by creating what is called an extraCoder like that:
In your case, you will have to create a fake encoder/decoder "just" to make the Debugger happy.
module CustomEncoders =
let wsSenderEncoder (_ : WsSender) = Encode.string "WsSender function"
let wsSenderDecoder = Decode.fail "Decoding is not supported for WsSender type"
let myExtraCoders =
Extra.empty
|> Extra.withCustom wsSenderEncoder wsSenderDecoder
let modelEncoder = Encode.Auto.generateEncoder(extra = myExtraCoders)
let modelDecoder = Decode.Auto.generateDecoder(extra = myExtraCoders)
In your Program creation, you should replace Program.withDebugger by Program.withDebuggerCoders and give it the encoder and decoder you created.
Program.withDebuggerCoders CustomEncoders.modelEncoder CustomEncoders.modelDecoder
I had a bit of a play around to try and come up with something that would make it easier to have multiple extra coders if required. This seems to work - thought it might be helpful to others.
module CustomEncoders =
let inline addDummyCoder<'b> extrasIn =
let typeName = string typeof<'b>
let simpleEncoder(_ : 'b) = Encode.string (sprintf "%s function" typeName)
let simpleDecoder = Decode.fail (sprintf "Decoding is not supported for %s type" typeName)
extrasIn |> Extra.withCustom simpleEncoder simpleDecoder
let inline buildExtras<'a> extraCoders =
let myEncoder:Encoder<'a> = Encode.Auto.generateEncoder(extra = extraCoders)
let myDecoder:Decoder<'a> = Decode.Auto.generateDecoder(extra = extraCoders)
(myEncoder, myDecoder)
type TestType = Msg -> Unit
type TestType2 = string -> Unit
let extras = Extra.empty
|> CustomEncoders.addDummyCoder<TestType>
|> CustomEncoders.addDummyCoder<TestType2>
|> CustomEncoders.buildExtras<Model.Model>
#if DEBUG
open Elmish.Debug
open Elmish.HMR
#endif
Program.mkProgram Model.init Model.update View.render
|> Program.withSubscription subs
#if DEBUG
|> Program.withConsoleTrace
#endif
|> Program.withReactBatched "elmish-app"
#if DEBUG
|> Program.withDebuggerCoders (fst extras) (snd extras)
#endif
|> Program.run
If anyone has a better idea of how to do it, I'd be happy to update this answer with their suggestions. Also, the apostrophe in the generic type seems to upset the code prettifier above - do I need to do something to fix that?
I want to create a sequence of 'candles' ( a simple structure) for performing testing on financial data. I have downloaded some financial data from a broker and they are in a csv format.
I have the following code, using type providers:
type DukasCandles = CsvProvider<"C:\Users\**\Documents\Visual Studio 2017\Projects\FI\FI\Data\schema.csv" >
let row2Candle (myTicker: string ) (mySide: Side) (aRow:DukasCandles.Row) : Candle =
{Open = aRow.Open ;
Close = aRow.Close;
Low = aRow.Low
High = aRow.High
StartTime = aRow.LocalTime
Volume = aRow.Volume
Ticker = myTicker
Side = mySide }
let sortedCandles aTicker aParsedFile =
aParsedFile
|> Seq.map ( row2Candle aTicker Side.Bid )
|> Seq.sortBy ( fun candle -> candle.StartTime)
let fileContent = inputFile |> DukasCandles.Load
let rows = fileContent.Rows
let dataContent = rows|> (sortedCandles "EUR_USD")
It is just a toy example to test my understanding of type providers. I have it in a FSX script after the necessary boilerplate for declaring the file name and open the necessary modules.
Now the question: if I highlight this as it is and try to execute in F# interactive, then type
dataContent;;
I get the following message
val it : seq<Candle> =
Error: Couldn't parse row 1057 according to schema: Expecting DateTime in LocalTime, got 13.01.2016 00:00:00.000
The odd thing is the following:
suppose I type the following code in F# interactive:
fileContent;;
get the answer and then refer to it through the 'it' keyword, as in:
it.Rows |> sortedCandles "EUR_USD";;
Then the code is executed with no problems.
Why do we have such an inconsistent behaviour? Any idea?
Any help highly appreciated. Thank you.
I wrote this script from some resources I found. It's working but I some files I have problem. I am new in F# so how can I change line with FileHelpersException to get exact line where is the problem? Thanks
// Learn more about F# at http://fsharp.net
// See the 'F# Tutorial' project for more help.
open FileHelpers
open System
[<DelimitedRecord(",")>]
type CsvRecord =
class
val field1 : string
val field2 : string
val field3 : int
new () = {
field1 = ""
field2 = ""
field3 = 0
}
end
[<EntryPoint>]
let main argv =
use file = System.IO.File.CreateText("result.txt")
let engine = new FileHelperEngine<CsvRecord>()
engine.Encoding <- new Text.UTF8Encoding()
let res =
try
engine.ReadFile("test.csv")
with
| :? FileHelpersException -> Array.empty<CsvRecord>
for record in res do
fprintfn file "%s" record.field1
printf "DONE!"
let s = Console.ReadLine()
0 // return an integer exit code
I suggest that you use CsvTypeProvider instead. When there's a mismatch the error message states the line which has the error
open FSharp.Data
[<EntryPoint>]
let main argv =
use file = System.IO.File.CreateText("result.txt")
let csv = new CsvProvider<"test.csv">()
for record in csv.Data do
fprintfn file "%s" record.field1
If you want to ignore the lines with errors, just pass IgnoreErrors=true as an extra parameter to CsvProvider
This question is about the FileHelpers library you are using, not F#, so looking at the docs for that might help. In this case you can check for ConvertException instead of FileHelpersException, which contains members that give you more details about the member.
try
engine.ReadFile("test.csv")
with
| :? ConvertException as ex ->
printfn "ERROR: Line %d Col %d" ex.LineNumber ex.ColumnNumber
Array.empty<CsvRecord>
I agree with Gustavo though, you might find it easier to use the CsvTypeProvider.