I want to create a sequence of 'candles' ( a simple structure) for performing testing on financial data. I have downloaded some financial data from a broker and they are in a csv format.
I have the following code, using type providers:
type DukasCandles = CsvProvider<"C:\Users\**\Documents\Visual Studio 2017\Projects\FI\FI\Data\schema.csv" >
let row2Candle (myTicker: string ) (mySide: Side) (aRow:DukasCandles.Row) : Candle =
{Open = aRow.Open ;
Close = aRow.Close;
Low = aRow.Low
High = aRow.High
StartTime = aRow.LocalTime
Volume = aRow.Volume
Ticker = myTicker
Side = mySide }
let sortedCandles aTicker aParsedFile =
aParsedFile
|> Seq.map ( row2Candle aTicker Side.Bid )
|> Seq.sortBy ( fun candle -> candle.StartTime)
let fileContent = inputFile |> DukasCandles.Load
let rows = fileContent.Rows
let dataContent = rows|> (sortedCandles "EUR_USD")
It is just a toy example to test my understanding of type providers. I have it in a FSX script after the necessary boilerplate for declaring the file name and open the necessary modules.
Now the question: if I highlight this as it is and try to execute in F# interactive, then type
dataContent;;
I get the following message
val it : seq<Candle> =
Error: Couldn't parse row 1057 according to schema: Expecting DateTime in LocalTime, got 13.01.2016 00:00:00.000
The odd thing is the following:
suppose I type the following code in F# interactive:
fileContent;;
get the answer and then refer to it through the 'it' keyword, as in:
it.Rows |> sortedCandles "EUR_USD";;
Then the code is executed with no problems.
Why do we have such an inconsistent behaviour? Any idea?
Any help highly appreciated. Thank you.
Related
A couple days ago, I posted a question about deserialization with enums in F#.
The question is here: Deserialization in F# vs. C#
The answer pointed to some code written by Isaac Abraham, at: https://gist.github.com/isaacabraham/ba679f285bfd15d2f53e
However I am facing another problem:
If the object to deserialize to has an object of type 'enum option', the deserialization will fail, whereas it'll work if the type is just 'enum'.
A minimal example:
type TestType =
| A = 0
| B = 1
type TestObjectA =
{
test : TestType
}
type TestObjectB =
{
test : TestType option
}
let x = "{\"test\":\"A\"}"
let TestA = Deserialize<TestObjectA> x // will work
let TestB = Deserialize<TestObjectB> x // will fail
and the large deserialization code is at: https://pastebin.com/95JZLa6j
I put the whole code in a fiddle: https://dotnetfiddle.net/0Vc0Rh
but it can't be run from there since the F# version they support will not accept the 'object' keyword.
So, my question is: why can't I use the option type on an enum, but it works on other types? As a side note, since I'm quite new to F#, I'm not fully understanding Isaac's code, although I spent some time going through it and trying to troubleshoot it.
My understanding is that this line:
|> Seq.map (fun (value, propertyInfo) -> Convert.ChangeType(value, propertyInfo.PropertyType))
will try to convert the type to the right enum, but not to the enum option.
As a bonus question, is there a working solution that does full idiomatic deserialization with enums? (without going through null types)
open System.IO
type TestType =
| A = 0
| B = 1
type TestObjectB =
{
test : TestType option
}
let jsonSerializeToString obj =
use writer = new StringWriter()
let ser = new Newtonsoft.Json.JsonSerializer()
ser.Formatting <- Newtonsoft.Json.Formatting.Indented
ser.Serialize(writer, obj)
writer.ToString()
let jsonDeserializeFromString str =
Newtonsoft.Json.JsonConvert.DeserializeObject<TestObjectB>(str)
let Test obj =
let str = jsonSerializeToString obj
let obj' = jsonDeserializeFromString str
obj'
[<EntryPoint>]
let main argv =
{ test = Some TestType.B } |> Test |> ignore
{ test = None } |> Test |> ignore
0
Note: if you need to serialize a large collection of objects, then stream them to a file instead of an in-memory string to avoid an OutOfMemoryException. Like use writer = File.CreateText(filePath).
As a bonus question, is there a working solution that does full
idiomatic deserialization with enums?
I use the Microsoft.FsharpLu.Json package in production and find it works quite well for serializing and deserializing between "plain" javascript and idiomatic F#. Note Microsoft.FsharpLu.Json relies on Newtonsoft.Json under the hood.
Below is an example with your types and your test string, using Expecto for tests.
namespace FsharpLuJsonTest
open Newtonsoft.Json
open Microsoft.FSharpLu.Json
open Expecto
open Expecto.Flip
// Setup for FSharpLu.Json
type JsonSettings =
static member settings =
let s = JsonSerializerSettings(
NullValueHandling = NullValueHandling.Ignore,
MissingMemberHandling = MissingMemberHandling.Ignore)
s.Converters.Add(CompactUnionJsonConverter())
s
static member formatting = Formatting.None
type JsonSerializer = With<JsonSettings>
// Your example
type TestType =
| A = 0
| B = 1
type TestObjectA = { test : TestType }
type TestObjectB = { test : TestType option }
module Tests =
let x = """{"test":"A"}"""
[<Tests>]
let tests =
testList "Deserialization Tests" [
testCase "To TestObjectA" <| fun _ ->
JsonSerializer.deserialize x
|> Expect.equal "" { TestObjectA.test = TestType.A }
testCase "To TestObjectB" <| fun _ ->
JsonSerializer.deserialize x
|> Expect.equal "" { TestObjectB.test = Some TestType.A }
]
module Main =
[<EntryPoint>]
let main args =
runTestsInAssembly defaultConfig args
As you can see FsharpLu.Json supports Discriminated Unions and option types out of the box in the way you prefer. FsharpLu.Json is a less flexible solution than some others like Chiron (which allow for much more customisation) but I tend to prefer the opinionated approach of FsharpLu.Json.
I haven't used it personally, but the new FSharp.SystemText.Json library with the JsonUnionEncoding.ExternalTag setting should work roughly the same way FsharpLu.Json does. That library uses Microsoft's new System.Text.Json library under the hood rather than Newtonsoft.Json.
This is my very first attempt at F#, I know I still need to read a lot and I will, but these challenges help me understand the next parts I will read better, so far I managed to solve some of the first challenges, so please be patient......
The xml document looks like this:
<BuildCollection xmlns="http://schemas.datacontract.org/2004/07/BuildModels" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
...
<PostEvents xmlns:a="http://schemas.datacontract.org/2004/07/BuildModels.Events">
<a:EventBase i:type="a:ExternalProcess">
<a:Description>Structuring assemblies</a:Description>
How do I access the elements that start with "a:......."?
In C# I would do it like this:
foreach (var postBuildEventElement in
document
.Root.Element(ns + "PostEvents")
.Elements()
.Where(_ => _.Name.LocalName == "EventBase"))
So far in F# I have this:
module PostBuildInstructions
open System.Xml.Linq
open System.Linq
let xn s = XName.Get(s)
let PostBuildInstructions (xdoc: System.Xml.Linq.XDocument) =
xdoc.Root.Elements(xn "PostEvents").Elements().Where(e => e.)
let ExecutePostBuildInstructions buildDescriptor =
let "I need to enumerate the post build events here"
In the "main" code block I started with this: (I'm focusing on line 6 with this question)
open Log
[<EntryPoint>]
let main argv =
TryLogArgs argv
ReadPostBuildInstructions |> ExecutePostBuildInstructions
log "Press enter to exit"
System.Console.ReadLine() |> ignore
0 // return an integer exit code
Your use of the xn function is not equivalent to the use in the C# version. Your xn function returns a name with no associated namespace. However you're dealing with names within namespaces you you should be generating names as such.
Your C# version should ideally be like this:
XNamespace ns = "http://schemas.datacontract.org/2004/07/BuildModels";
XNamespace a = "http://schemas.datacontract.org/2004/07/BuildModels.Events";
var query = doc.Root
.Elements(ns + "PostEvents")
.Elements(a + "EventBase");
The F# version shouldn't be that much different:
let ns = XNamespace.Get("http://schemas.datacontract.org/2004/07/BuildModels")
let a = XNamespace.Get("http://schemas.datacontract.org/2004/07/BuildModels.Events")
let query = doc.Root.Elements(ns + "PostEvents").Elements(a + "EventBase")
But you may want to keep them as function calls. Just make sure you associate with the correct namespaces for the functions.
let xn n = XName.Get(n)
let ns n = XName.Get(n, "http://schemas.datacontract.org/2004/07/BuildModels")
let a n = XName.Get(n, "http://schemas.datacontract.org/2004/07/BuildModels.Events")
let query = doc.Root.Elements(ns "PostEvents").Elements(a "EventBase")
I'm using the FSharp.Data.JsonProvider to read Twitter Tweets.
Playing with this sample code
https://github.com/tpetricek/Documents/tree/master/Samples/Twitter.API
I want to expand the urls in the tweet with
let expandUrl (txt:string) (url:Search.DomainTypes<...>.DomainTypes.Url) =
txt.Replace( url.Url, url.ExpandedUrl )
This results in Error:
Lookup on object of indeterminate type based on information prior to this program point.
A type annotation may be needed prior to this program point to constrain the type of the object.
My problem is how to define the TypeProvider Type for url in the expandUrl function above?
The type inferance shows me this
val urls : FSharp.Data.JsonProvider<...>.DomainTypes.Url []
but this is not accepted in the type declaration. I assume "<...>" is not F# synatx.
How to do a type annotation for using a TypeProvider type e.g. FSharp.Data.JsonProvider<...>.DomainTypes.Url ?
Here is the complete code snippet:
open TwitterAPI // github.com/tpetricek/Documents/tree/master/Samples/Twitter.API
let twitter = TwitterAPI.TwitterContext( _consumerKey, _consumerSecret, _accessToken, _accessTokenSecret )
let query = "water"
let ts = Twitter.Search.Tweets(twitter, Utils.urlEncode query, count=100)
let ret =
[ for x in ts.Statuses do
// val urls : FSharp.Data.JsonProvider<...>.DomainTypes.Url []
let urls = x.Entities.Urls
// fully declarated to help the type inference at expandUrl
let replace (txt:string) (oldValue:string) (newValue:string) =
txt.Replace( oldValue, newValue)
// Error:
// Lookup on object of indeterminate type based on information prior to this program point.
// A type annotation may be needed prior to this program point to constrain the type of the object.
// This may allow the lookup to be resolved.
let expandUrl (txt:string) (url:FSharp.Data.JsonProvider<_>.DomainTypes.Url) =
replace txt url.Url url.ExpandedUrl
let textWithExpandedUrls = Array.fold expandUrl x.Text urls
yield textWithExpandedUrls
]
When you call Twitter.Search.Tweets (https://github.com/tpetricek/Documents/blob/master/Samples/Twitter.API/Twitter.fs#L284), the return type of that is one of the domain types of TwitterTypes.SearchTweets, which is a type alias for JsonProvider<"references\\search_tweets.json"> (https://github.com/tpetricek/Documents/blob/master/Samples/Twitter.API/Twitter.fs#L183).
Although in the tooltip it shows up as JsonProvider<...>.DomainTypes.Url, you'll have to use the type alias TwitterTypes.SearchTweets.DomainTypes.Url
I had a similar problem trying to figure out how to use the FSharp.Data HtmlProvider.
I am using Wikipedia to get information about USA presidents. The HtmlProvider does a great job of discovering the various tables in that webpage, but I wanted to extract the logic for processing a row of "president data" into a separate function called processRow.
And the problem was trying to work out what the type of such a row is for processRow's parameter row. The following code does the trick:
#load "Scripts\load-references.fsx"
open FSharp.Data
let presidents = new HtmlProvider<"https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States">()
let ps = presidents.Tables.``List of presidents``
ps.Headers |> Option.map (fun hs -> for h in hs do printf "%s " h)
printfn ""
type Presidents = ``HtmlProvider,Sample="https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States"``.ListOfPresidents
let processRow (row:Presidents.Row) =
printfn "%d %s" row.``№`` row.President2
ps.Rows |> Seq.iter processRow
I did not type in the long type alias for Presidents, I used Visual Studio auto-completion by guessing that the type for List of presidents would be discoverable from something starting with Html, and it was, complete with the four single back quotes.
So let's say I have a CSV file with a header containing columns Population and Profit, and I'd like to work with it in F# interactive. I have the following code:
#r "../packages/FSharp.Data.1.1.10/lib/net40/FSharp.Data.dll"
open FSharp.Data
// load csv header
let cities = new CsvProvider<"cities.csv">()
// how to reach data
let firstRow = cities.Data |> Seq.head
let firstPopulation = firstRow.Population
let firstProfit = firstRow.Profit
I get an error from F# interactive:
error FS0039: The field, constructor or member 'Population' is not defined
This seems confusing to me, because intellisense in VS has no problem picking up this column from my data via a CSV type provider.
Also, I tried creating a program with the same type provider and it all works just fine. Like this:
open FSharp.Data
[<EntryPoint>]
let main argv =
use file = System.IO.File.CreateText("result.txt")
let csv = new CsvProvider<"cities.csv">()
for record in csv.Data do
fprintfn file "%A" record.Population
0
Am I missing something? Thanks for any answer.
Try this code
let Cities = new CsvProvider<"cities.csv">()
let cities = new Cities()
let firstRow = cities.Rows |> Seq.head
Does anyone know of 'prior art' regarding the following subject :
I have data that take some decent time to load. they are historical level for various stocks.
I would like to preload them somehow, to avoid the latency when using my app
However, preloading them in one chunk at start makes my app unresponsive first which is not user friendly
So I would like to not load my data.... unless the user is not requesting any and playing with what he already has, in which case I would like to get little by little. So it is neither 'lazy' nor 'eager', more 'lazy when you need' and 'eager when you can', hence the acronym LWYNEWYC.
I have made the following which seems to work, but I just wonder if there is a recognized and blessed approach for such thing ?
let r = LoggingFakeRepo () :> IQuoteRepository
r.getHisto "1" |> ignore //prints Getting histo for 1 when called
let rc = RepoCached (r) :> IQuoteRepository
rc.getHisto "1" |> ignore //prints Getting histo for 1 the first time only
let rcc = RepoCachedEager (r) :> IQuoteRepository
rcc.getHisto "100" |> ignore //prints Getting histo 1..100 by itself BUT
//prints Getting histo 100 immediately when called
And the classes
type IQuoteRepository =
abstract getUnderlyings : string seq
abstract getHisto : string -> string
type LoggingFakeRepo () =
interface IQuoteRepository with
member x.getUnderlyings = printfn "getting underlyings"
[1 .. 100] |> List.map string :> _
member x.getHisto udl = printfn "getting histo for %A" udl
"I am a historical dataset in a disguised party"
type RepoCached (rep : IQuoteRepository) =
let memoize f =
let cache = new System.Collections.Generic.Dictionary<_, _>()
fun x ->
if cache.ContainsKey(x) then cache.[x]
else let res = f x
cache.[x] <- res
res
let udls = lazy (rep.getUnderlyings )
let gethistom = memoize rep.getHisto
interface IQuoteRepository with
member x.getUnderlyings = udls.Force()
member x.getHisto udl = gethistom udl
type Message = string * AsyncReplyChannel<UnderlyingWrap>
type RepoCachedEager (rep : IQuoteRepository) =
let udls = rep.getUnderlyings
let agent = MailboxProcessor<Message>.Start(fun inbox ->
let repocached = RepoCached (rep) :> IQuoteRepository
let rec loop l =
async { try
let timeout = if l|> List.isEmpty then -1 else 50
let! (udl, replyChannel) = inbox.Receive(timeout)
replyChannel.Reply(repocached.getHisto udl)
do! loop l
with
| :? System.TimeoutException ->
let udl::xs = l
repocached.getHisto udl |> ignore
do! loop xs
}
loop (udls |> Seq.toList))
interface IQuoteRepository with
member x.getUnderlyings = udls
member x.getHisto udl = agent.PostAndReply(fun reply -> udl, reply)
I like your solution. I think using agent to implement some background loading with a timeout is a great way to go - agents can nicely encapsulate mutable state, so it is clearly safe and you can encode the behaviour you want quite easily.
I think asynchronous sequences might be another useful abstraction (if I'm correct, they are available in FSharpX these days). An asynchronous sequence represents a computation that asynchronously produces more values, so they might be a good way to separate the data loader from the rest of the code.
I think you'll still need an agent to synchronize at some point, but you can nicely separate different concerns using async sequences.
The code to load the data might look something like this:
let loadStockPrices repo = asyncSeq {
// TODO: Not sure how you detect that the repository has no more data...
while true do
// Get next item from the repository, preferably asynchronously!
let! data = repo.AsyncGetNextHistoricalValue()
// Return the value to the caller...
yield data }
This code represents the data loader, and it separates it from the code that uses it. From the agent that consumes the data source, you can use AsyncSeq.iterAsync to consume the values and do something with them.
With iterAsync, the function that you specify as a consumer is asynchronous. It may block (i.e. using Sleep) and when it blocks, the source - that is.your loader - is also blocked. This is quite nice implicit way to control the loader from the code that consumes the data.
A feature that is not in the library yet (but would be useful) is an partially eager evaluator that takes AsyncSeq<'T> and returns a new AsyncSeq<'T> but obtains a certain number of elements from the source as soon as possible and caches them (so that the consumer does not have to wait when it asks for a value, as long as the source can produce values fast enough).