This is the part of the function that causes me trouble. I have isolated this part from my main function after some debugging:
let testfunction view =
printfn "Entry point: %s" view
let df = ETS.getTimeSeries myrun.tsds (myrun.initiate2 "0019.HK" view defaultbegintime defaultendtime) "unusedlegacystring"
printfn "I am here"
df |> ETS.printRecords
let async1 = async{testfunction("BID")}
let async2 = async{testfunction("ASK")}
let gglist = [async1;async2] //Async<unit> list
ETS.getTimeSeries returns a Deedle frame from the internet which is saved to df.
ETS.printRecords is a helper function that prints the Deedle frame nicely.
The parameters are all correct. I know it from running:
Async.Start async1
// prints out the dataframe as expected. Same goes for async2 as well.
If I run:
gglist
|> Async.Parallel
|> Async.Ignore
|> Async.RunSynchronously
The output is only
Entry point: BID
Entry point: ASK
I suspect this may have something to do with the fact that ETS.getTimeSeries takes anywhere from 3 seconds to 1 minute to pull the data to a Deedle frame and that I am missing some sort of command to wait for getTimeSeries is finished.
As always, any help would be much appreciated, thank you.
Update: this may have something to do with the Eikon API itself not being "thread-safe"(???) (ETS.getTimeSeries). Someone had a solution in C#, and I am trying very hard to re-write it to F#. I will come back with a solution if possible, thanks
https://community.developers.refinitiv.com/questions/11837/eikon-net-api-accepts-2-3-time-series-requests-and.html
For anyone that is interested, here is the F# implementation of the API (which I copied from the web and claims no credit for):
let getTimeSeries (tsds:ITimeSeriesDataService) (setRequestParameters:SetDataRequestParameters) (ric:ReutersInstrumentCode) : seq<IData> =
let mutable records = Seq.empty<IData> // IData is seq<KeyValuePair>
let mutable transformer = Seq.empty<IData>
let frame = DispatcherFrame()
let dataReceivedCallback (dataChunk:DataChunk) =
//transformer <- DataChunk.Records.ToTickRecords()
records <- Seq.append records dataChunk.Records
frame.Continue <- dataChunk.IsLast |> not
()
(tsds.SetupDataRequest(ric)
|> setRequestParameters) // apply the function to the TimeSeriesDataRequestSetup object
//.WithFields([]).From(Nullable<DateTime> DateTime.Now).WithAdjustedPrice(true).WithNumberOfPoints(10).WithView("BID")
.OnDataReceived(Action<DataChunk> dataReceivedCallback)
.CreateAndSend()
|> ignore
Dispatcher.PushFrame(frame) // "Enters an execute loop"
records
Credit: https://gist.github.com/roguetrainer/59e4f4664528c40239f1564ef7d6dcee
Update2: Here is a really stupid (but also effective? solution)
Create 3 instances of ETS.getTimeSeries. Let's call them ETS.getTimeSeries1, ETS.getTimeSeries2 and ETS.getTimeSeries3
Manually hard code task1 with getTimeSeries1 and so on
[task1; task2; task3] |> Async.Parallel |> Async.Ignore |> Async.StartImmediate
//work as expected
I have a 16core/32thread Gen1 TR CPU, so I guess I can hard code up to 32 threads to fully take advantage of this....
Related
I am using the csv type provider to collect some data from a series of files I have on Azure blob storage:
#r "../packages/FSharp.Data.2.0.9/lib/portable-net40+sl5+wp8+win8/FSharp.Data.dll"
open FSharp.Data
type censusDataContext = CsvProvider<"https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/AK.TXT">
type stateCodeContext = CsvProvider<"https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/states.csv">
let stateCodes = stateCodeContext.Load("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/states.csv");
let fetchStateData (stateCode:string)=
let uri = System.String.Format("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/{0}.TXT",stateCode)
censusDataContext.Load(uri).Rows
let usaData = stateCodes.Rows
|> Seq.collect(fun r -> fetchStateData(r.Abbreviation))
|> Seq.length
I now want to run these async and I am running into a problem with AsyncLoad:
let fetchStateDataAsync(stateCode:string)=
async{
let uri = System.String.Format("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/{0}.TXT",stateCode)
let! stateData = censusDataContext.AsyncLoad(uri)
return stateData.Rows
}
let usaData = stateCodes.Rows
|> Seq.collect(fun r -> fetchStateDataAsync(r.Abbreviation))
|> Seq.length
The error message is
The type 'Async<seq<CsvProvider<...>.Row>>' is not compatible with the type 'seq<'a>'
Forgive my lack of async knowledge, but do I have to use something other than Seq.Collect when applying async functions?
Thanks in advance
The problem is that turning code to asynchronous (by wrapping it in the async { .. } block) changes the result from seq<Row> to Async<seq<Row>> - that is, you now get an asynchronous computation that will eventually complete and return the sequence.
To fix this, you need to somehow start the computation and wait for the result. There is a number of choices - like running one by one sequentially. Probably the easiest option (and maybe the best - depending on what you want to do) is to run the computations in parallel:
let getAll =
stateCodes.Rows
|> Seq.map(fun r -> fetchStateDataAsync(r.Abbreviation))
|> Async.Parallel
This gives you an asynchronous computation that runs all the downloads and returns an array of results. You can run this synchronously (and block) and get the results:
getAll |> Async.RunSynchronously
|> Seq.collect id
|> Seq.length
If you want to run the downloads asynchronously in the background you can do that to, but you need to specify what to do with the result. For example:
async {
let! all = getAll
all |> Seq.collect id |> Seq.length |> printfn "Length %d" }
|> Async.Start
There is any way to do it like C/C#?
For example (C# style)
for (int i = 0; i < 100; i++)
{
if (i == 66)
break;
}
The short answer is no. You would generally use some higher-order function to express the same functionality. There is a number of functions that let you do this, corresponding to different patterns (so if you describe what exactly you need, someone might give you a better answer).
For example, tryFind function returns the first value from a sequence for which a given predicate returns true, which lets you write something like this:
seq { 0 .. 100 } |> Seq.tryFind (fun i ->
printfn "%d" i
i=66)
In practice, this is the best way to go if you are expressing some high-level logic and there is a corresponding function. If you really need to express something like break, you can use a recursive function:
let rec loop n =
if n < 66 then
printfn "%d" n
loop (n + 1)
loop 0
A more exotic option (that is not as efficient, but may be nice for DSLs) is that you can define a computation expression that lets you write break and continue. Here is an example, but as I said, this is not as efficient.
This is really ugly, but in my case it worked.
let mutable Break = false
while not Break do
//doStuff
if breakCondition then
Break <- true
done
This is useful for do-while loops, because it guarantees that the loop is executed at least once.
I hope there's a more elegant solution. I don't like the recursive one, because I'm afraid of stack overflows. :-(
You have to change it to a while loop.
let (i, ans) = (ref 0, ref -1)
while(!i < 100 and !ans < 0) do
if !i = 66 then
ans := !i
ans
(This breaks when i gets to 66--but yes the syntax is quite different, another variable is introduced, etc.)
seq {
for i = 0 to 99 do
if i = 66 then yield ()
}
|> Seq.tryItem 0
|> ignore
Try this:
exception BreakException
try
for i = 0 to 99 do
if i = 66 then
raise BreakException
with BreakException -> ()
I know that some folks don't like to use exceptions. But it has merits.
You don't have to think about complicated recursive function. Of
cause you can do that, but sometimes it is unnecessarily bothersome
and using exception is simpler.
This method allows you to break at halfway of the loop body. (Break "flag" method is simple too but it only allows to break at the end of the loop body.)
You can easily escape from nested loop.
For these kind of problems you could use a recursive function.
let rec IfEqualsNumber start finish num =
if start = finish then false
elif
start = num then true
else
let start2 = start + 1
IfEqualsNumber start2 finish num
Recently I tried to solve a similar situation:
A list of, say, 10 pieces of data. Each of them must be queried against a Restful server, then get a result for each.
let lst = [4;6;1;8]
The problem:
If there is a failed API call (e.g. network issue), there is no point making further calls as we need all the 10 results available. The entire process should stop ASAP when an API call fails.
The naive approach: use List.map()
lst |> List.map (fun x ->
try
use sqlComd = ...
sqlComd.Parameters.Add("#Id", SqlDbType.BigInt).Value <- x
sqlComd.ExecuteScala() |> Some
with
| :? System.Data.SqlClient.SqlException as ex -> None
)
But as said, it's not optimal. When a failed API occurs, the remaining items keep being processed. They do something that is ignored at the end anyway.
The hacky approach: use List.tryFindIndex()
Unlike map(), we must store the results somewhere in the lamda function. A reasonable choice is to use mutable list. So when tryFindIndex() returns None, we know that everything was ok and can start making use of the mutable list.
val myList: List<string>
let res = lst |> List.tryFindIndex (fun x ->
try
use sqlComd = ...
sqlComd.Parameters.Add("#Id", SqlDbType.BigInt).Value <- x
myList.Add(sqlComd.ExecuteScala())
false
with
|:? System.Data.SqlClient.SqlException as ex -> true
)
match res with
| Some _ -> printfn "Something went wrong"
| None -> printfn "Here is the 10 results..."
The idiomatic approach: use recursion
Not very idiomatic as it uses Exception to stop the operation.
exception MyException of string
let makeCall lstLocal =
match lstLocal with
| [] -> []
| head::tail ->
try
use sqlComd = ...
sqlComd.Parameters.Add("#Id", SqlDbType.BigInt).Value <- x
let temp = sqlComd.ExecuteScala()
temp :: makeCall (tail)
with
|:? System.Data.SqlClient.SqlException as ex -> raise MyException ex.Message
try
let res = makeCall lst
printfn "Here is the 10 results..."
with
| :? MyException -> printfn "Something went wrong"
The old-fashion imperative approach: while... do
This still involves mutable list.
Does anyone know of 'prior art' regarding the following subject :
I have data that take some decent time to load. they are historical level for various stocks.
I would like to preload them somehow, to avoid the latency when using my app
However, preloading them in one chunk at start makes my app unresponsive first which is not user friendly
So I would like to not load my data.... unless the user is not requesting any and playing with what he already has, in which case I would like to get little by little. So it is neither 'lazy' nor 'eager', more 'lazy when you need' and 'eager when you can', hence the acronym LWYNEWYC.
I have made the following which seems to work, but I just wonder if there is a recognized and blessed approach for such thing ?
let r = LoggingFakeRepo () :> IQuoteRepository
r.getHisto "1" |> ignore //prints Getting histo for 1 when called
let rc = RepoCached (r) :> IQuoteRepository
rc.getHisto "1" |> ignore //prints Getting histo for 1 the first time only
let rcc = RepoCachedEager (r) :> IQuoteRepository
rcc.getHisto "100" |> ignore //prints Getting histo 1..100 by itself BUT
//prints Getting histo 100 immediately when called
And the classes
type IQuoteRepository =
abstract getUnderlyings : string seq
abstract getHisto : string -> string
type LoggingFakeRepo () =
interface IQuoteRepository with
member x.getUnderlyings = printfn "getting underlyings"
[1 .. 100] |> List.map string :> _
member x.getHisto udl = printfn "getting histo for %A" udl
"I am a historical dataset in a disguised party"
type RepoCached (rep : IQuoteRepository) =
let memoize f =
let cache = new System.Collections.Generic.Dictionary<_, _>()
fun x ->
if cache.ContainsKey(x) then cache.[x]
else let res = f x
cache.[x] <- res
res
let udls = lazy (rep.getUnderlyings )
let gethistom = memoize rep.getHisto
interface IQuoteRepository with
member x.getUnderlyings = udls.Force()
member x.getHisto udl = gethistom udl
type Message = string * AsyncReplyChannel<UnderlyingWrap>
type RepoCachedEager (rep : IQuoteRepository) =
let udls = rep.getUnderlyings
let agent = MailboxProcessor<Message>.Start(fun inbox ->
let repocached = RepoCached (rep) :> IQuoteRepository
let rec loop l =
async { try
let timeout = if l|> List.isEmpty then -1 else 50
let! (udl, replyChannel) = inbox.Receive(timeout)
replyChannel.Reply(repocached.getHisto udl)
do! loop l
with
| :? System.TimeoutException ->
let udl::xs = l
repocached.getHisto udl |> ignore
do! loop xs
}
loop (udls |> Seq.toList))
interface IQuoteRepository with
member x.getUnderlyings = udls
member x.getHisto udl = agent.PostAndReply(fun reply -> udl, reply)
I like your solution. I think using agent to implement some background loading with a timeout is a great way to go - agents can nicely encapsulate mutable state, so it is clearly safe and you can encode the behaviour you want quite easily.
I think asynchronous sequences might be another useful abstraction (if I'm correct, they are available in FSharpX these days). An asynchronous sequence represents a computation that asynchronously produces more values, so they might be a good way to separate the data loader from the rest of the code.
I think you'll still need an agent to synchronize at some point, but you can nicely separate different concerns using async sequences.
The code to load the data might look something like this:
let loadStockPrices repo = asyncSeq {
// TODO: Not sure how you detect that the repository has no more data...
while true do
// Get next item from the repository, preferably asynchronously!
let! data = repo.AsyncGetNextHistoricalValue()
// Return the value to the caller...
yield data }
This code represents the data loader, and it separates it from the code that uses it. From the agent that consumes the data source, you can use AsyncSeq.iterAsync to consume the values and do something with them.
With iterAsync, the function that you specify as a consumer is asynchronous. It may block (i.e. using Sleep) and when it blocks, the source - that is.your loader - is also blocked. This is quite nice implicit way to control the loader from the code that consumes the data.
A feature that is not in the library yet (but would be useful) is an partially eager evaluator that takes AsyncSeq<'T> and returns a new AsyncSeq<'T> but obtains a certain number of elements from the source as soon as possible and caches them (so that the consumer does not have to wait when it asks for a value, as long as the source can produce values fast enough).
Is there a way to creating an observable from a sequence in F#?
The required behaviour is that an observer subscribing to the resulting observable receives all the values of the sequence one at a time.
Edit: The question can be framed as: Is there an equivalent to Rx.Observable.FromArray([1,2,3]) as shown here in F#?
Edit 2: Thanks everyone. It took me some time to find the answer, only to find that desco has answered already. For the sake of completeness, here is how it works:
//Using the Reactive Framework.
//References to System.CoreEx.dll and System.Reactive.dll are needed.
open System.Linq
let observable = Observable.ToObservable [1..10]
let odds = observable |> Observable.filter (fun i -> i%2=1)
let disp = odds.Subscribe (fun i -> printfn "%d" i)
disp.Dispose ()
Just in case you wanted to avoid the additional dependency, it is actually quite easy to implement the behavior using F# events and Observable.guard function that I already used in another answer here at StackOverflow.
Here is the implementation together with a brief example:
module Observable =
let ofSeq s =
let evt = new Event<_>()
evt.Publish |> Observable.guard (fun o ->
for n in s do evt.Trigger(n))
[ 1 .. 10 ] |> Observable.ofSeq
|> Observable.filter (fun n -> n%2 = 0)
|> Observable.add (printfn "%d")
And for completeness, the implementation of Observable.guard looks like this:
/// Creates an observable that calls the specified function after someone
/// subscribes to it (useful for waiting using 'let!' when we need to start
/// operation after 'let!' attaches handler)
let guard f (e:IObservable<'Args>) =
{ new IObservable<'Args> with
member x.Subscribe(observer) =
let rm = e.Subscribe(observer) in f(); rm }
Is System.Reactive.Linq.Observable.ToObservable(this IEnumerable source) located in System.Reactive.dll the thing your wanted?
Yoou can try Reactive Extensions. Please refer to the following blog entry which might help you a lot.
http://www.navision-blog.de/2009/10/20/iobservableiobserver-using-the-reactive-framework-with-f/
I just finish writing my first F# program. Functionality wise the code works the way I wanted, but not sure if the code is efficient. I would much appreciate if someone could review the code for me and point out the areas where the code can be improved.
Thanks
Sudaly
open System
open System.IO
open System.IO.Pipes
open System.Text
open System.Collections.Generic
open System.Runtime.Serialization
[<DataContract>]
type Quote = {
[<field: DataMember(Name="securityIdentifier") >]
RicCode:string
[<field: DataMember(Name="madeOn") >]
MadeOn:DateTime
[<field: DataMember(Name="closePrice") >]
Price:float
}
let m_cache = new Dictionary<string, Quote>()
let ParseQuoteString (quoteString:string) =
let data = Encoding.Unicode.GetBytes(quoteString)
let stream = new MemoryStream()
stream.Write(data, 0, data.Length);
stream.Position <- 0L
let ser = Json.DataContractJsonSerializer(typeof<Quote array>)
let results:Quote array = ser.ReadObject(stream) :?> Quote array
results
let RefreshCache quoteList =
m_cache.Clear()
quoteList |> Array.iter(fun result->m_cache.Add(result.RicCode, result))
let EstablishConnection() =
let pipeServer = new NamedPipeServerStream("testpipe", PipeDirection.InOut, 4)
let mutable sr = null
printfn "[F#] NamedPipeServerStream thread created, Wait for a client to connect"
pipeServer.WaitForConnection()
printfn "[F#] Client connected."
try
// Stream for the request.
sr <- new StreamReader(pipeServer)
with
| _ as e -> printfn "[F#]ERROR: %s" e.Message
sr
while true do
let sr = EstablishConnection()
// Read request from the stream.
printfn "[F#] Ready to Receive data"
sr.ReadLine()
|> ParseQuoteString
|> RefreshCache
printfn "[F#]Quot Size, %d" m_cache.Count
let quot = m_cache.["MSFT.OQ"]
printfn "[F#]RIC: %s" quot.RicCode
printfn "[F#]MadeOn: %s" (String.Format("{0:T}",quot.MadeOn))
printfn "[F#]Price: %f" quot.Price
In general, you should try using immutable data types and avoid imperative constructs such as global variables and imperative loops - although using them in F# is fine in many cases, they should be used only when there is a good reason for doing so. Here are a couple of examples where you could use functional approach:
First of all, to make the code more functional, you should avoid using global mutable cache. Instead, your RefreshCache function should return the data as the result (preferably using some functional data structure, such as F# Map type):
let PopulateCache quoteList =
quoteList
// Generate a sequence of tuples containing key and value
|> Seq.map (fun result -> result.RicCode, result)
// Turn the sequence into an F# immutable map (replacement for hashtable)
|> Map.ofSeq
The code that uses it would be changed like this:
let cache =
sr.ReadLine()
|> ParseQuoteString
|> PopulateCache
printfn "[F#]Quot Size, %d" m_cache.Count
let quot = m_cache.["MSFT.OQ"]
// The rest of the sample stays the same
In the EstablishConnection function, you definitely don't need to declare a mutable variable sr, because in case of an exception, the function will return null. I would instead use option type to make sure that this case is handled:
let EstablishConnection() =
let pipeServer =
new NamedPipeServerStream("testpipe", PipeDirection.InOut, 4)
printfn "[F#] NamedPipeServerStream thread created..."
pipeServer.WaitForConnection()
printfn "[F#] Client connected."
try // Wrap the result in 'Some' to denote success
Some(new StreamReader(pipeServer))
with e ->
printfn "[F#]ERROR: %s" e.Message
// Return 'None' to denote a failure
None
The main loop can be written using a recursive function that stops when EstablishConnection fails:
let rec loop() =
match EstablishConnection() with
| Some(conn) ->
printfn "[F#] Ready to Receive data"
// rest of the code
loop() // continue looping
| _ -> () // Quit
Just a couple thoughts...
You probably want a 'use' rather than a 'let' in a few places, as I think some of the objects in the program are IDisposable.
You may consider wrapping the EstablishConnection method and the final while loop in async blocks (and make other minor changes), so that e.g. you can wait asynchronously for connections without blocking a thread.
At first glance it is written in imperative style rather than functional style, which does make sense given that most of the program involves side effects (i.e. I/O). Line for line, it almost looks like a C# program.
Given the amount of I/O that is taking place, I don't know that there is much you can do to this particular program to make it more of a functional style of coding.