I'm hopeful that someone could potentially post an example of using FParsec where the data is based on some sort of incoming live stream.
Some examples could be producing a result based on mouse gestures, generating an alert or notification based on a specific sequence of stock ticks.
If someone could post an example it would be greatly appreciated.
Thanks!
What you're looking for is the Reactive Parsers out of Rxx.
This isn't F# but rather a .NET library that let's you write code such as (following from your stock example):
var alerts = ticks.Parse(parser =>
from next in parser
let ups = next.Where(tick => tick.Change > 0)
let downs = next.Where(tick => tick.Change < 0)
let downAlert = from manyUps in ups.AtLeast(2).ToList()
from reversalDown in downs.NonGreedy()
where reversalDown.Change <= -11
select new StockAlert(manyUps, reversalDown)
let upAlert = from manyDowns in downs.AtLeast(2).ToList()
from reversalUp in ups.NonGreedy()
where reversalUp.Change >= 21
select new StockAlert(manyDowns, reversalUp)
select downAlert.Or(upAlert).Ambiguous(untilCount: 1));
Credit of course goes to Dave Sexton and James Miles who did the majority of this work.
For background reading, the parser extensions to Rxx came out of this discussion: http://qa.social.msdn.microsoft.com/Forums/eu/rx/thread/0f72e5c0-1476-4969-92da-633000346d0d
Here's a very simple example of how this could be used in F#:
open Rxx.Parsers.Reactive
open Rxx.Parsers.Reactive.Linq
// F# shortcuts to Rxx
let where f (a:IObservableParser<_,_>) = a.Where(fun b -> f b)
let toList (parser:IObservableParser<_,_>) = parser.ToList()
let (<&>) (a:IObservableParser<'a,'b>) (b:IObservableParser<'a,'b>) = a.And(b)
let create a =
{ new ObservableParser<_,_>() with
override x.Start = a(x.Next) } :> IObservableParser<_,_>
let parse (parser:IObservableParser<_,_>) (obs:IObservable<_>) = obs.Parse(parser)
// example of grammar
let grammar =
(fun (parser:IObservableParser<_,_>) ->
let next = parser.Next
let bigs = next |> where(fun i -> i > 25)
let smalls = next |> where(fun i -> i <= 25)
bigs <&> smalls |> toList )
|> create
// the test
let random = Random()
let values = Observable.Interval(TimeSpan.FromMilliseconds(500.0)).Select( fun _ -> random.Next(1,50)).Trace().TraceSubscriptions("subbing","subbed","disposing","disposed").Publish()
let sub = values |> parse grammar |> Observable.add(printfn "BIG THEN SMALL: %A")
let test = values.Connect()
Related
Suppose I have a two step process. First data collection/cleaning and second some operation.
For example:
#r "nuget: Deedle"
open Deedle
type Person =
{ Name:string; Birthday:DateTime}
let fixB b =
if b > DateTime(2023,01,01) then OptionalValue.Missing else OptionalValue b
let peopleRecds = [ { Name = "Joe"; Birthday = DateTime(9999,12,31) }
{ Name = "Jim"; Birthday = DateTime(2000,12,31) }]
let df = Frame.ofRecords peopleRecds
let step1 = df.Clone()
step1.ReplaceColumn("Birthday", df |> Frame.mapRowValues (fun row -> fixB (row.GetAs<DateTime>"Birthday")))
step1.SaveCsv(__SOURCE_DIRECTORY__ + "step1.csv")
let step1' = Frame.ReadCsv(__SOURCE_DIRECTORY__ + "step1.csv")
step1.Print()
Name Birthday
0 -> Joe <missing>
1 -> Jim 12/31/2000 12:00:00 AM
If I save it (step1') or not (step1), I would like to continue without having to deal with different cases in step2.
let payout b =
match b with
| OptionalValue.Present c -> if c > DateTime(2000,01,01) then 100 else 0
| OptionalValue.Missing -> 0
let step2 = step1.Clone()
step2.AddColumn("Payout", step1 |> Frame.mapRowValues (fun row -> payout (row.TryGetAs<DateTime>"Birthday")))
Error: System.InvalidCastException: Object must implement IConvertible.
The first issue is that the way you use mapRowValues introduces optional values into the data frame (this is something that is often automatically eliminated, but not in this case it seems). OptionValue<'T> does not implement IConvertible, so this later causes issues. You can solve this by calculating birthday as follows:
let fixB b =
if b > DateTime(2023,01,01) then None else Some b
let bday =
df.Columns.["Birthda y"].As<DateTime>()
|> Series.mapAll (fun _ v -> Option.bind fixB v)
step1.ReplaceColumn("Birthday", bday)
The second issue with saving and loading data frame is that the CSV parser does not seem to automatically figure out that Birthday is DateTime. You can solve this by adding an explicit schema (and you can also disable saving of keys to make sure the frame you load is exactly the same as the one you save):
step1.SaveCsv(__SOURCE_DIRECTORY__ + "step1.csv",includeRowKeys=false)
let step1' = Frame.ReadCsv(__SOURCE_DIRECTORY__ + "step1.csv", schema="string,date")
I want to sort items of a class and collect them in Collection-Classes that beside a List-Member also contain further information that are necessary for the sorting process.
The following example is a a very simplified example for my problem. Although it doesn't make sense, I hope it still can help to understand my Question.
type ItemType = Odd|Even //realworld: more than two types possible
type Item(number) =
member this.number = number
member this.Type = if (this.number % 2) = 0 then Even else Odd
type NumberTypeCollection(numberType:ItemType , ?items:List<Item>) =
member this.ItemType = numberType
member val items:List<Item> = defaultArg items List.empty<Item> with get,set
member this.append(item:Item) = this.items <- item::this.items
let addToCollection (collections:List<NumberTypeCollection>) (item:Item) =
let possibleItem =
collections
|> Seq.where (fun c -> c.ItemType = item.Type) //in my realworld code, several groups may be returned
|> Seq.tryFind(fun _ -> true)
match possibleItem with
|Some(f) -> f.append item
collections
|None -> NumberTypeCollection(item.Type, [item]) :: collections
let rec findTypes (collections:List<NumberTypeCollection>) (items:List<Item>) =
match items with
| [] -> collections
| h::t -> let newCollections = ( h|> addToCollection collections)
findTypes newCollections t
let items = [Item(1);Item(2);Item(3);Item(4)]
let finalCollections = findTypes List.empty<NumberTypeCollection> items
I'm unsatisfied with the addToCollection method, since it requires the items in NumberTypeCollection to be mutual. Maybe there are further issues.
What can be a proper functional solution to solve this issue?
Edit: I'm sorry. May code was too simplified. Here is a little more complex example that should hopefully illustrate why I chose the mutual class-member (although this could still be the wrong decision):
open System
type Origin = Afrika|Asia|Australia|Europa|NorthAmerika|SouthAmerica
type Person(income, taxrate, origin:Origin) =
member this.income = income
member this.taxrate = taxrate
member this.origin = origin
type PersonGroup(origin:Origin , ?persons:List<Person>) =
member this.origin = origin
member val persons:List<Person> = defaultArg persons List.empty<Person> with get,set
member this.append(person:Person) = this.persons <- person::this.persons
//just some calculations to group people into some subgroups
let isInGroup (person:Person) (personGroup:PersonGroup) =
let avgIncome =
personGroup.persons
|> Seq.map (fun p -> float(p.income * p.taxrate) / 100.0)
|> Seq.average
Math.Abs ( (avgIncome / float person.income) - 1.0 ) < 0.5
let addToGroup (personGroups:List<PersonGroup>) (person:Person) =
let possibleItem =
personGroups
|> Seq.where (fun p -> p.origin = person.origin)
|> Seq.where (isInGroup person)
|> Seq.tryFind(fun _ -> true)
match possibleItem with
|Some(f) -> f.append person
personGroups
|None -> PersonGroup(person.origin, [person]) :: personGroups
let rec findPersonGroups (persons:List<Person>) (personGroups:List<PersonGroup>) =
match persons with
| [] -> personGroups
| h::t -> let newGroup = ( h|> addToGroup personGroups)
findPersonGroups t newGroup
let persons = [Person(1000,20, Afrika);Person(1300,22,Afrika);Person(500,21,Afrika);Person(400,20,Afrika)]
let c = findPersonGroups persons List.empty<PersonGroup>
What I may need to emphasize: There can be several different groups with the same origin.
Tomas' solution using groupby is the optimal approach if you want to generate your collections only once, it's a simple and concise.
If you want to be able to add/remove items in a functional, referentially transparent style for this type of problem, I suggest you move away from seq and start using Map.
You have a setup which is fundamentally dictionary-like. You have a unique key and a value. The functional F# equivalent to a dictionary is a Map, it is an immutable data structure based on an AVL tree. You can insert, remove and search in O(log n) time. When you append/remove from the Map, the old Map is maintained and you receive a new Map.
Here is your code expressed in this style
type ItemType =
|Odd
|Even
type Item (number) =
member this.Number = number
member this.Type = if (this.Number % 2) = 0 then Even else Odd
type NumTypeCollection = {Items : Map<ItemType, Item list>}
/// Functions on NumTypeCollection
module NumberTypeCollection =
/// Create empty collection
let empty = {Items = Map.empty}
/// Append one item to the collection
let append (item : Item) numTypeCollection =
let key = item.Type
match Map.containsKey key numTypeCollection.Items with
|true ->
let value = numTypeCollection.Items |> Map.find key
let newItems =
numTypeCollection.Items
|> Map.remove key
|> Map.add key (item :: value) // append item
{Items = newItems }
|false -> {Items = numTypeCollection.Items |> Map.add key [item]}
/// Append a list of items to the collections
let appendList (item : Item list) numTypeCollection =
item |> List.fold (fun acc it -> append it acc) numTypeCollection
Then call it using:
let items = [Item(1);Item(2);Item(3);Item(4)]
let finalCollections = NumberTypeCollection.appendList items (NumberTypeCollection.empty)
If I understand your problem correctly, you're trying to group the items by their type. The easiest way to do that is to use the standard library function Seq.groupBy. The following should implement the same logic as your code:
items
|> Seq.groupBy (fun item -> item.Type)
|> Seq.map (fun (key, values) ->
NumberTypeCollection(key, List.ofSeq values))
Maybe there are further issues.
Probably. It's difficult to tell, since it's hard to detect the purpose of the OP code... still:
Why do you even need an Item class? Instead, you could simply have a itemType function:
let itemType i = if i % 2 = 0 then Even else Odd
This function is referentially transparent, which means that you can replace it with its value if you wish. That makes it as good as a property getter method, but now you've already saved yourself from introducing a new type.
Why define a NumberTypeCollection class? Why not a simple record?
type NumberTypeList = { ItemType : ItemType; Numbers : int list }
You can implement addToCollection like something like this:
let addToCollection collections i =
let candidate =
collections
|> Seq.filter (fun c -> c.ItemType = (itemType i))
|> Seq.tryHead
match candidate with
| Some x ->
let x' = { x with Numbers = i :: x.Numbers }
collections |> Seq.filter ((<>) x) |> Seq.append [x']
| None ->
collections |> Seq.append [{ ItemType = (itemType i); Numbers = [i] }]
Being immutable, it doesn't mutate the input collections, but instead returns a new sequence of NumberTypeList.
Also notice the use of Seq.tryHead instead of Seq.tryFind(fun _ -> true).
Still, if you're attempting to group items, then Tomas' suggestion of using Seq.groupBy is more appropriate.
I am trying out the samples for FsSql and I seem to be stuck on how to properly use the Sql.execReaderF function. The example code uses an int parameter but I have a string. The following code blocks show my attempts. Does FsSql only support int for this function maybe?
Setup code:
module FsSqlTests
open System
open System.Data
open System.Data.SqlClient
open NUnit.Framework
open Swensen.Unquote
let openConn() =
let conn = new SqlConnection(#"Data Source=MYSERVER;Initial Catalog=MYDB;Integrated Security=True")
conn.Open()
conn :> IDbConnection
let connMgr = Sql.withNewConnection openConn
let P = Sql.Parameter.make
let execReader sql = Sql.execReader connMgr sql
let execReaderf sql = Sql.execReaderF connMgr sql
Using Sql.execReader (Test case passes using this one)
let selectSummaryByeFolderName eFolderName =
execReader "select summary from ework.V_DQ_Iccm_Activity_By_Team WHERE efoldername = #eFolderName"
[P("#eFolderName", eFolderName)]
Using Sql.execReaderF (Test case fails using this one)
let selectSummaryByeFolderName =
execReaderf "select summary from ework.V_DQ_Iccm_Activity_By_Team WHERE efoldername = '%s'"
Calling code in the test case:
[<TestCase>]
let ``Gets CM summary given eFolderName``() =
let c = selectSummaryByeFolderName "CM008671"
let r = c
|> Seq.ofDataReader
|> Seq.map(fun dr ->
let s =
match dr?summary with
| None -> "No Summary"
| Some x -> x
s)
|> Seq.length
test <# r > 0 #>
How can I modify my call to execReaderF to make it pass the parameter and run correctly?
UPDATE:
I tried it out with an integer parameter and it works fine. It seems the function may only support integers.
let selectSummaryByCallPriority =
execReaderf "select top 10 summary from ework.V_DQ_Iccm_Activity_By_Team WHERE callpriority = %d"
I had a look at the implementation to try and verify this but it's over my head. Anyway the Sql.execReader function works fine for other datatypes so I can just switch to that function for my string parameters.
I wrote this script to plot the historical financial data:
open FSharp.Data
#load "C:\Users\Nick\Documents\Visual Studio 2013\Projects\TryFsharp\packages\FSharp.Charting.0.90.9\FSharp.Charting.fsx"
open FSharp.Charting
open System
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s="+nasdaqcode
let company = CsvFile.Load(url)
let companyPrices = [ for r in company.Rows -> r.GetColumn "Date", r.GetColumn "Close" ]
(companyPrices
|> List.sort
|> Chart.Line).WithTitle(nasdaqcode, InsideArea=false)
plotprice "MSFT"
plotprice "ORCL"
plotprice "GOOG"
plotprice "NTES"
This works well.
Question:
Some of the data starts from the year 1986, some from 2000. I would like to plot the data from year 2000 to 2015. How to select this time period?
Is it possible to display the time when the mouse hovers over the chart?
If you are accessing Yahoo data, then it's better to use the CsvProvider rather than using CsvFile from F# Data. You can find more about the type provider here. Sadly, the naming in the standard F# Data library and on TryFSharp.org is different, so this is a bit confusing.
The CSV type provider will automatically infer the types:
open FSharp.Data
open FSharp.Charting
open System
// Generate type based on a sample
type Stocks = CsvProvider<"http://ichart.finance.yahoo.com/table.csv?s=FB">
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s=" + nasdaqcode
let company = Stocks.Load(url)
// Now you can access the columns in a statically-typed way
// and the types of the columns are inferred from the sample
let companyPrices = [ for r in company.Rows -> r.Date, r.Close ]
// If you want to do filtering, you can now use the `r.Date` property
let companyPrices =
[ for r in company.Rows do
if r.Date > DateTime(2010, 1, 1) && r.Date < DateTime(2011, 1, 1) then
yield r.Date, r.Close ]
// Charting as before
companyPrices |> (...)
I'm not sure if the F# Charting library has a way for showing the price based on mouse pointer location - it is based on standard .NET Windows Forms charting controls, so you could have a look at the documentation for the underlying library.
1) GetColumn gets a string. You need to first convert it to DateTime and simply compare it. i.e.
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s="+nasdaqcode
let company = CsvFile.Load(url)
let companyPrices = [ for r in company.Rows -> DateTime.Parse(r.GetColumn "Date"), r.GetColumn "Close" ]
(companyPrices
|> List.filter (fun (date, _) -> date > DateTime(2000, 1, 1))
|> List.sort
|> Chart.Line).WithTitle(nasdaqcode, InsideArea=false)
2) You can try with adding labels (not sure how to do on hover though...)
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s="+nasdaqcode
let company = CsvFile.Load(url)
let companyPrices = [ for r in company.Rows -> DateTime.Parse(r.GetColumn "Date"), r.GetColumn "Close" ]
(companyPrices
|> List.filter (fun (date, _) -> date > DateTime(2000, 1, 1))
|> List.sort
|> fun data -> Chart.Line(data, Labels=(Seq.map (fst >> string) data))).WithTitle(nasdaqcode, InsideArea=false)
I thought I could force to retrieve all results through multiple page and skip, using the statistics function
type Linq.IRavenQueryable<'T>
with member q.getAll() = let mutable stat = Linq.RavenQueryStatistics()
let total = stat.TotalResults
let a = q.Statistics(&stat)
let rec addone n = seq { yield q.Skip(n*1024).Take(1024).ToArray()
if n*1024 < total then
yield! addone (n + 1) }
addone 0 |> Array.concat
It works when you do
let q = session.Query<productypfield>()
let r = q.getAll()
but breaks with
let q = session.Query<productypfield>().Where(System.Func ....)
let r = q.getAll()
As the type Linq.IRavenQueryable is not idempotent through Linq composition : If I use Linq, I get an IEnumerable on which no q.Statistics(&stat) is defined.
I read the doc, and I dont see anyway to keep the type through Linq composition.
IS the only way to loop a fixed (high) amount of times, or set a high servepagesize, and take(a lot of elements) ?
edit : actually, even the code above does not work as apparently, to get a valid count, you need to run the query once. one has to call Take(0) to trigger it.
use session = store.OpenSession()
let q = session.Query<productypfield>()
let mutable s = Linq.RavenQueryStatistics()
let a = q.Statistics(&s)
s.TotalResults = 0 //true
printfn "%A" a //triggers evaluation
s.TotalResults = 0 //false
Can you change your 2nd code sample to this (I'm not familiar with F#):
let q = session.Query<productypfield>().Where(Expression<System.Func<....>>)
let r = q.getAll()
that should let you keep the IQueryable that you need