How to select a time period in the in FSharp.Data? - f#

I wrote this script to plot the historical financial data:
open FSharp.Data
#load "C:\Users\Nick\Documents\Visual Studio 2013\Projects\TryFsharp\packages\FSharp.Charting.0.90.9\FSharp.Charting.fsx"
open FSharp.Charting
open System
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s="+nasdaqcode
let company = CsvFile.Load(url)
let companyPrices = [ for r in company.Rows -> r.GetColumn "Date", r.GetColumn "Close" ]
(companyPrices
|> List.sort
|> Chart.Line).WithTitle(nasdaqcode, InsideArea=false)
plotprice "MSFT"
plotprice "ORCL"
plotprice "GOOG"
plotprice "NTES"
This works well.
Question:
Some of the data starts from the year 1986, some from 2000. I would like to plot the data from year 2000 to 2015. How to select this time period?
Is it possible to display the time when the mouse hovers over the chart?

If you are accessing Yahoo data, then it's better to use the CsvProvider rather than using CsvFile from F# Data. You can find more about the type provider here. Sadly, the naming in the standard F# Data library and on TryFSharp.org is different, so this is a bit confusing.
The CSV type provider will automatically infer the types:
open FSharp.Data
open FSharp.Charting
open System
// Generate type based on a sample
type Stocks = CsvProvider<"http://ichart.finance.yahoo.com/table.csv?s=FB">
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s=" + nasdaqcode
let company = Stocks.Load(url)
// Now you can access the columns in a statically-typed way
// and the types of the columns are inferred from the sample
let companyPrices = [ for r in company.Rows -> r.Date, r.Close ]
// If you want to do filtering, you can now use the `r.Date` property
let companyPrices =
[ for r in company.Rows do
if r.Date > DateTime(2010, 1, 1) && r.Date < DateTime(2011, 1, 1) then
yield r.Date, r.Close ]
// Charting as before
companyPrices |> (...)
I'm not sure if the F# Charting library has a way for showing the price based on mouse pointer location - it is based on standard .NET Windows Forms charting controls, so you could have a look at the documentation for the underlying library.

1) GetColumn gets a string. You need to first convert it to DateTime and simply compare it. i.e.
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s="+nasdaqcode
let company = CsvFile.Load(url)
let companyPrices = [ for r in company.Rows -> DateTime.Parse(r.GetColumn "Date"), r.GetColumn "Close" ]
(companyPrices
|> List.filter (fun (date, _) -> date > DateTime(2000, 1, 1))
|> List.sort
|> Chart.Line).WithTitle(nasdaqcode, InsideArea=false)
2) You can try with adding labels (not sure how to do on hover though...)
let plotprice nasdaqcode =
let url = "http://ichart.finance.yahoo.com/table.csv?s="+nasdaqcode
let company = CsvFile.Load(url)
let companyPrices = [ for r in company.Rows -> DateTime.Parse(r.GetColumn "Date"), r.GetColumn "Close" ]
(companyPrices
|> List.filter (fun (date, _) -> date > DateTime(2000, 1, 1))
|> List.sort
|> fun data -> Chart.Line(data, Labels=(Seq.map (fst >> string) data))).WithTitle(nasdaqcode, InsideArea=false)

Related

Set missing value with Deedle

Suppose I have a two step process. First data collection/cleaning and second some operation.
For example:
#r "nuget: Deedle"
open Deedle
type Person =
{ Name:string; Birthday:DateTime}
let fixB b =
if b > DateTime(2023,01,01) then OptionalValue.Missing else OptionalValue b
let peopleRecds = [ { Name = "Joe"; Birthday = DateTime(9999,12,31) }
{ Name = "Jim"; Birthday = DateTime(2000,12,31) }]
let df = Frame.ofRecords peopleRecds
let step1 = df.Clone()
step1.ReplaceColumn("Birthday", df |> Frame.mapRowValues (fun row -> fixB (row.GetAs<DateTime>"Birthday")))
step1.SaveCsv(__SOURCE_DIRECTORY__ + "step1.csv")
let step1' = Frame.ReadCsv(__SOURCE_DIRECTORY__ + "step1.csv")
step1.Print()
Name Birthday
0 -> Joe <missing>
1 -> Jim 12/31/2000 12:00:00 AM
If I save it (step1') or not (step1), I would like to continue without having to deal with different cases in step2.
let payout b =
match b with
| OptionalValue.Present c -> if c > DateTime(2000,01,01) then 100 else 0
| OptionalValue.Missing -> 0
let step2 = step1.Clone()
step2.AddColumn("Payout", step1 |> Frame.mapRowValues (fun row -> payout (row.TryGetAs<DateTime>"Birthday")))
Error: System.InvalidCastException: Object must implement IConvertible.
The first issue is that the way you use mapRowValues introduces optional values into the data frame (this is something that is often automatically eliminated, but not in this case it seems). OptionValue<'T> does not implement IConvertible, so this later causes issues. You can solve this by calculating birthday as follows:
let fixB b =
if b > DateTime(2023,01,01) then None else Some b
let bday =
df.Columns.["Birthda y"].As<DateTime>()
|> Series.mapAll (fun _ v -> Option.bind fixB v)
step1.ReplaceColumn("Birthday", bday)
The second issue with saving and loading data frame is that the CSV parser does not seem to automatically figure out that Birthday is DateTime. You can solve this by adding an explicit schema (and you can also disable saving of keys to make sure the frame you load is exactly the same as the one you save):
step1.SaveCsv(__SOURCE_DIRECTORY__ + "step1.csv",includeRowKeys=false)
let step1' = Frame.ReadCsv(__SOURCE_DIRECTORY__ + "step1.csv", schema="string,date")

(F#) Inbuilt function to filter a list if it does not contain a specific value

My question is regarding list filtering in F#. Is there a built in function that allows the filtering of lists where it only returns those that do not satisfy a condition?
let listOfList = [ [1;2;3;4;5]; [6;7;8;9;10]; [11;2;5;14;1] ]
let neededValue = 1
I know that F# features List.Contains() however I want to only return lists which do not satisfy the condition.
let sortedLists = listOfList |> List.filter(fun x -> x <> x.Contains(neededValue)
This obviously does not work because in this instance I'm comparing a list to whether a list contains a specific value. How would I do this? My desired output in this instance would be:
sortedLists = [ [6;7;8;9;10] ]
You were so close! Change x <> to not <|, and it will work.
let listOfList = [ [1;2;3;4;5]; [6;7;8;9;10]; [11;2;5;14;1] ]
let neededValue = 1
let sortedLists = listOfList |> List.filter(fun x -> not <| x.Contains(neededValue))
The not function allows you to negate a boolean value, so that the types in the filter expression match up.
In f# it's more idiomatic to use
List.contains neededValue x
instead of
x.Contains(neededValue)
So I would express it like this
let sortedLists =
listOfList
|> List.filter (List.contains neededValue >> not)

parse log files with f#

I'm trying to parse data from iis log files.
Each row has a date that I need like this:
u_ex15090503.log:3040:2015-09-05 03:57:45
And a name and email address I need in here:
&actor=%7B%22name%22%3A%5B%22James%2C%20Smith%22%5D%2C%22mbox%22%3A%5B%22mailto%3AJames.Smith%40student.colled.edu%22%5D%7D&
I start off by getting the correct column like this. This part works fine.
//get the correct column
let getCol =
let line = fileReader inputFile
line
|> Seq.filter (fun line -> not (line.StartsWith("#")))
|> Seq.map (fun line -> line.Split())
|> Seq.map (fun line -> line.[7],1)
|> Seq.toArray
getCol
Now I need to parse the above and get the date, name, and email, but I'm having a hard time figuring out how to do that.
So far I have this, which gives me 2 errors(below):
//split the above column at every "&"
let getDataInCol =
let line = getCol
line
|> Seq.map (fun line -> line.Split('&'))
|> Seq.map (fun line -> line.[5], 1)
|> Seq.toArray
getDataInCol
Seq.map (fun line -> line.Split('&'))
the field constructor 'Split' is not defined
The errors:
Seq.map (fun line -> line.[5], 1)
the operator 'expr.[idx]' has been used on an object of indeterminate type based on information prior to this program point.
Maybe I'm going about this all wrong. I'm very new to f# so I apologize for the sloppy code.
Something like this would get the name and email. You'll still need to parse the date.
#r "Newtonsoft.Json.dll"
open System
open System.Text.RegularExpressions
open Newtonsoft.Json.Linq
let (|Regex|_|) pattern input =
let m = Regex.Match(input, pattern)
if m.Success then Some(List.tail [ for g in m.Groups -> g.Value ])
else None
type ActorDetails =
{
Date: DateTime
Name: string
Email: string
}
let parseActorDetails queryString =
match queryString with
| Regex #"[\?|&]actor=([^&]+)" [json] ->
let jsonValue = JValue.Parse(Uri.UnescapeDataString(json))
{
Date = DateTime.UtcNow (* replace with parsed date *)
Name = jsonValue.Value<JArray>("name").[0].Value<string>()
Email = jsonValue.Value<JArray>("mbox").[0].Value<string>().[7..]
}
| _ -> invalidArg "queryString" "Invalid format"
parseActorDetails "&actor=%7B%22name%22%3A%5B%22James%2C%20Smith%22%5D%2C%22mbox%22%3A%5B%22mailto%3AJames.Smith%40student.colled.edu%22%5D%7D&"
val it : ActorDetails = {Date = 11/10/2015 9:14:25 PM;
Name = "James, Smith";
Email = "James.Smith#student.colled.edu";}

FParsec Reactive Example

I'm hopeful that someone could potentially post an example of using FParsec where the data is based on some sort of incoming live stream.
Some examples could be producing a result based on mouse gestures, generating an alert or notification based on a specific sequence of stock ticks.
If someone could post an example it would be greatly appreciated.
Thanks!
What you're looking for is the Reactive Parsers out of Rxx.
This isn't F# but rather a .NET library that let's you write code such as (following from your stock example):
var alerts = ticks.Parse(parser =>
from next in parser
let ups = next.Where(tick => tick.Change > 0)
let downs = next.Where(tick => tick.Change < 0)
let downAlert = from manyUps in ups.AtLeast(2).ToList()
from reversalDown in downs.NonGreedy()
where reversalDown.Change <= -11
select new StockAlert(manyUps, reversalDown)
let upAlert = from manyDowns in downs.AtLeast(2).ToList()
from reversalUp in ups.NonGreedy()
where reversalUp.Change >= 21
select new StockAlert(manyDowns, reversalUp)
select downAlert.Or(upAlert).Ambiguous(untilCount: 1));
Credit of course goes to Dave Sexton and James Miles who did the majority of this work.
For background reading, the parser extensions to Rxx came out of this discussion: http://qa.social.msdn.microsoft.com/Forums/eu/rx/thread/0f72e5c0-1476-4969-92da-633000346d0d
Here's a very simple example of how this could be used in F#:
open Rxx.Parsers.Reactive
open Rxx.Parsers.Reactive.Linq
// F# shortcuts to Rxx
let where f (a:IObservableParser<_,_>) = a.Where(fun b -> f b)
let toList (parser:IObservableParser<_,_>) = parser.ToList()
let (<&>) (a:IObservableParser<'a,'b>) (b:IObservableParser<'a,'b>) = a.And(b)
let create a =
{ new ObservableParser<_,_>() with
override x.Start = a(x.Next) } :> IObservableParser<_,_>
let parse (parser:IObservableParser<_,_>) (obs:IObservable<_>) = obs.Parse(parser)
// example of grammar
let grammar =
(fun (parser:IObservableParser<_,_>) ->
let next = parser.Next
let bigs = next |> where(fun i -> i > 25)
let smalls = next |> where(fun i -> i <= 25)
bigs <&> smalls |> toList )
|> create
// the test
let random = Random()
let values = Observable.Interval(TimeSpan.FromMilliseconds(500.0)).Select( fun _ -> random.Next(1,50)).Trace().TraceSubscriptions("subbing","subbed","disposing","disposed").Publish()
let sub = values |> parse grammar |> Observable.add(printfn "BIG THEN SMALL: %A")
let test = values.Connect()

F# interactive window problem

i am getting this error in the interactive window on http://www.tryfsharp.org. It works fine in visual studio and im not sure how to tackle it.Any help would be appreciated
let randomNumberGenerator count =
let rnd = System.Random()
List.init count (fun numList -> rnd.Next(0, 100))
let rec sortFunction = function
| [] -> []
| l -> let minNum = List.min l in
let rest = List.filter (fun i -> i <> minNum) l in
let sortedList = sortFunction rest in
minNum :: sortedList
let List = randomNumberGenerator 10
let sortList = sortFunction List
printfn "Randomly Generated numbers in a NON-SORTED LIST\n"
printfn "%A" List
printfn "\nSORTED LIST \n"
printfn "%A \n" sortList
error FS0039: The field, constructor or member 'init' is not defined
Aprreciate your help
You should be getting the error only when you run the code for the second time and it shoul behave the same in the TryF# console as well as locally in Visual Studio.
The problem is that you're declaring a value named List:
let List = randomNumberGenerator 10
which hides the standard module List. After you declare the value List.init tries to access a member of this List value instead of accessing a function in the standard List module.
There is a good reason for naming conventions, such as using lowercase for local variable names :-)

Resources