My coworker and I were comparing the speed of C# functions when passing in lambdas to do work compared to inlined functions with regards to time spent doing work. We found that you incurred a cost when passing in a lambda projection to a C# select function (for example) and wanted to see if F# had the same issues, or if it did something different.
Regardless of our original intent, we stumbled onto something that we can't figure out. In the following example we sum a list 3 different ways
Reduce
Sum
SumBy
module fs
open NUnit.Framework
open FsUnit
open System
open System.Diagnostics;
[<Test>]
let sumTest() =
let nums = [0..1000]
let repeat = 100000
let stopWatch = new Stopwatch()
stopWatch.Start()
let sumsReduce =
[
for i in [0..repeat] do
yield List.reduce (+) nums
]
Console.WriteLine("reduce = {0} - Time = {1}", List.head sumsReduce, stopWatch.Elapsed.TotalSeconds);
stopWatch.Restart()
let sumsSum =
[
for i in [0..repeat] do
yield List.sum nums
]
Console.WriteLine("sum = {0} - Time = {1}", List.head sumsSum, stopWatch.Elapsed.TotalSeconds);
stopWatch.Restart()
let sumsSumBy =
[
for i in [0..repeat] do
yield List.sumBy id nums
]
Console.WriteLine("sumBy = {0} - Time = {1}", List.head sumsSumBy, stopWatch.Elapsed.TotalSeconds);
stopWatch.Restart()
The output to this looks like:
reduce = 500500 - Time = 0.2725156
sum = 500500 - Time = 1.1183165
sumBy = 500500 - Time = 1.1126781
So clearly reduce is the big winner here. In the decompilation, I can see that reduce gets boiled down
[Serializable]
internal class sumsReduce\u004021\u002D1 : OptimizedClosures.FSharpFunc<int, int, int>
{
internal sumsReduce\u004021\u002D1()
{
base.\u002Ector();
}
public override int Invoke(int x, int y)
{
return x + y;
}
}
But I'm having a hard time figuring out what sum and sumBy are doing. Where is the timing discrepancy from?
The current answer suggested that reduce is 5 times faster because originally I was giving reduce an unchecked operator. However, updating the test to use a checked operator (from the Checked module) and I still get the same result
let sumsReduce =
[
for i in [0..repeat] do
yield List.reduce (Checked.(+)) nums
]
Notice the time discrepancy still exists
reduce = 500500 - Time = 0.274697
sum = 500500 - Time = 1.1126796
sumBy = 500500 - Time = 1.1370642
Sum and SumBy use an enumerator:
while e.MoveNext() do
acc <- Checked.(+) acc e.Current
acc
whereas reduce uses an recursive loop with an optimised closure: (reduce uses fold under the cover's - fold f head tail)
let fold<'T,'State> f (s:'State) (list: 'T list) =
match list with
| [] -> s
| _ ->
let f = OptimizedClosures.FSharpFunc<_,_,_>.Adapt(f)
let rec loop s xs =
match xs with
| [] -> s
| h::t -> loop (f.Invoke(s,h)) t
loop s list
Using optimised closures can often yield a performance boost.
sum and sumBy use checked arithmetic, but you're passing unchecked operator + to reduce – not exactly apples to apples.
Related
I would like to join a record to the next record at least X days/minutes/seconds into the future. I need to do this with arrays with a few hundred thousand records. I am open to sequences/lists/arrays but I believe arrays are likely to be fastest.
I can do this quickly in Deedle with Frame.joinAlign JoinKind.Left Lookup.ExactOrGreater, but I have an easier time reasoning about transformations using standard arrays/sequences/lists.
The following example is fine with 1000 records but very slow when 100k. A comment here suggests a binary search but I do not see how to do that here where the search is based on an inequality.
type Test1 = {
Date : DateTime
Value : float
}
type Test2 = {
Date1 :DateTime
Value1 : float
Date2 : DateTime
Value2 : float
}
let rng = System.Random()
let rng2 = System.Random()
let rs =
[| for i = 1 to 1000 do
let baseDay = DateTime(2016,1,1).AddDays(float i)
let actualDay = baseDay.AddDays(float (rng2.Next(7)))
yield {Date = actualDay; Value = rng.NextDouble() }|]
[| for r in rs do
let futureDay = r.Date.AddDays(float 4)
let r2 =
rs
|> Array.filter (fun x -> x.Date > futureDay)
|> Array.tryHead
let nr =
match r2 with
| Some x -> Some {Date1 = r.Date;Value1 = r.Value; Date2=x.Date;Value2 = x.Value}
| None -> None
if nr.IsSome then yield nr.Value |]
The problem is this expression:
let r2 =
rs
|> Array.filter (fun x -> x.Date > futureDay)
|> Array.tryHead
This filters the entire array and creates a new array with all the matching items, when you really just want the first matching item. And this is happening for every r. Try this instead:
let r2 = rs |> Array.tryFind (fun x -> x.Date > futureDay)
N.b. your logic would have been fine if you were dealing with sequences rather than arrays as the filter would have been evaluated lazily, but of course sequences are going to be slower than arrays in general. The thing to keep in mind is that whereas the Seq module is lazy (with some exceptions), when using the Array and List (and Set and Map, etc.) modules, every step in the chain/pipeline will eagerly allocate a new list/array and consequently can be very expensive when working with large collections.
If sorting rs doesn't affect your logic or expected output, a further improvement can be made by using Array.FindIndex to start searching at r's index rather than from the beginning of the array each time:
Array.sortInPlace rs
rs
|> Seq.mapi (fun i r ->
let futureDay = r.Date.AddDays 4.0
let r2Index = Array.FindIndex (rs, i, (fun x -> x.Date > futureDay))
match r2Index with
| -1 -> None
| i' -> let x = rs.[i']
Some { Date1=r.Date; Value1=r.Value; Date2=x.Date; Value2=x.Value })
|> Seq.choose id
|> Array.ofSeq
This should offer a significant improvement over even the Array.tryFind approach as only a handful of array elements will need to be scanned each time.
Here are FSI timings from my ageing tablet with the system under otherwise nil load:
10k elements:
unsorted + Array.filter (original code): 00:00:08.783
unsorted + Array.tryFind: 00:00:03.844
Array.sort + Seq.mapi: 00:00:00.027
100k elements:
unsorted + Array.filter: I didn't bother.
unsorted + Array.tryFind: 00:06:14.288
Array.sort + Seq.mapi: 00:00:00.305
While the code I have does what I need, I feel I am missing some coding technique to implement this kind of stuff in a more concise way.
The goal is to compose items and give them an Id value along the way.
Here the code which I feel can be simplified and improved in many ways. If only I knew, how...
type Foo = | A of int | B of int | C of int
let ids startvalue = Seq.initInfinite (fun i -> i + startvalue)
let makeA ids =
A(Seq.head ids), Seq.skip 1 ids
let makeB ids =
B(Seq.head ids), Seq.skip 1 ids
let makeC ids =
C(Seq.head ids), Seq.skip 1 ids
let config = [makeA; makeA; makeC; makeB]
let create (ids : seq<int>) (cfg : (seq<int> -> Foo * seq<int>) list) : Foo list * seq<int> =
let rec cre ids1 acc cl =
match cl with
| [] -> (acc,ids1)
| x::xs ->
let v,ids2 = x ids1
cre ids2 (acc # [v]) xs
cre ids [] cfg
let result : Foo list * seq<int> = create (ids 0) config
Which results in the very simple:
val result : Foo list * seq = ([A 0; A 1; C 2; B 3], )
Somehow I feel there should be an easier way to accomplish the same.
In fact, I know one way to make it simpler but this would involve mutable state and memoization (and would thusly be probably considered worse):
let idgen startvalue =
let v = ref startvalue
fun () ->
let result = !v
v := !v + 1
result
With the thusly received generator function I could get rid of all those tuples, at least and I could also get rid of create function and simply write:
let ids = idgen 0
let result =
[
A(ids())
A(ids())
C(ids())
B(ids())
]
But there should also exist a "functional" way to get it done more simply.
It seems that what you want is to take two sequences, one of functions, the other of arguments, and produce new sequence by applying functions to corresponding arguments, where in your particular case arguments are successive integer numbers and functions are union case constructors. Would that be a correct assessment?
If so, here's what I would do:
let create args funs =
Seq.zip args funs
|> Seq.map (fun (arg, fn) -> fn arg)
|> List.ofSeq
let result = create (ids 0) [A; A; C; B]
type Foo = | A of int | B of int | C of int
let ids startvalue = Seq.initInfinite (fun i -> i + startvalue)
let config = [A; A; C; B]
let create ids i cfg =
let ids' = ids i
let nextI = i + List.length cfg
(Seq.map2 id cfg ids'), nextI
let result, nextI = create ids 0 config
let result2, nextI2 = create ids nextI config
There are several considerations here:
A, B, and C are constructors of the type int -> Foo. You may directly use them as functions and eliminate repetitive wrapper functions;
Seq.map2 is able to process sequences of different length and ignore the remaining elements of the longer one;
id is a shortcut for fun f number -> f number, please use the longer one if you find id unclear
You may also refactor ids function into Seq.initInfinite id |> Seq.skip startvalue.
At the end, you may convert a sequence into a List if you really need.
Updated to receive an keep i and nextI.
As Fyodor pointed out in his answer, you really just want to apply your constructors to consecutive integers. You can use the built-in mapi for that, writing your entire program as:
type Foo = | A of int | B of int | C of int
let config = [A; A; B; C]
let create offset = List.mapi (fun i f -> f (offset + i))
create 78 config
// val it : Foo list = [A 78; A 79; B 80; C 81]
IE,
What am I doing wrong here? Does it have to to with lists, sequences and arrays and the way the limitations work?
So here is the setup: I'm trying to generate some primes. I see that there are a billion text files of a billion primes. The question isn't why...the question is how are the guys using python calculating all of the primes below 1,000,000 in milliseconds on this post...and what am I doing wrong with the following F# code?
let sieve_primes2 top_number =
let numbers = [ for i in 2 .. top_number do yield i ]
let sieve (n:int list) =
match n with
| [x] -> x,[]
| hd :: tl -> hd, List.choose(fun x -> if x%hd = 0 then None else Some(x)) tl
| _ -> failwith "Pernicious list error."
let rec sieve_prime (p:int list) (n:int list) =
match (sieve n) with
| i,[] -> i::p
| i,n' -> sieve_prime (i::p) n'
sieve_prime [1;0] numbers
With the timer on in FSI, I get 4.33 seconds worth of CPU for 100000... after that, it all just blows up.
Your sieve function is slow because you tried to filter out composite numbers up to top_number. With Sieve of Eratosthenes, you only need to do so until sqrt(top_number) and remaining numbers are inherently prime. Suppose we havetop_number = 1,000,000, your function does 78498 rounds of filtering (the number of primes until 1,000,000) while the original sieve only does so 168 times (the number of primes until 1,000).
You can avoid generating even numbers except 2 which cannot be prime from the beginning. Moreover, sieve and sieve_prime can be merged into a recursive function. And you could use lightweight List.filter instead of List.choose.
Incorporating above suggestions:
let sieve_primes top_number =
let numbers = [ yield 2
for i in 3..2..top_number -> i ]
let rec sieve ns =
match ns with
| [] -> []
| x::xs when x*x > top_number -> ns
| x::xs -> x::sieve (List.filter(fun y -> y%x <> 0) xs)
sieve numbers
In my machine, the updated version is very fast and it completes within 0.6s for top_number = 1,000,000.
Based on my code here: stackoverflow.com/a/8371684/124259
Gets the first 1 million primes in 22 milliseconds in fsi - a significant part is probably compiling the code at this point.
#time "on"
let limit = 1000000
//returns an array of all the primes up to limit
let table =
let table = Array.create limit true //use bools in the table to save on memory
let tlimit = int (sqrt (float limit)) //max test no for table, ints should be fine
let mutable curfactor = 1;
while curfactor < tlimit-2 do
curfactor <- curfactor+2
if table.[curfactor] then //simple optimisation
let mutable v = curfactor*2
while v < limit do
table.[v] <- false
v <- v + curfactor
let out = Array.create (100000) 0 //this needs to be greater than pi(limit)
let mutable idx = 1
out.[0]<-2
let mutable curx=1
while curx < limit-2 do
curx <- curx + 2
if table.[curx] then
out.[idx]<-curx
idx <- idx+1
out
There have been several good answers both as to general trial division algorithm using lists (#pad) and in choice of an array for a sieving data structure using the Sieve of Eratosthenes (SoE) (#John Palmer and #Jon Harrop). However, #pad's list algorithm isn't particularly fast and will "blow up" for larger sieving ranges and #John Palmer's array solution is somewhat more complex, uses more memory than necessary, and uses external mutable state so is not different than if the program were written in an imperative language such as C#.
EDIT_ADD: I've edited the below code (old code with line comments) modifying the sequence expression to avoid some function calls so as to reflect more of an "iterator style" and while it saved 20% of the speed it still doesn't come close to that of a true C# iterator which is about the same speed as the "roll your own enumerator" final F# code. I've modified the timing information below accordingly. END_EDIT
The following true SoE program only uses 64 KBytes of memory to sieve primes up to a million (due to only considering odd numbers and using the packed bit BitArray) and still is almost as fast as #John Palmer's program at about 40 milliseconds to sieve to one million on a i7 2700K (3.5 GHz), with only a few lines of code:
open System.Collections
let primesSoE top_number=
let BFLMT = int((top_number-3u)/2u) in let buf = BitArray(BFLMT+1,true)
let SQRTLMT = (int(sqrt (double top_number))-3)/2
let rec cullp i p = if i <= BFLMT then (buf.[i] <- false; cullp (i+p) p)
for i = 0 to SQRTLMT do if buf.[i] then let p = i+i+3 in cullp (p*(i+1)+i) p
seq { for i = -1 to BFLMT do if i<0 then yield 2u
elif buf.[i] then yield uint32(3+i+i) }
// seq { yield 2u; yield! seq { 0..BFLMT } |> Seq.filter (fun i->buf.[i])
// |> Seq.map (fun i->uint32 (i+i+3)) }
primesSOE 1000000u |> Seq.length;;
Almost all of the elapsed time is spent in the last two lines enumerating the found primes due to the inefficiency of the sequence run time library as well as the cost of enumerating itself at about 28 clock cycles per function call and return with about 16 function calls per iteration. This could be reduced to only a few function calls per iteration by rolling our own iterator, but the code is not as concise; note that in the following code there is no mutable state exposed other than the contents of the sieving array and the reference variable necessary for the iterator implementation using object expressions:
open System
open System.Collections
open System.Collections.Generic
let primesSoE top_number=
let BFLMT = int((top_number-3u)/2u) in let buf = BitArray(BFLMT+1,true)
let SQRTLMT = (int(sqrt (double top_number))-3)/2
let rec cullp i p = if i <= BFLMT then (buf.[i] <- false; cullp (i+p) p)
for i = 0 to SQRTLMT do if buf.[i] then let p = i+i+3 in cullp (p*(i+1)+i) p
let nmrtr() =
let i = ref -2
let rec nxti() = i:=!i+1;if !i<=BFLMT && not buf.[!i] then nxti() else !i<=BFLMT
let inline curr() = if !i<0 then (if !i= -1 then 2u else failwith "Enumeration not started!!!")
else let v = uint32 !i in v+v+3u
{ new IEnumerator<_> with
member this.Current = curr()
interface IEnumerator with
member this.Current = box (curr())
member this.MoveNext() = if !i< -1 then i:=!i+1;true else nxti()
member this.Reset() = failwith "IEnumerator.Reset() not implemented!!!"
interface IDisposable with
member this.Dispose() = () }
{ new IEnumerable<_> with
member this.GetEnumerator() = nmrtr()
interface IEnumerable with
member this.GetEnumerator() = nmrtr() :> IEnumerator }
primesSOE 1000000u |> Seq.length;;
The above code takes about 8.5 milliseconds to sieve the primes to a million on the same machine due to greatly reducing the number of function calls per iteration to about three from about 16. This is about the same speed as C# code written in the same style. It's too bad that F#'s iterator style as I used in the first example doesn't automatically generate the IEnumerable boiler plate code as C# iterators do, but I guess that is the intention of sequences - just that they are so damned inefficient as to speed performance due to being implemented as sequence computation expressions.
Now, less than half of the time is expended in enumerating the prime results for a much better use of CPU time.
What am I doing wrong here?
You've implemented a different algorithm that goes through each possible value and uses % to determine if it needs to be removed. What you're supposed to be doing is stepping through with a fixed increment removing multiples. That would be asymptotically.
You cannot step through lists efficiently because they don't support random access so use arrays.
I am trying to learn F# so I paid a visit to Project Euler and I am currently working on Problem 3.
The prime factors of 13195 are 5, 7,
13 and 29.
What is the largest prime
factor of the number 600851475143?
Some things to consider:
My first priority is to learn good functional habits.
My second priority is I would like it to be fast and efficient.
Within the following code I have marked the section this question is regarding.
let isPrime(n:int64) =
let rec check(i:int64) =
i > n / 2L or (n % i <> 0L && check(i + 1L))
check(2L)
let greatestPrimeFactor(n:int64) =
let nextPrime(prime:int64):int64 =
seq { for i = prime + 1L to System.Int64.MaxValue do if isPrime(i) then yield i }
|> Seq.skipWhile(fun v -> n % v <> 0L)
|> Seq.hd
let rec findNextPrimeFactor(number:int64, prime:int64):int64 =
if number = 1L then prime else
//************* No variable
(fun p -> findNextPrimeFactor(number / p, p))(nextPrime(prime))
//*************
//************* Variable
let p = nextPrime(prime)
findNextPrimeFactor(number / p, p)
//*************
findNextPrimeFactor(n, 2L)
Update
Based off some of the feedback I have refactored the code to be 10 times faster.
module Problem3
module private Internal =
let execute(number:int64):int64 =
let rec isPrime(value:int64, current:int64) =
current > value / 2L or (value % current <> 0L && isPrime(value, current + 1L))
let rec nextPrime(prime:int64):int64 =
if number % prime = 0L && isPrime(prime, 2L) then prime else nextPrime(prime + 1L)
let rec greatestPrimeFactor(current:int64, prime:int64):int64 =
if current = 1L then prime else nextPrime(prime + 1L) |> fun p -> greatestPrimeFactor(current / p, p)
greatestPrimeFactor(number, 2L)
let execute() = Internal.execute(600851475143L)
Update
I would like to thank everyone for there advice. This latest version is a compilation of all the advice I received.
module Problem3
module private Internal =
let largestPrimeFactor number =
let rec isPrime value current =
current > value / 2L || (value % current <> 0L && isPrime value (current + 1L))
let rec nextPrime value =
if number % value = 0L && isPrime value 2L then value else nextPrime (value + 1L)
let rec find current prime =
match current / prime with
| 1L -> prime
| current -> nextPrime (prime + 1L) |> find current
find number (nextPrime 2L)
let execute() = Internal.largestPrimeFactor 600851475143L
Functional programming becomes easier and more automatic with practice, so don't sweat it if you don't get it absolutely right on the first try.
With that in mind, let's take your sample code:
let rec findNextPrimeFactor(number:int64, prime:int64):int64 =
if number = 1L then prime else
//************* No variable
(fun p -> findNextPrimeFactor(number / p, p))(nextPrime(prime))
//*************
//************* Variable
let p = nextPrime(prime)
findNextPrimeFactor(number / p, p)
//*************
Your no variable version is just weird, don't use it. I like your version with the explicit let binding.
Another way to write it would be:
nextPrime(prime) |> fun p -> findNextPrimeFactor(number / p, p)
Its ok and occasionally useful to write it like this, but still comes across as a little weird. Most of the time, we use |> to curry values without needing to name our variables (in "pointfree" style). Try to anticipate how your function will be used, and if possible, re-write it so you can use it with the pipe operator without explicit declared variables. For example:
let rec findNextPrimeFactor number prime =
match number / prime with
| 1L -> prime
| number' -> nextPrime(prime) |> findNextPrimeFactor number'
No more named args :)
Ok, now that we have that out of the way, let's look at your isPrime function:
let isPrime(n:int64) =
let rec check(i:int64) =
i > n / 2L or (n % i <> 0L && check(i + 1L))
check(2L)
You've probably heard to use recursion instead of loops, and that much is right. But, wherever possible, you should abstract away recursion with folds, maps, or higher order functions. Two reasons for this:
its a little more readable, and
improperly written recursion will result in a stack overflow. For example, your function is not tail recursive, so it'll blow up on large values of n.
I'd rewrite isPrime like this:
let isPrime n = seq { 2L .. n / 2L } |> Seq.exists (fun i -> n % i = 0L) |> not
Most of the time, if you can abstract away your explicit looping, then you're just applying transformations to your input sequence until you get your results:
let maxFactor n =
seq { 2L .. n - 1L } // test inputs
|> Seq.filter isPrime // primes
|> Seq.filter (fun x -> n % x = 0L) // factors
|> Seq.max // result
We don't even have intermediate variables in this version. Coolness!
My second priority is I would like it
to be fast and efficient.
Most of the time, F# is going to be pretty comparable with C# in terms of speed, or it's going to be "fast enough". If you find your code takes a long time to execute, it probably means you're using the wrong data structure or a bad algorithm. For a concrete example, read the comments on this question.
So, the code I've written is "elegant" in the sense that its concise, gives the correct results, and doesn't rely on any trickery. Unfortunately, its not very fast. For start:
it uses trial division to create a sequence of primes, when the Sieve of Eratosthenes would be much faster. [Edit: I wrote a somewhat naive version of this sieve which didn't work for numbers larger than Int32.MaxValue, so I've removed the code.]
read Wikipedia's article on the prime counting function, it'll give you pointers on calculating the first n primes as well as estimating the upper and lower bounds for the nth prime.
[Edit: I included some code with a somewhat naive implementation of a sieve of eratosthenes. It only works for inputs less than int32.MaxValue, so it probably isn't suitable for project euler.]
Concerning "good functional habit" or rather good practice I see three minor things. Using the yield in your sequence is a little harder to read than just filter. Unnecessary type annotations in a type inferred language leads to difficult refactoring and makes the code harder to read. Don't go overboard and try to remove every type annotation though if you're finding it difficult. Lastly making a lambda function which only takes a value to use as a temp variable reduces readability.
As far as personal style goes I prefer more spaces and only using tupled arguments when the data makes sense being grouped together.
I'd write your original code like this.
let isPrime n =
let rec check i =
i > n / 2L || (n % i <> 0L && check (i + 1L))
check 2L
let greatestPrimeFactor n =
let nextPrime prime =
seq {prime + 1L .. System.Int64.MaxValue}
|> Seq.filter isPrime
|> Seq.skipWhile (fun v -> n % v <> 0L)
|> Seq.head
let rec findNextPrimeFactor number prime =
if number = 1L then
prime
else
let p = nextPrime(prime)
findNextPrimeFactor (number / p) p
findNextPrimeFactor n 2L
Your updated code is optimal for your approach. You would have to use a different algorithm like Yin Zhu answer to go faster. I wrote a test to check to see if F# makes the "check" function tail recursive and it does.
the variable p is actually a name binding, not a variable. Using name binding is not a bad style. And it is more readable. The lazy style of nextPrime is good, and it actually prime-test each number only once during the whole program.
My Solution
let problem3 =
let num = 600851475143L
let rec findMax (n:int64) (i:int64) =
if n=i || n<i then
n
elif n%i=0L then
findMax (n/i) i
else
findMax n (i+1L)
findMax num 2L
I basically divides num from 2, 3, 4.. and don't consider any prime numbers. Because if we divides all 2 from num, then we won't be able to divide it by 4,8, etc.
on this number, my solution is quicker:
> greatestPrimeFactor 600851475143L;;
Real: 00:00:01.110, CPU: 00:00:00.702, GC gen0: 1, gen1: 1, gen2: 0
val it : int64 = 6857L
>
Real: 00:00:00.001, CPU: 00:00:00.000, GC gen0: 0, gen1: 0, gen2: 0
val problem3 : int64 = 6857L
I think that the code with the temporary binding is significantly easier to read. It's pretty unusual to create an anonymous function and then immediately apply it to a value as you do in the other case. If you really want to avoid using a temporary value, I think that the most idiomatic way to do that in F# would be to use the (|>) operator to pipe the value into the anonymous function, but I still think that this isn't quite as readable.
Background:
I have a sequence of contiguous, time-stamped data. The data-sequence has gaps in it where the data is not contiguous. I want create a method to split the sequence up into a sequence of sequences so that each subsequence contains contiguous data (split the input-sequence at the gaps).
Constraints:
The return value must be a sequence of sequences to ensure that elements are only produced as needed (cannot use list/array/cacheing)
The solution must NOT be O(n^2), probably ruling out a Seq.take - Seq.skip pattern (cf. Brian's post)
Bonus points for a functionally idiomatic approach (since I want to become more proficient at functional programming), but it's not a requirement.
Method signature
let groupContiguousDataPoints (timeBetweenContiguousDataPoints : TimeSpan) (dataPointsWithHoles : seq<DateTime * float>) : (seq<seq< DateTime * float >>)= ...
On the face of it the problem looked trivial to me, but even employing Seq.pairwise, IEnumerator<_>, sequence comprehensions and yield statements, the solution eludes me. I am sure that this is because I still lack experience with combining F#-idioms, or possibly because there are some language-constructs that I have not yet been exposed to.
// Test data
let numbers = {1.0..1000.0}
let baseTime = DateTime.Now
let contiguousTimeStamps = seq { for n in numbers ->baseTime.AddMinutes(n)}
let dataWithOccationalHoles = Seq.zip contiguousTimeStamps numbers |> Seq.filter (fun (dateTime, num) -> num % 77.0 <> 0.0) // Has a gap in the data every 77 items
let timeBetweenContiguousValues = (new TimeSpan(0,1,0))
dataWithOccationalHoles |> groupContiguousDataPoints timeBetweenContiguousValues |> Seq.iteri (fun i sequence -> printfn "Group %d has %d data-points: Head: %f" i (Seq.length sequence) (snd(Seq.hd sequence)))
I think this does what you want
dataWithOccationalHoles
|> Seq.pairwise
|> Seq.map(fun ((time1,elem1),(time2,elem2)) -> if time2-time1 = timeBetweenContiguousValues then 0, ((time1,elem1),(time2,elem2)) else 1, ((time1,elem1),(time2,elem2)) )
|> Seq.scan(fun (indexres,(t1,e1),(t2,e2)) (index,((time1,elem1),(time2,elem2))) -> (index+indexres,(time1,elem1),(time2,elem2)) ) (0,(baseTime,-1.0),(baseTime,-1.0))
|> Seq.map( fun (index,(time1,elem1),(time2,elem2)) -> index,(time2,elem2) )
|> Seq.filter( fun (_,(_,elem)) -> elem <> -1.0)
|> PSeq.groupBy(fst)
|> Seq.map(snd>>Seq.map(snd))
Thanks for asking this cool question
I translated Alexey's Haskell to F#, but it's not pretty in F#, and still one element too eager.
I expect there is a better way, but I'll have to try again later.
let N = 20
let data = // produce some arbitrary data with holes
seq {
for x in 1..N do
if x % 4 <> 0 && x % 7 <> 0 then
printfn "producing %d" x
yield x
}
let rec GroupBy comp (input:LazyList<'a>) : LazyList<LazyList<'a>> =
LazyList.delayed (fun () ->
match input with
| LazyList.Nil -> LazyList.cons (LazyList.empty()) (LazyList.empty())
| LazyList.Cons(x,LazyList.Nil) ->
LazyList.cons (LazyList.cons x (LazyList.empty())) (LazyList.empty())
| LazyList.Cons(x,(LazyList.Cons(y,_) as xs)) ->
let groups = GroupBy comp xs
if comp x y then
LazyList.consf
(LazyList.consf x (fun () ->
let (LazyList.Cons(firstGroup,_)) = groups
firstGroup))
(fun () ->
let (LazyList.Cons(_,otherGroups)) = groups
otherGroups)
else
LazyList.cons (LazyList.cons x (LazyList.empty())) groups)
let result = data |> LazyList.of_seq |> GroupBy (fun x y -> y = x + 1)
printfn "Consuming..."
for group in result do
printfn "about to do a group"
for x in group do
printfn " %d" x
You seem to want a function that has signature
(`a -> bool) -> seq<'a> -> seq<seq<'a>>
I.e. a function and a sequence, then break up the input sequence into a sequence of sequences based on the result of the function.
Caching the values into a collection that implements IEnumerable would likely be simplest (albeit not exactly purist, but avoiding iterating the input multiple times. It will lose much of the laziness of the input):
let groupBy (fun: 'a -> bool) (input: seq) =
seq {
let cache = ref (new System.Collections.Generic.List())
for e in input do
(!cache).Add(e)
if not (fun e) then
yield !cache
cache := new System.Collections.Generic.List()
if cache.Length > 0 then
yield !cache
}
An alternative implementation could pass cache collection (as seq<'a>) to the function so it can see multiple elements to chose the break points.
A Haskell solution, because I don't know F# syntax well, but it should be easy enough to translate:
type TimeStamp = Integer -- ticks
type TimeSpan = Integer -- difference between TimeStamps
groupContiguousDataPoints :: TimeSpan -> [(TimeStamp, a)] -> [[(TimeStamp, a)]]
There is a function groupBy :: (a -> a -> Bool) -> [a] -> [[a]] in the Prelude:
The group function takes a list and returns a list of lists such that the concatenation of the result is equal to the argument. Moreover, each sublist in the result contains only equal elements. For example,
group "Mississippi" = ["M","i","ss","i","ss","i","pp","i"]
It is a special case of groupBy, which allows the programmer to supply their own equality test.
It isn't quite what we want, because it compares each element in the list with the first element of the current group, and we need to compare consecutive elements. If we had such a function groupBy1, we could write groupContiguousDataPoints easily:
groupContiguousDataPoints maxTimeDiff list = groupBy1 (\(t1, _) (t2, _) -> t2 - t1 <= maxTimeDiff) list
So let's write it!
groupBy1 :: (a -> a -> Bool) -> [a] -> [[a]]
groupBy1 _ [] = [[]]
groupBy1 _ [x] = [[x]]
groupBy1 comp (x : xs#(y : _))
| comp x y = (x : firstGroup) : otherGroups
| otherwise = [x] : groups
where groups#(firstGroup : otherGroups) = groupBy1 comp xs
UPDATE: it looks like F# doesn't let you pattern match on seq, so it isn't too easy to translate after all. However, this thread on HubFS shows a way to pattern match sequences by converting them to LazyList when needed.
UPDATE2: Haskell lists are lazy and generated as needed, so they correspond to F#'s LazyList (not to seq, because the generated data is cached (and garbage collected, of course, if you no longer hold a reference to it)).
(EDIT: This suffers from a similar problem to Brian's solution, in that iterating the outer sequence without iterating over each inner sequence will mess things up badly!)
Here's a solution that nests sequence expressions. The imperitave nature of .NET's IEnumerable<T> is pretty apparent here, which makes it a bit harder to write idiomatic F# code for this problem, but hopefully it's still clear what's going on.
let groupBy cmp (sq:seq<_>) =
let en = sq.GetEnumerator()
let rec partitions (first:option<_>) =
seq {
match first with
| Some first' -> //'
(* The following value is always overwritten;
it represents the first element of the next subsequence to output, if any *)
let next = ref None
(* This function generates a subsequence to output,
setting next appropriately as it goes *)
let rec iter item =
seq {
yield item
if (en.MoveNext()) then
let curr = en.Current
if (cmp item curr) then
yield! iter curr
else // consumed one too many - pass it on as the start of the next sequence
next := Some curr
else
next := None
}
yield iter first' (* ' generate the first sequence *)
yield! partitions !next (* recursively generate all remaining sequences *)
| None -> () // return an empty sequence if there are no more values
}
let first = if en.MoveNext() then Some en.Current else None
partitions first
let groupContiguousDataPoints (time:TimeSpan) : (seq<DateTime*_> -> _) =
groupBy (fun (t,_) (t',_) -> t' - t <= time)
Okay, trying again. Achieving the optimal amount of laziness turns out to be a bit difficult in F#... On the bright side, this is somewhat more functional than my last attempt, in that it doesn't use any ref cells.
let groupBy cmp (sq:seq<_>) =
let en = sq.GetEnumerator()
let next() = if en.MoveNext() then Some en.Current else None
(* this function returns a pair containing the first sequence and a lazy option indicating the first element in the next sequence (if any) *)
let rec seqStartingWith start =
match next() with
| Some y when cmp start y ->
let rest_next = lazy seqStartingWith y // delay evaluation until forced - stores the rest of this sequence and the start of the next one as a pair
seq { yield start; yield! fst (Lazy.force rest_next) },
lazy Lazy.force (snd (Lazy.force rest_next))
| next -> seq { yield start }, lazy next
let rec iter start =
seq {
match (Lazy.force start) with
| None -> ()
| Some start ->
let (first,next) = seqStartingWith start
yield first
yield! iter next
}
Seq.cache (iter (lazy next()))
Below is some code that does what I think you want. It is not idiomatic F#.
(It may be similar to Brian's answer, though I can't tell because I'm not familiar with the LazyList semantics.)
But it doesn't exactly match your test specification: Seq.length enumerates its entire input. Your "test code" calls Seq.length and then calls Seq.hd. That will generate an enumerator twice, and since there is no caching, things get messed up. I'm not sure if there is any clean way to allow multiple enumerators without caching. Frankly, seq<seq<'a>> may not be the best data structure for this problem.
Anyway, here's the code:
type State<'a> = Unstarted | InnerOkay of 'a | NeedNewInner of 'a | Finished
// f() = true means the neighbors should be kept together
// f() = false means they should be split
let split_up (f : 'a -> 'a -> bool) (input : seq<'a>) =
// simple unfold that assumes f captured a mutable variable
let iter f = Seq.unfold (fun _ ->
match f() with
| Some(x) -> Some(x,())
| None -> None) ()
seq {
let state = ref (Unstarted)
use ie = input.GetEnumerator()
let innerMoveNext() =
match !state with
| Unstarted ->
if ie.MoveNext()
then let cur = ie.Current
state := InnerOkay(cur); Some(cur)
else state := Finished; None
| InnerOkay(last) ->
if ie.MoveNext()
then let cur = ie.Current
if f last cur
then state := InnerOkay(cur); Some(cur)
else state := NeedNewInner(cur); None
else state := Finished; None
| NeedNewInner(last) -> state := InnerOkay(last); Some(last)
| Finished -> None
let outerMoveNext() =
match !state with
| Unstarted | NeedNewInner(_) -> Some(iter innerMoveNext)
| InnerOkay(_) -> failwith "Move to next inner seq when current is active: undefined behavior."
| Finished -> None
yield! iter outerMoveNext }
open System
let groupContigs (contigTime : TimeSpan) (holey : seq<DateTime * int>) =
split_up (fun (t1,_) (t2,_) -> (t2 - t1) <= contigTime) holey
// Test data
let numbers = {1 .. 15}
let contiguousTimeStamps =
let baseTime = DateTime.Now
seq { for n in numbers -> baseTime.AddMinutes(float n)}
let holeyData =
Seq.zip contiguousTimeStamps numbers
|> Seq.filter (fun (dateTime, num) -> num % 7 <> 0)
let grouped_data = groupContigs (new TimeSpan(0,1,0)) holeyData
printfn "Consuming..."
for group in grouped_data do
printfn "about to do a group"
for x in group do
printfn " %A" x
Ok, here's an answer I'm not unhappy with.
(EDIT: I am unhappy - it's wrong! No time to try to fix right now though.)
It uses a bit of imperative state, but it is not too difficult to follow (provided you recall that '!' is the F# dereference operator, and not 'not'). It is as lazy as possible, and takes a seq as input and returns a seq of seqs as output.
let N = 20
let data = // produce some arbitrary data with holes
seq {
for x in 1..N do
if x % 4 <> 0 && x % 7 <> 0 then
printfn "producing %d" x
yield x
}
let rec GroupBy comp (input:seq<_>) = seq {
let doneWithThisGroup = ref false
let areMore = ref true
use e = input.GetEnumerator()
let Next() = areMore := e.MoveNext(); !areMore
// deal with length 0 or 1, seed 'prev'
if not(e.MoveNext()) then () else
let prev = ref e.Current
while !areMore do
yield seq {
while not(!doneWithThisGroup) do
if Next() then
let next = e.Current
doneWithThisGroup := not(comp !prev next)
yield !prev
prev := next
else
// end of list, yield final value
yield !prev
doneWithThisGroup := true }
doneWithThisGroup := false }
let result = data |> GroupBy (fun x y -> y = x + 1)
printfn "Consuming..."
for group in result do
printfn "about to do a group"
for x in group do
printfn " %d" x