The sample code below in a single page Websharper application exhibits the issue encountered in my project.
Over time, there is some action that progressively takes longer. It occurs every few seconds. After an extended period, 20 mins or more, Chrome starts alerting that setTimeout and requestAnimationFrame are taking longer than 50ms.
Observing memory graphs in Chrome, there appears to be a memory leak as the usage increases even when garbage collection is manually activated. My suspicion is that this is causing load on the regular garbage collection and causing the extended execution times.
Any ideas how to find and fix this problem?
open WebSharper
open WebSharper.JavaScript
open WebSharper.JQuery
open WebSharper.UI.Next
open WebSharper.UI.Next.Client
open WebSharper.UI.Next.Html
open WebSharper.UI.Next.Notation
[<JavaScript>]
module Client =
type IndexTemplate = Templating.Template<"index.html">
type T = {
i : int
n : float
d : float
}
let Main =
JQuery.Of("#main").Empty().Ignore
let v = Var.Create {
i = 0
n = 0.0
d = 0.0
}
let rec f (n : float) =
let w = !v
v :=
{w with
i = w.i + 1
n = n
d = n - w.n
}
s()
and s () =
JS.RequestAnimationFrame f |> ignore
s()
div [
div [v.View |> View.Map (fun t -> "Frame " + string t.i) |> textView]
div [v.View |> View.Map (fun t -> sprintf "Started: %.1f" t.n) |> textView]
div [v.View |> View.Map (fun t -> sprintf "Duration: %.1fms" t.d) |> textView]
]
|> Doc.RunById "main"
I am using Websharper 3.6.20.6, WebSharper.UI.Next 3.6.18.2 and Chrome 59.0.3071.115.
Thanks for the report, I have linked it to this ticket: https://github.com/intellifactory/websharper.ui.next/issues/129
There will be a WebSharper 4 beta stack release with a fix today, and we will look into back-porting some important improvements like this to WebSharper 3.
Related
Friends, the following code runs fine when using Seq.unfold. However, List.unfold or Array.unfold (as shown below) causes the program to never terminate. I'm mostly just curious as to why that is. However, I am biased in general towards only using Arrays. Can anyone explain what is the reason for this behavior and if possible how to work within the confines of Arrays for a problem with this general structure.
open MathNet.Numerics.LinearAlgebra
open MathNet.Numerics.Distributions
let randn() = Normal.Sample(0., 1.)
let N = 100
let y = DenseVector.init N (fun _ -> 10. + sqrt(1.) * randn())
let SIM =
Array.unfold (fun (c1_, c2_) ->
let D = 1./(1. / 100. + float(N) / c2_)
let c1 = D *(0. / 100. + y.Sum() / c2_) + sqrt(D) * randn()
let a1 = (3. + float(N) / 2.)
let a2 = (0.5 + ((y-c1).PointwisePower(2.)).Sum() / 2.)
let c2 = InverseGamma.Sample(a1, a2)
Some((c1_, c2_), (c1, c2))
) (0., 1.)
|> Array.take (100000)
let result = SIM |> Array.map (fun (i, j) -> i)
I think the problem is caused by your generator since it never terminates the unfold algorithm by returning None.
Your code is working with Seq because sequences are evaluated lazily and Seq.unfold execute the generator only when you try to read a value from the sequence. The fact that the generator never terminates is not a problem because sequences can be infinite.
On the other hand, lists and arrays are not lazily evaluated and the generator is run until it returns None. With your generator, you end up with an "infinite loop".
I want to generate large xml files for testing purpose but the code I ended up with is really slow, the time grows exponentially with the number of rows I write to the file. Th example below shows that it takes milliseconds to write 100 rows, but it takes over 20 seconds to write 1000 rows (on my machine). I really can't figured out what is making this slow, since I think that writing 1000 rows shouldn't take that long. Also, writing 200 rows takes about 4 times as long as writing 100 rows which is not good. To run the code you might want to change the path for the StreamWriter.
open System.IO
open System.Diagnostics
let xmlSeq = Seq.initInfinite (fun index -> sprintf "<author><name>name%d</name><age>%d</age><books><book>book%d</book></books></author>" index index index)
let createFile (seq: string seq) numberToTake fileName =
use streamWriter = new StreamWriter("C:\\tmp\\FSharpXmlTest\\FSharpXmlTest\\" + fileName, false)
streamWriter.WriteLine("<startTag>")
let rec internalWriter (seq: string seq) (sw:StreamWriter) i (endTag:string) =
match i with
| 0 -> (sw.WriteLine(Seq.head seq);
sw.WriteLine(endTag))
| _ -> (sw.WriteLine(Seq.head seq);
internalWriter (Seq.skip 1 seq) sw (i-1) endTag)
internalWriter seq streamWriter numberToTake "</startTag>"
let funcTimer fn =
let stopWatch = Stopwatch.StartNew()
printfn "Timing started"
fn()
stopWatch.Stop()
printfn "Time elased: %A" stopWatch.Elapsed
(funcTimer (fun () -> createFile xmlSeq 100 "file100.xml"))
(funcTimer (fun () -> createFile xmlSeq 1000 "file1000.xml"))
You observed a quadratic behaviour O(n^2) on manipulating sequences. When you call Seq.skip, a brand new sequence will be created, so you implicitly traverse the remaining part. More detailed explanation could be found at https://stackoverflow.com/a/1306267.
In this example, you don't need to decompose sequences. Replacing your inner function by:
let internalWriter (seq: string seq) (sw:StreamWriter) i (endTag:string) =
for node in Seq.take i seq do
sw.WriteLine(node)
sw.WriteLine(endTag)
I can write 10000 rows in fraction of a second.
You can refactor further by remove this inner function and copy its body to the parent function.
As the link above mentioned, if you ever need decomposing sequences, LazyList should be better to use.
pad in his answer has pointed to the cause of slowdown. Another idiomatic approach might be instead of infinite sequence generating sequence of needed length with Seq.unfold, which makes the code really trivial:
let xmlSeq n = Seq.unfold (fun i ->
if i = 0 then None
else Some((sprintf "<author><name>name%d</name><age>%d</age><books><book>book%d</book></books></author>" i i i), i - 1)) n
let createFile seqLen fileName =
use streamWriter = new StreamWriter("C:\\tmp\\FSharpXmlTest\\" + fileName, false)
streamWriter.WriteLine("<startTag>")
seqLen |> xmlSeq |> Seq.iter streamWriter.WriteLine
streamWriter.WriteLine("</startTag>")
(funcTimer (fun () -> createFile 10000 "file10000.xml"))
Generating 10000 elements takes around 500ms on my laptop.
I came up with the following solution:
namespace FSharpBasics
module Program2 =
open System
open System.IO
open System.Diagnostics
let seqTest count : seq<string> =
let template = "<author>\
<name>Name {0}</name>\
<age>{0}</age>\
<books>\
<book>Book {0}</book>\
</books>\
</author>"
let row (i: int) =
String.Format (template, i)
seq {
yield "<authors>"
for x in [ 1..count ] do
yield row x
yield "</authors>"
}
[<EntryPoint>]
let main argv =
printfn "File will be written now"
let stopwatch = Stopwatch.StartNew()
File.WriteAllLines (#".\test.xml", seqTest 10000) |> ignore
stopwatch.Stop()
printf "Ended, took %f seconds" stopwatch.Elapsed.TotalSeconds
System.Console.ReadKey() |> ignore
0
It takes less than 90 milliseconds on my laptop to create a well-formed test.xml file with 10,000 authors.
My coworker and I were comparing the speed of C# functions when passing in lambdas to do work compared to inlined functions with regards to time spent doing work. We found that you incurred a cost when passing in a lambda projection to a C# select function (for example) and wanted to see if F# had the same issues, or if it did something different.
Regardless of our original intent, we stumbled onto something that we can't figure out. In the following example we sum a list 3 different ways
Reduce
Sum
SumBy
module fs
open NUnit.Framework
open FsUnit
open System
open System.Diagnostics;
[<Test>]
let sumTest() =
let nums = [0..1000]
let repeat = 100000
let stopWatch = new Stopwatch()
stopWatch.Start()
let sumsReduce =
[
for i in [0..repeat] do
yield List.reduce (+) nums
]
Console.WriteLine("reduce = {0} - Time = {1}", List.head sumsReduce, stopWatch.Elapsed.TotalSeconds);
stopWatch.Restart()
let sumsSum =
[
for i in [0..repeat] do
yield List.sum nums
]
Console.WriteLine("sum = {0} - Time = {1}", List.head sumsSum, stopWatch.Elapsed.TotalSeconds);
stopWatch.Restart()
let sumsSumBy =
[
for i in [0..repeat] do
yield List.sumBy id nums
]
Console.WriteLine("sumBy = {0} - Time = {1}", List.head sumsSumBy, stopWatch.Elapsed.TotalSeconds);
stopWatch.Restart()
The output to this looks like:
reduce = 500500 - Time = 0.2725156
sum = 500500 - Time = 1.1183165
sumBy = 500500 - Time = 1.1126781
So clearly reduce is the big winner here. In the decompilation, I can see that reduce gets boiled down
[Serializable]
internal class sumsReduce\u004021\u002D1 : OptimizedClosures.FSharpFunc<int, int, int>
{
internal sumsReduce\u004021\u002D1()
{
base.\u002Ector();
}
public override int Invoke(int x, int y)
{
return x + y;
}
}
But I'm having a hard time figuring out what sum and sumBy are doing. Where is the timing discrepancy from?
The current answer suggested that reduce is 5 times faster because originally I was giving reduce an unchecked operator. However, updating the test to use a checked operator (from the Checked module) and I still get the same result
let sumsReduce =
[
for i in [0..repeat] do
yield List.reduce (Checked.(+)) nums
]
Notice the time discrepancy still exists
reduce = 500500 - Time = 0.274697
sum = 500500 - Time = 1.1126796
sumBy = 500500 - Time = 1.1370642
Sum and SumBy use an enumerator:
while e.MoveNext() do
acc <- Checked.(+) acc e.Current
acc
whereas reduce uses an recursive loop with an optimised closure: (reduce uses fold under the cover's - fold f head tail)
let fold<'T,'State> f (s:'State) (list: 'T list) =
match list with
| [] -> s
| _ ->
let f = OptimizedClosures.FSharpFunc<_,_,_>.Adapt(f)
let rec loop s xs =
match xs with
| [] -> s
| h::t -> loop (f.Invoke(s,h)) t
loop s list
Using optimised closures can often yield a performance boost.
sum and sumBy use checked arithmetic, but you're passing unchecked operator + to reduce – not exactly apples to apples.
Disclaim: This is micro-benchmark, please do not comment quotes such as "premature optimization is evil" if you feel unhappy about the topic.
Examples are release targeted for x64, .Net4.5 Visual Studio 2012 F# 3.0 and run in windows 7 x64
After profiling, I narrowed down the bottleneck of one of my applications, so that I want to raise this question:
Observation
If there is no loop inside for in loop or Seq.iter, then it is clear they are both of similar speed. (update2 vs update4)
If there is a loop inside for in loop or Seq.iter, it seems Seq.iter is 2x as faster as for in. (update vs update3) strange? (if run in fsi they would be similar)
If it is targeted for anycpu and run in x64, there is no difference in time. So the question becomes: Seq.iter (update3) would boost up 2x speed if target is x64
Time taken:
update: 00:00:11.4250483 // 2x as much as update3, why?
updatae2: 00:00:01.4447233
updatae3: 00:00:06.0863791
updatae4: 00:00:01.4939535
Source Code:
open System.Diagnostics
open System
[<EntryPoint>]
let main argv =
let pool = seq {1 .. 1000000}
let ret = Array.zeroCreate 100
let update pool =
for x in pool do
for y in 1 .. 200 do
ret.[2] <- x + y
let update2 pool =
for x in pool do
//for y in 1 .. 100 do
ret.[2] <- x
let update3 pool =
pool
|> Seq.iter (fun x ->
for y in 1 .. 200 do
ret.[2] <- x + y)
let update4 pool =
pool
|> Seq.iter (fun x ->
//for y in 1 .. 100 do
ret.[2] <- x)
let test n =
let run = match n with
| 1 -> update
| 2 -> update2
| 3 -> update3
| 4 -> update4
for i in 1 .. 50 do
run pool
let sw = new Stopwatch()
sw.Start()
test(1)
sw.Stop()
Console.WriteLine(sw.Elapsed);
sw.Restart()
test(2)
sw.Stop()
Console.WriteLine(sw.Elapsed)
sw.Restart()
test(3)
sw.Stop()
Console.WriteLine(sw.Elapsed)
sw.Restart()
test(4)
sw.Stop()
Console.WriteLine(sw.Elapsed)
0 // return an integer exit code
This isn't a complete answer, but hope it helps you to go further.
I can reproduce the behaviour using the same configuration. Here is a simpler example for profiling:
open System
let test1() =
let ret = Array.zeroCreate 100
let pool = {1 .. 1000000}
for x in pool do
for _ in 1..50 do
for y in 1..200 do
ret.[2] <- x + y
let test2() =
let ret = Array.zeroCreate 100
let pool = {1 .. 1000000}
Seq.iter (fun x ->
for _ in 1..50 do
for y in 1..200 do
ret.[2] <- x + y) pool
let time f =
let sw = new Diagnostics.Stopwatch()
sw.Start()
let result = f()
sw.Stop()
Console.WriteLine(sw.Elapsed)
result
[<EntryPoint>]
let main argv =
time test1
time test2
0
In this example, Seq.iter and for x in pool is executed once but there is still 2x time difference between test1 and test2:
00:00:06.9264843
00:00:03.6834886
Their ILs are very similar, so compiler optimization isn't a problem. It seems that x64 jitter fails to optimize test1 though it is able to do so with test2. Interestingly, if I refactor nested for loops in test1 as a function, JIT optimization succeeds again:
let body (ret: _ []) x =
for _ in 1..50 do
for y in 1..200 do
ret.[2] <- x + y
let test3() =
let ret = Array.zeroCreate 100
let pool = {1..1000000}
for x in pool do
body ret x
// 00:00:03.7012302
When I disable JIT optimization using the technique described here, execution times of these functions are comparable.
Why x64 jitter fails in the particular example, I don't know. You can disassemble optimized jitted code to compare ASM instructions line by line. Maybe someone with good ASM knowledge can find out their differences.
When I run the experiment on my machine (using F# 3.0 in VS 2012 in Release mode), I do not get the times you describe. Do you consistently get the same numbers when you run it repeatedly?
I tried it about 4 times and I always get numbers that are very similar. The version with Seq.iter tends to be slightly faster, but this is probably not statistically significant. Something like (using Stopwatch):
test(1) = 15321ms
test(2) = 5149ms
test(3) = 14290ms
test(4) = 4999ms
I'm running the test on a laptop with Intel Core2 Duo (2.26Ghz), using 64bit Windows 7.
I am trying to build a list from a sequence by recursively appending the first element of the sequence to the list:
open System
let s = seq[for i in 2..4350 -> i,2*i]
let rec copy s res =
if (s|>Seq.isEmpty) then
res
else
let (a,b) = s |> Seq.head
Console.WriteLine(string a)
let newS = s |> Seq.skip(1)|> Seq.cache
let newRes = List.append res ([(a,b)])
copy newS newRes
copy s ([])
Two problems:
. getting a Stack overflow which means my tail recusive ploy sucks
and
. why is the code 100x faster when I put |> Seq.cache here let newS = s |> Seq.skip(1)|> Seq.cache.
(Note this is just a little exercise, I understand you can do Seq.toList etc.. )
Thanks a lot
One way that works is ( the two points still remain a bit weird to me ):
let toList (s:seq<_>) =
let rec copyRev res (enum:Collections.Generic.IEnumerator<_*_>) =
let somethingLeft = enum.MoveNext()
if not(somethingLeft) then
res
else
let curr = enum.Current
Console.WriteLine(string curr)
let newRes = curr::res
copyRev newRes enum
let enumerator = s.GetEnumerator()
(copyRev ([]) (enumerator)) |>List.rev
You say it's just an exercise, but it's useful to point to my answer to
While or Tail Recursion in F#, what to use when?
and reiterate that you should favor more applicative/declarative constructs when possible. E.g.
let rec copy2 s = [
for tuple in s do
System.Console.WriteLine(string(fst tuple))
yield tuple
]
is a nice and performant way to express your particular function.
That said, I'd feel remiss if I didn't also say "never create a list that big". For huge data, you want either array or seq.
In my short experience with F# it is not a good idea to use Seq.skip 1 like you would with lists with tail. Seq.skip creates a new IEnumerable/sequence and not just skips n. Therefore your function will be A LOT slower than List.toSeq. You should properly do it imperative with
s.GetEnumerator()
and iterates through the sequence and hold a list which you cons every single element.
In this question
Take N elements from sequence with N different indexes in F#
I started to do something similar to what you do but found out it is very slow. See my method for inspiration for how to do it.
Addition: I have written this:
let seqToList (xs : seq<'a>) =
let e = xs.GetEnumerator()
let mutable res = []
while e.MoveNext() do
res <- e.Current :: res
List.rev res
And found out that the build in method actually does something very similar (including the reverse part). It do, however, checks whether the sequence you have supplied is in fact a list or an array.
You will be able to make the code entirely functional: (which I also did now - could'nt resist ;-))
let seqToList (xs : seq<'a>) =
Seq.fold (fun state t -> t :: state) [] xs |> List.rev
Your function is properly tail recursive, so the recursive calls themselves are not what is overflowing the stack. Instead, the problem is that Seq.skip is poorly behaved when used recursively, as others have pointed out. For instance, this code overflows the stack on my machine:
let mutable s = seq { 1 .. 20001 }
for i in 1 .. 20000 do
s <- Seq.skip 1 s
let v = Seq.head s
Perhaps you can see the vague connection to your own code, which also eventually takes the head of a sequence which results from repeatedly applying Seq.skip 1 to your initial sequence.
Try the following code.
Warning: Before running this code you will need to enable tail call generation in Visual Studio. This can be done through the Build tab on the project properties page. If this is not enabled the code will StackOverflow processing the continuation.
open System
open System.Collections.Generic
let s = seq[for i in 2..1000000 -> i,2*i]
let rec copy (s : (int * int) seq) =
use e = s.GetEnumerator()
let rec inner cont =
if e.MoveNext() then
let (a,b) = e.Current
printfn "%d" b
inner (fun l -> cont (b :: l))
else cont []
inner (fun x -> x)
let res = copy s
printfn "Done"