Are the results from F#'s Async.Parallel operation guaranteed to arrive in the order jobs were submitted? My sample code returns the results in order, but I can't find any mention in the MSDN docs, or the F# spec, assuring this must be the case -- that it's not a coincidence.
Here is my sample code:
let r = System.Random()
Async.Parallel [
for i in 0..10 ->
async {
let rand_num = r.Next(10)
do! Async.Sleep(rand_num) (* Simulate jobs taking a variable amount of time *)
printfn "%i %i" i rand_num
return i
}
]
|> Async.RunSynchronously
|> printfn "%A"
And here's the output.
0 0
5 1
4 1
3 3
10 6
9 4
7 5
2 5
1 5
8 7
6 9
[|0; 1; 2; 3; 4; 5; 6; 7; 8; 9; 10|]
You can see that, in this run, the async functions complete in indeterminate order, yet the resulting array is sorted. Is this behavior guaranteed?
At the moment, the source of the function is written so that this guarantee is enforced. Looking at control.fs around line #1300 for the definition, we can see the function that puts the results into the output array is
let recordSuccess i res =
results.[i] <- res;
finishTask(Interlocked.Decrement count)
this function is called in this segment
tasks |> Array.iteri (fun i p ->
queueAsync
innerCTS.Token
// on success, record the result
(fun res -> recordSuccess i res)
where tasks has the original tasks in sorted order. This guarantees that the output list is in the same order as the input.
UPDATE
The spec at least seems to imply that the order is fixed - it contains this code:
let rec fib x = if x < 2 then 1 else fib(x-1) + fib(x-2)
let fibs =
Async.Parallel [ for i in 0..40 -> async { return fib(i) } ]
|> Async.RunSynchronously
printfn "The Fibonacci numbers are %A" fibs //I changed this line to be accurate
System.Console.ReadKey(true)
If the spec didn't guarantee the output order, this code would be incorrect.
Related
I'm trying to cache the result of an infinite sequence of triangle numbers.
let triCalc n = (n*(n+1))/2
let triNumC = 0 |> Seq.unfold (fun n -> Some((triCalc n, n+1))) |> Seq.cache
However, when I try to cache it doesn't seem to work as I expect
let x = Seq.take 4 (triNumC)
let y = Seq.take 4 (triNumC)
printf "%A" x
printf "%A" y
This prints
seq [0; 1; 3; 6]seq [0; 1; 3; 6]
Am I caching incorrectly? Shouldn't the second sequence printed be a continuation of the first? I am not quite sure how to progress.
If you want the continuation you need to skip. The sequence remains the same
let y = triNumC
|> Seq.skip 4
|> Seq.take 4
Caching is fine. The benefit of caching is that when calculating y is that starts from the 5th term onwards since the first 4 are already cached.
Why is the following function returning a sequence of incorrect length when the repl variable is set to false?
open MathNet.Numerics.Distributions
open MathNet.Numerics.LinearAlgebra
let sample (data : seq<float>) (size : int) (repl : bool) =
let n = data |> Seq.length
// without replacement
let rec generateIndex idx =
let m = size - Seq.length(idx)
match m > 0 with
| true ->
let newIdx = DiscreteUniform.Samples(0, n-1) |> Seq.take m
let idx = (Seq.append idx newIdx) |> Seq.distinct
generateIndex idx
| false ->
idx
let sample =
match repl with
| true ->
DiscreteUniform.Samples(0, n-1)
|> Seq.take size
|> Seq.map (fun index -> Seq.item index data)
| false ->
generateIndex (seq [])
|> Seq.map (fun index -> Seq.item index data)
sample
Running the function...
let requested = 1000
let dat = Normal.Samples(0., 1.) |> Seq.take 10000
let resultlen = sample dat requested false |> Seq.length
printfn "requested -> %A\nreturned -> %A" requested resultlen
Resulting lengths are wrong.
>
requested -> 1000
returned -> 998
>
requested -> 1000
returned -> 1001
>
requested -> 1000
returned -> 997
Any idea what mistake I'm making?
First, there's a comment I want to make about coding style. Then I'll get to the explanation of why your sequences are coming back with different lengths.
In the comments, I mentioned replacing match (bool) with true -> ... | false -> ... with a simple if ... then ... else expression, but there's another coding style that you're using that I think could be improved. You wrote:
let sample (various_parameters) = // This is a function
// Other code ...
let sample = some_calculation // This is a variable
sample // Return the variable
While F# allows you to reuse names like that, and the name inside the function will "shadow" the name outside the function, it's generally a bad idea for the reused name to have a totally different type than the original name. In other words, this can be a good idea:
let f (a : float option) =
let a = match a with
| None -> 0.0
| Some value -> value
// Now proceed, knowing that `a` has a real value even if had been None before
Or, because the above is exactly what F# gives you defaultArg for:
let f (a : float option) =
let a = defaultArg a 0.0
// This does exactly the same thing as the previous snippet
Here, we are making the name a inside our function refer to a different type than the parameter named a: the parameter was a float option, and the a inside our function is a float. But they're sort of the "same" type -- that is, there's very little mental difference between "The caller may have specified a floating-point value or they may not" and "Now I definitely have a floating-point value". But there's a very large mental gap between "The name sample is a function that takes three parameters" and "The name sample is a sequence of floats". I strongly recommend using a name like result for the value you're going to return from your function, rather than re-using the function name.
Also, this seems unnecessarily verbose:
let result =
match repl with
| true ->
DiscreteUniform.Samples(0, n-1)
|> Seq.take size
|> Seq.map (fun index -> Seq.item index data)
| false ->
generateIndex (seq [])
|> Seq.map (fun index -> Seq.item index data)
result
Anytime I find myself writing "let result = (something) ; result" at the end of my function, I usually just want to replace that whole code block with just the (something). I.e., the above snippet could just become:
match repl with
| true ->
DiscreteUniform.Samples(0, n-1)
|> Seq.take size
|> Seq.map (fun index -> Seq.item index data)
| false ->
generateIndex (seq [])
|> Seq.map (fun index -> Seq.item index data)
Which in turn can be replaced with an if...then...else expression:
if repl then
DiscreteUniform.Samples(0, n-1)
|> Seq.take size
|> Seq.map (fun index -> Seq.item index data)
else
generateIndex (seq [])
|> Seq.map (fun index -> Seq.item index data)
And that's the last expression in your code. In other words, I would probably rewrite your function as follows (changing ONLY the style, and making no changes to the logic):
open MathNet.Numerics.Distributions
open MathNet.Numerics.LinearAlgebra
let sample (data : seq<float>) (size : int) (repl : bool) =
let n = data |> Seq.length
// without replacement
let rec generateIndex idx =
let m = size - Seq.length(idx)
if m > 0 then
let newIdx = DiscreteUniform.Samples(0, n-1) |> Seq.take m
let idx = (Seq.append idx newIdx) |> Seq.distinct
generateIndex idx
else
idx
if repl then
DiscreteUniform.Samples(0, n-1)
|> Seq.take size
|> Seq.map (fun index -> Seq.item index data)
else
generateIndex (seq [])
|> Seq.map (fun index -> Seq.item index data)
If I can figure out why your sequences have the wrong length, I'll update this answer with that information as well.
UPDATE: Okay, I think I see what's happening in your generateIndex function that's giving you unexpected results. There are two things tripping you up: one is sequence laziness, and the other is randomness.
I copied your generateIndex function into VS Code and added some printfn statements to look at what's going on. First, the code I ran, and then the results:
let rec generateIndex n size idx =
let m = size - Seq.length(idx)
printfn "m = %d" m
match m > 0 with
| true ->
let newIdx = DiscreteUniform.Samples(0, n-1) |> Seq.take m
printfn "Generating newIdx as %A" (List.ofSeq newIdx)
let idx = (Seq.append idx newIdx) |> Seq.distinct
printfn "Now idx is %A" (List.ofSeq idx)
generateIndex n size idx
| false ->
printfn "Done, returning %A" (List.ofSeq idx)
idx
All those List.ofSeq idx calls are so that F# Interactive would print more than four items of the seq when I print it out (by default, if you try to print a seq with %A, it will only print out four values and then print an ellipsis if there are more values available in the seq). Also, I turned n and size into parameters (that I don't change between calls) so that I could test it easily. I then called it as generateIndex 100 5 (seq []) and got the following result:
m = 5
Generating newIdx as [74; 76; 97; 78; 31]
Now idx is [68; 28; 65; 58; 82]
m = 0
Done, returning [37; 58; 24; 48; 49]
val it : seq<int> = seq [12; 69; 97; 38; ...]
See how the numbers keep changing? That was my first clue that something was up. See, seqs are lazy. They don't evaluate their contents until they have to. You shouldn't think of a seq as a list of numbers. Instead, think of it as a generator that will, when asked for numbers, produce them according to some rule. In your case, the rule is "Choose random integers between 0 and n-1, then take m of those numbers". And the other thing about seqs is that they do not cache their contents (although there's a Seq.cache function available that will cache their contents). Therefore, if you have a seq based on a random number generator, its results will be different each time, as you can see in my output. When I printed out newIdx, it printed out as [74; 76; 97; 78; 31], but when I appended it to an empty seq, the result printed out as [68; 28; 65; 58; 82].
Why this difference? Because Seq.append does not force evaluation. It simply creates a new seq whose rule is "take all items from the first seq, then when that one exhausts, take all items from the second seq. And when that one exhausts, end." And Seq.distinct does not force evaluation either; it simply creates a new seq whose rule is "take the items from the seq handed to you, and start handing them out when asked. But memorize them as you go, and if you've handed one of them out before, don't hand it out again." So what you are passing around between your calls to generateIdx is an object that, when evaluated, will pick a set of random numbers between 0 and n-1 (in my simple case, between 0 and 100) and then reduce that set down to a distinct set of numbers.
Now, here's the thing. Every time you evaluate that seq, it will start from the beginning: first calling DiscreteUniform.Samples(0, n-1) to generate an infinite stream of random numbers, then selecting m numbers from that stream, then throwing out any duplicates. (I'm ignoring the Seq.append for now, because it would create unnecessary mental complexity and it isn't really part of the bug anyway). Now, at the start of each go-round of your function, you check the length of the sequence, which does cause it to be evaluated. That means that it selects (in the case of my sample code) 5 random numbers between 0 and 99, then makes sure that they're all distinct. If they are all distinct, then m = 0 and the function will exit, returning... not the list of numbers, but the seq object. And when that seq object is evaluated, it will start over from the beginning, choosing a different set of 5 random numbers and then throwing out any duplicates. Therefore, there's still a chance that at least one of that set of 5 numbers will end up being a duplicate, because the sequence whose length was tested (which we know contained no duplicates, otherwise m would have been greater than 0) was not the sequence that was returned. The sequence that was returned has a 1.0 * 0.99 * 0.98 * 0.97 * 0.96 chance of not containing any duplicates, which comes to about 0.9035. So there's a just-under-10% chance that even though you checked Seq.length and it was 5, the length of the returned seq ends up being 4 after all -- because it was choosing a different set of random numbers than the one you checked.
To prove this, I ran the function again, this time only picking 4 numbers so that the result would be completely shown at the F# Interactive prompt. And my run of generateIndex 100 4 (seq []) produced the following output:
m = 4
Generating newIdx as [36; 63; 97; 31]
Now idx is [39; 93; 53; 94]
m = 0
Done, returning [47; 94; 34]
val it : seq<int> = seq [48; 24; 14; 68]
Notice how when I printed "Done, returning (value of idx)", it had only 3 values? Even though it eventually returned 4 values (because it picked a different selection of random numbers for the actual result, and that selection had no duplicates), that demonstrated the problem.
By the way, there's one other problem with your function, which is that it's far slower than it needs to be. The function Seq.item, in some circumstances, has to run through the sequence from the beginning in order to pick the nth item of the sequence. It would be far better to store your data in an array at the start of your function (let arrData = data |> Array.ofSeq), then replace
|> Seq.map (fun index -> Seq.item index data)
with
|> Seq.map (fun index -> arrData.[index])
Array lookups are done in constant time, so that takes your sample function down from O(N^2) to O(N).
TL;DR: Use Seq.distinct before you take m values from it and the bug will go away. You can just replace your entire generateIdx function with a simple DiscreteUniform.Samples(0, n-1) |> Seq.distinct |> Seq.take size. (And use an array for your data lookups so that your function will run faster). In other words, here's the final almost-final version of how I would rewrite your code:
let sample (data : seq<float>) (size : int) (repl : bool) =
let arrData = data |> Array.ofSeq
let n = arrData |> Array.length
if repl then
DiscreteUniform.Samples(0, n-1)
|> Seq.take size
|> Seq.map (fun index -> arrData.[index])
else
DiscreteUniform.Samples(0, n-1)
|> Seq.distinct
|> Seq.take size
|> Seq.map (fun index -> arrData.[index])
That's it! Simple, easy to understand, and (as far as I can tell) bug-free.
Edit: ... but not completely DRY, because there's still a bit of repeated code in that "final" version. (Credit to CaringDev for pointing it out in the comments below). The Seq.take size |> Seq.map is repeated in both branches of the if expression, so there's a way to simplify that expression. We could do this:
let randomIndices =
if repl then
DiscreteUniform.Samples(0, n-1)
else
DiscreteUniform.Samples(0, n-1) |> Seq.distinct
randomIndices
|> Seq.take size
|> Seq.map (fun index -> arrData.[index])
So here's a truly-final version of my suggestion:
let sample (data : seq<float>) (size : int) (repl : bool) =
let arrData = data |> Array.ofSeq
let n = arrData |> Array.length
let randomIndices =
if repl then
DiscreteUniform.Samples(0, n-1)
else
DiscreteUniform.Samples(0, n-1) |> Seq.distinct
randomIndices
|> Seq.take size
|> Seq.map (fun index -> arrData.[index])
I am experienced in C# but new to F# and Functional Programming. Now I am trying to implement a class library in F#. Here is one of the functions:
It takes a list of integers <=9 and change consecutive 9 like 9,9,9,9 to 9, 10, 11, 12. For example [9;9;9;1;4;0;1;9;9;9;9] will be changed to [9; 10; 11; 1; 4; 0; 1; 9; 10; 11; 12].
C# function is trivial:
void ReleaseCap(List<int> items)
{
for (int i = 1; i < items.Count; i++)
{
var current = items[i];
var previous = items[i - 1];
//If curernt value = 9 and previous >=9, then current value should be previous+1
if (current == 9 && previous >= 9)
{
items[i] = previous + 1;
}
}
}
Now is my F# tail recursive one. Instead of loop the List by index, it recursively move item from an initial list to an processed list until everything in the initial list is gone:
let releaseCap items =
let rec loop processed remaining = //tail recursion
match remaining with
| [] -> processed //if nothing left, the job is done.
| current :: rest when current = 9 -> //if current item =9
match processed with
// previous value >= 9, then append previous+1 to the processed list
| previous :: _ when previous >= 9 -> loop (previous+1 :: processed) rest
//if previous < 9, the current one should be just 9
| _ -> loop (current :: processed) rest
//otherwise, just put the current value to the processed list
| current :: rest -> loop (current :: processed) rest
loop [] items |> List.rev
While the C# version is trivial and intuitive, the F# code is verbose and not as intuitive. Is there any part of the F# code can be improved to make it more elegant?
You can reuse existing functions in order to simplify your code.
Normally when you change items in a list you think of a map but in this case you have something to remember from your previous computation which should be passed for each item, so you should aim to fold related functions.
Here's one: List.scan
let releaseCap items =
items
|> List.scan (fun previous current ->
if current = 9 && previous >= 9 then previous + 1
else current) 0
|> List.tail
FP is not just about using recursion instead of loops. Recursion is typically used in basic and reusable functions, then by combining those functions you can solve complex problems.
NOTE: You are comparing your C# solution with your F# solution, but did you notice that apart from the language there is an important difference between both solutions? Your C# solution uses mutability.
In your C# code you correctly assume that the 1st element of the list will never change and start with the 2nd element.
If you include this in the F# code you can skip the match on accumulator.
The result will be a bit simpler.
let reduceCap l =
let rec reduceCapRec acc l =
let previous = List.head acc
match l with
| x::xs ->
if x = 9 && previous >= 9
then reduceCapRec ((previous+1) :: acc) xs
else reduceCapRec (x :: acc) xs
| [] -> acc
reduceCapRec [List.head l] (List.tail l)
|> List.rev
(Although Gustavo's solution is still much better - I'm also new to FP)
I thought fold was the important one! :-)
So:
let list = [9;9;9;1;4;0;1;9;9;9;9]
let incr acc e =
match acc, e with
| h::t, e when e = 9 && h >= 9 -> (h + 1)::acc
| _ -> e::acc
list
|> List.fold incr []
|> List.rev
//val it : int list = [9; 10; 11; 1; 4; 0; 1; 9; 10; 11; 12]
I have a sequence of prime number divisors that I want to iterate over for each prime candidate. I use GetEnumerator() MoveNext() and Current. I can't reinitialize the enumerator to start from the beginning. I tried Reset(), which compiled, but gives a runtime error of not implemented.
I am using F# 2.0 Interactive build 4.0.40219.1
Any suggestions?
Regards,
Doug
To clarify the problem: For each prime candidate N I want to iterate thru the prime divisors sequence (up to approx sqrt N) and completely factor N or determine if it is prime. Using the GetEnumerator, MoveNext, Current approach works for the first prime candidate, but on the second prime candidate I want to iterate on my divisors sequence from the beginning. It appears that the only way to do this is to create a new iterator (which is awkward for a large number of prime candidates) or create a new prime sequence (which I don't want to do).
The suggestion of using something like "divisors in seqPrimes" appears to exhaust all divisors before stopping, but I want to stop as soon as a prime divisor divides the prime candidate.
If there is an error in my logic in the above statements, please let me know.
I investigated Seq.cache, and this worked for me. The resulting code follows:
// Recursive isprime function (modified from MSDN)
let isPrime n =
let rec check i =
i > n/2 || (n % i <> 0 && check (i + 2))
if n = 2 then true
elif (n%2) = 0 then false
else check 3
let seqPrimes = seq { for n in 2 .. 100000 do if isPrime n then yield n }
// Cache the sequence to avoid recomputing the sequence elements.
let cachedSeq = Seq.cache seqPrimes
// find the divisors of n (or determine prime) using the seqEnum enumerator
let rec testPrime n (seqEnum:System.Collections.Generic.IEnumerator<int>) =
if n = 1 then printfn "completely factored"
else
let nref = ref n
if seqEnum.MoveNext() then
let divisor = seqEnum.Current
//printfn "trial divisor %A" divisor
if divisor*divisor > n then printfn "prime %A" !nref
else
while ((!nref % divisor) = 0) do
printfn "divisor %A" divisor
nref := !nref / divisor
testPrime !nref seqEnum
// test
for x = 1000000 to 1000010 do
printfn "\ndivisors of %d = " x
let seqEnum = cachedSeq.GetEnumerator()
testPrime x seqEnum
seqEnum.Dispose() // not needed
If you mean that the cause of your attempt to reset the Enumerator is the high cost of regenerating your sequence of primes you may consider caching your sequence. This manner of using your sequence would be idiomatic to F#. To show you how to do this I refer you to the following snippet taken from this context:
let rec primes =
Seq.cache <| seq { yield 2; yield! Seq.unfold nextPrime 3 }
and nextPrime n =
if isPrime n then Some(n, n + 2) else nextPrime(n + 2)
and isPrime n =
if n >= 2 then
primes
|> Seq.tryFind (fun x -> n % x = 0 || x * x > n)
|> fun x -> x.Value * x.Value > n
else false
You may play with this snippet to see that the penalty of re-enumeration here gets negligible.
Talking of Reset() method of IEnumerator, I recall that it is not implemented in current F#, i.e. throws System.NotSupportedException. See MSDN reference for justification.
ADDITION:
In order to test it with the test you've suggested below:
for x in [1000000..1000010] do
printfn "\ndivisors of %d" x
primes
|> Seq.takeWhile ((>) (int(sqrt(float x))))
|> Seq.iter (fun n -> if x%n = 0 then printf "%d " n)
On my laptop test execution takes mere 3ms.
I have this code written for a Project Euler problem in c++:
int sum = 0;
for(int i =0; i < 1000; i++)
{
//Check if multiple of 3 but not multiple of 5 to prevent duplicate
sum += i % 3 == 0 && i % 5 != 0 ? i: 0;
//check for all multiple of 5, including those of 3
sum += i % 5 == 0 ? i: 0;
}
cout << sum;
I'm trying to learn f# and rewriting this in f#. This is what I have so far:
open System
//function to calculate the multiples
let multiple3v5 num =
num
//function to calculate sum of list items
let rec SumList xs =
match xs with
| [] -> 0
| y::ys -> y + SumList ys
let sum = Array.map multiple3v5 [|1 .. 1000|]
What I have may be complete nonsense...so help please?
Your sumList function is a good start. It already iterates (recursively) over the entire list, so you don't need to wrap it in an additional Array.map. You just need to extend your sumList so that it adds the number only when it matches the specified condition.
Here is a solution to a simplified problem - add all numbers that are divisible by 3:
open System
let rec sumList xs =
match xs with
| [] -> 0 // If the list is empty, the sum is zero
| y::ys when y % 3 = 0 ->
// If the list starts with y that is divisible by 3, then we add 'y' to the
// sum that we get by recursively processing the rest of the list
y + sumList ys
| y::ys ->
// This will only execute when y is not divisible by 3, so we just
// recursively process the rest of the list and return
/// that (without adding current value)
sumList ys
// For testing, let's sum all numbers divisble by 3 between 1 and 10.
let sum = sumList [ 1 .. 10 ]
This is the basic way of writing the function using explicit recursion. In practice, the solution by jpalmer is how I'd solve it too, but it is useful to write a few recursive functions yourself if you're learning F#.
The accumulator parameter mentioned by sashang is a more advanced way to write this. You'll need to do that if you want to run the function on large inputs (which is likely the case in Euler problem). When using accumulator parameter, the function can be written using tail recursion, so it avoids stack overflow even when processing long lists.
The idea of a accumulator-based version is that the function takes additional parameter, which represents the sum calculated so far.
let rec sumList xs sumSoFar = ...
When you call it initially, you write sumList [ ... ] 0. The recursive calls will not call y + sumList xs, but will instead add y to the accumulator and then make the recursive call sumList xs (y + sumSoFar). This way, the F# compiler can do tail-call optimization and it will translate code to a loop (similar to the C++ version).
I'm not sure if translating from an imperative language solution is a good approach to developing a functional mindset as instrument (C++ in your case) had already defined an (imperative) approach to solution, so it's better sticking to original problem outlay.
Overall tasks from Project Euler are excellent for mastering many F# facilities. For example, you may use list comprehensions like in the snippet below
// multipleOf3Or5 function definition is left for your exercise
let sumOfMultiples n =
[ for x in 1 .. n do if multipleOf3Or5 x then yield x] |> List.sum
sumOfMultiples 999
or you can a bit generalize the solution suggested by #jpalmer by exploiting laziness:
Seq.initInfinite id
|> Seq.filter multipleOf3Or5
|> Seq.takeWhile ((>) 1000)
|> Seq.sum
or you may even use this opportunity to master active patterns:
let (|DivisibleBy|_) divisior num = if num % divisor = 0 the Some(num) else None
{1..999}
|> Seq.map (fun i ->
match i with | DivisibleBy 3 i -> i | DivisibleBy 5 i -> i | _ -> 0)
|> Seq.sum
All three variations above implement a common pattern of making a sequence of members with sought property and then folding it by calculating sum.
F# has many more functions than just map - this problem suggests using filter and sum, my approach would be something like
let valid n = Left as an exercise
let r =
[1..1000]
|> List.filter valid
|> List.sum
printfn "%i" r
I didn't want to do the whole problem, but filling in the missing function shouldn't be too hard
This is how you turn a loop with a counter into a recursive function. You do this by passing an accumulator parameter to the loop function that holds the current loop count.
For example:
let rec loop acc =
if acc = 10 then
printfn "endloop"
else
printfn "%d" acc
loop (acc + 1)
loop 0
This will stop when acc is 10.