Asynchronous barrier in F# - f#

Question

I wrote a program in F# that asynchronously lists all directories on disk. An async task lists all files in a given directory and creates separate async tasks (daemons: I start them using Async.Start) to list subdirectories. They all communicate the results to the central MailboxProcessor.
My problem is, how do I detect that all the daemon tasks have finished and there will be no more files arriving. Essentially I need a barrier for all tasks that are (direct and indirect) children of my top task. I couldn't find anything like that in the F#'s async model.
What I did instead is to create a separate MailboxProcessor where I register each task's start and termination. When the active count goes to zero, I'm done. But I'm not happy with that solution. Any other suggestions?

Answer 1

Have you tried using Async.Parallel? That is, rather than Async.Start each subdirectory, just combine the subdirectory tasks into a single async via Async.Parallel. Then you end up with a (nested) fork-join task that you can RunSynchronously and await the final result.
EDIT
Here is some approximate code, that shows the gist, if not the full detail:
open System.IO
let agent = MailboxProcessor.Start(fun mbox ->
async {
while true do
let! msg = mbox.Receive()
printfn "%s" msg
})
let rec traverse dir =
async {
agent.Post(dir)
let subDirs = Directory.EnumerateDirectories(dir)
return! [for d in subDirs do yield traverse d]
|> Async.Parallel |> Async.Ignore
}
traverse "d:\\" |> Async.RunSynchronously
// now all will be traversed,
// though Post-ed messages to agent may still be in flight
EDIT 2
Here is the waiting version that uses replies:
open System.IO
let agent = MailboxProcessor.Start(fun mbox ->
async {
while true do
let! dir, (replyChannel:AsyncReplyChannel<unit>) = mbox.Receive()
printfn "%s" dir
replyChannel.Reply()
})
let rec traverse dir =
async {
let r = agent.PostAndAsyncReply(fun replyChannel -> dir, replyChannel)
let subDirs = Directory.EnumerateDirectories(dir)
do! [for d in subDirs do yield traverse d]
|> Async.Parallel |> Async.Ignore
do! r // wait for Post to finish
}
traverse "c:\\Projects\\" |> Async.RunSynchronously
// now all will be traversed to completion

Answer 2

You could just use Interlocked to increment and decrement as you begin/end tasks, and be all done when it goes to zero. I've used this strategy in similar code with MailboxProcessors.

Answer 3

This is probably a learning exercise, but it seems that you would be happy with a lazy list of all of the files. Stealing from Brian's answer above... (and I think something like this is in all of the F# books, which I don't have with me at home)
open System.IO
let rec traverse dir =
seq {
let subDirs = Directory.EnumerateDirectories(dir)
yield dir
for d in subDirs do
yield! traverse d
}
For what it is worth, I have found the Async workflow in F# very useful for "embarrassingly easy" parallel problems, though I haven't tried much general multitasking.

Answer 4

You may be better off just using Task.Factory.StartNew() and Task.WaitAll().

Answer 5

Just for clarification: I thought there might have been a better solution similar to what one can do in Chapel. There you have a "sync" statement, a barrier that waits for all the tasks spawned within a statement to finish. Here's an example from the Chapel manual:
def concurrentUpdate(tree: Tree) {
if requiresUpdate(tree) then
begin update(tree);
if !tree.isLeaf {
concurrentUpdate(tree.left);
concurrentUpdate(tree.right);
}
}
sync concurrentUpdate(tree);
The "begin" statement creates a task that is run in parallel, somewhat similar to F# "async" block with Async.Start.