Suppose I have an async that hits some external service:
fetchFoo : Async<string>
In order not to hit the service too hard, I want to rate limit it to n requests per minute.
let fetchFooWithRateLimit : Async<string> = applyRateLimit 6 fetchFoo
If fetchFooWithRateLimit is run more than n times per minute, it will internally wait a little in order delay the underlying call to fetchFoo.
How can I achieve this in F#?
One thing to note about this problem is that in a classic producer-consumer scenario where the producer may outrun the consumer, queuing is inevitable.
A simple approach is to calculate a value of delay for the next item: if the rate limit is hit, delay until the next time slot. The downside is that it might end up using the thread pool as its queue.
With that said, we can use MailboxProcessor as our async queue implementation, as it provides much of what we want out of the box.
let rateLimit fetch period limit =
let now () = DateTimeOffset.Now
let cts = new CancellationTokenSource()
let mailbox =
MailboxProcessor.Start(fun inbox ->
let rec loop nextTime remaining = async {
let diff = int (nextTime - now()).TotalMilliseconds
if remaining = 0 || diff < 0 then
do! Async.Sleep (max diff 0)
return! loop (now() + period) limit
else
let! request = inbox.Receive()
do! fetch request
return! loop nextTime (remaining - 1)
}
loop (now ()) 0
, cts.Token)
{| Post = mailbox.Post; Stop = cts.Cancel |}
The basic idea is to delay de-queuing if we've already exceeded the rate limit.
Test:
let fetch args = async { do printfn "%A %A" DateTime.Now args }
let rl = rateLimit fetch (TimeSpan.FromSeconds 5.0) 5
Observable.interval(TimeSpan.FromSeconds 0.5) |> Observable.subscribe(rl.Post)
Output:
6/4/2020 12:48:19 AM 0L
6/4/2020 12:48:19 AM 1L
6/4/2020 12:48:19 AM 2L
6/4/2020 12:48:19 AM 3L
6/4/2020 12:48:19 AM 4L
6/4/2020 12:48:23 AM 5L
6/4/2020 12:48:23 AM 6L
6/4/2020 12:48:23 AM 7L
6/4/2020 12:48:23 AM 8L
6/4/2020 12:48:23 AM 9L
6/4/2020 12:48:28 AM 10L
6/4/2020 12:48:28 AM 11L
6/4/2020 12:48:28 AM 12L
6/4/2020 12:48:28 AM 13L
6/4/2020 12:48:28 AM 14L
Note: I've used an anonymous record to create a simple API with Post and Stop methods.
If your F# version does not support this yet, just change it to return a tuple.
Basically we need to maintain the moments of the last n executions. Let's name that list lastMoments. When a new execution comes, we calculate the delay based on the current time and the first (oldest) element in lastMoments. Then we update lastMoments - make sure its length is not exceed n.
Below is the code using mutation and lock (you can convert it to using MailboxProcessor if you prefer):
open System
open System.Collections.Generic
let applyRateLimit period n computation =
let lastMoments = LinkedList<DateTime> ()
async {
let delay = lock lastMoments <| fun _ ->
let now = DateTime.Now
let delay =
if lastMoments.Count < n then 0
else period - int (now - lastMoments.Last.Value).TotalMilliseconds
|> max 0
lastMoments.AddLast (LinkedListNode (now.AddMilliseconds (float delay)))
if lastMoments.Count > n then lastMoments.RemoveFirst ()
delay
if delay > 0 then do! Async.Sleep delay
return! computation
}
I have an observable sequence of things that need to be mapped to
a C# task. These C# tasks should not run concurrently, but one after
another. Basically, I need to achieve the F# equivalent of this C#
question:
Enforcing one async observable at a time
Naively translating this C# code gives something like the following:
let run (idx:int) (delay:int) =
async {
sprintf "start: %i (%i)" idx delay |> System.Diagnostics.Trace.WriteLine
let! t = System.Threading.Tasks.Task.Delay(delay) |> Async.AwaitTask
sprintf "finish: %i" idx |> System.Diagnostics.Trace.WriteLine
t
}
Observable.generate (new Random()) (fun _ -> true) id (fun s -> s.Next(250, 500))
|> Observable.take 20
|> Observable.mapi(fun idx delay -> idx, delay)
|> Observable.bind(fun (idx, delay) -> Observable.ofAsync (run idx delay))
|> Observable.subscribe ignore
|> ignore
Which does not work as expected because I don't wait for a result anywhere. Is there even
a way to do this in F# without blocking a thread, just like C#'s await would?
There's a handy library that exists in F# called AsyncSeq:
https://www.nuget.org/packages/FSharp.Control.AsyncSeq/
It's very similar to IAsyncEnumerable<T> that was added to C# 8.0, this gives you a nice solution to what you are after.
Solution:
open System
open FSharp.Control
open FSharp.Control.Reactive
[<EntryPoint>]
let main _ =
let run (idx:int) (delay:int) =
async {
sprintf "start: %i (%i)" idx delay |> Console.WriteLine
do! Async.Sleep delay
sprintf "finish: %i" idx |> Console.WriteLine
}
Observable.generate (new Random()) (fun _ -> true) id (fun s -> s.Next(250, 500))
|> Observable.take 20
|> Observable.mapi(fun idx delay -> idx, delay)
|> AsyncSeq.ofObservableBuffered
|> AsyncSeq.iterAsync (fun (idx,delay) -> run idx delay)
|> Async.RunSynchronously
0
AsyncSeq.ofObservableBuffered does the work of subscribing to your Observable and acts as AsyncSeq source which you can pipeline on top of. Finally we call Async.RunSynchronously to kick it off and wait on the entrypoint thread.
Note: I also update run as it was returning Async<Async<unit>> and I assume that wasn't intended.
The following code takes about 20 seconds to run. However, it took less than a second after uncommenting the do!. Why there is such a huge difference?
Update:
it takes 9 seconds when using ag.Add. I've updated the code.
open FSharpx.Control
let test () =
let ag = new BlockingQueueAgent<int option>(500)
let enqueue() = async {
for i = 1 to 500 do
//do! ag.AsyncAdd (Some i) // less than a second with do!
ag.AsyncAdd (Some i) // it takes about 20 seconds without do!
//ag.Add (Some i) // This one takes about 9 seconds
//printfn "=> %d" i
}
async {
do! [ for i = 1 to 100 do yield enqueue() ]
|> Async.Parallel |> Async.Ignore
for i = 1 to 5 do ag.Add None
} |> Async.Start
let rec dequeue() =
async {
let! m = ag.AsyncGet()
match m with
| Some v ->
//printfn "<= %d" v
return! dequeue()
| None ->
printfn "Done"
}
[ for i = 1 to 5 do yield dequeue() ]
|> Async.Parallel |> Async.Ignore |> Async.RunSynchronously
0
Without the do!, you're not awaiting the results of AsyncAdd. That means that you're kicking off five hundred AsyncAdd operations as fast as possible for each call to enqueue(). And although each AsyncAdd call will block if the queue is full, if you don't await the result of AsyncAdd then your enqueue() code won't be blocked, and it will continue to launch new AsyncAdd operations.
And since you're launching 100 enqueue() operations in parallel, that's potentially up to fifty thousand AsyncAdd operations that will be trying to run at the same time, which means 49,500 blocked threads being handled by the thread pool. That's a LOT of demand to put on your system. In practice, you won't launch 100 enqueue() operations in parallel at the same time, but you'll launch as many enqueue() operations as you have logical CPUs. For the rest of this answer, I'm going to assume that you have a quad-core processor with hyperthreading (as your F# Async.Parallel |> Async.RunSynchronously only uses one of the eight CPU core? question seems to suggest), so that's 8 logical CPUs so you'll launch eight copies of enqueue() before anything blocks, meaning you'll have 4,000 AsyncAdd threads running, 3,500 of which will be blocked.
When you use do!, on the other hand, then if AsyncAdd is blocked, your enqueue() operation will also block until there's a slot open in the queue. So once there are 500 items in the queue, instead of (8*500 - 500 = 3500) blocked AsyncAdd threads sitting in the thread pool, there will be 8 blocked AsyncAdd threads (one for each of the eight enqueue() operations running on each of your eight logical CPUs). Eight blocked threads instead of 3,500 means that the thread pool isn't making 3,500 allocations, using much less RAM and much less CPU time to process all those threads.
As I said in my answer to your previous question, it really seems like you need a deeper understanding of asynchronous operations. Besides the articles I linked to in that answer (this article and this series), I'm also going to recommend reading https://medium.com/jettech/f-async-guide-eb3c8a2d180a which is a pretty long and detailed guide to F# async operations and some of the "gotchas" you can encounter. I'd strongly suggest going and reading those articles, then coming back and looking at your questions again. With the deeper understanding you've gained from reading those articles, you just might be able to answer your own questions!
Continued from this question. Here is the experiment based on your code:
// Learn more about F# at http://fsharp.org
module Test.T1
open System
open System.Collections.Generic
open System.Diagnostics
type Msg<'T> =
| AsyncAdd of 'T * AsyncReplyChannel<unit>
| Add of 'T
| AsyncGet of AsyncReplyChannel<'T>
let sw = Stopwatch()
let mutable scanned = 0
let mutable scanTimeStart = 0L
let createQueue maxLength = MailboxProcessor.Start(fun inbox ->
let queue = new Queue<'T>()
let rec emptyQueue() =
inbox.Scan(fun msg ->
match msg with
| AsyncAdd(value, reply) -> Some(enqueueAndContinueWithReply(value, reply))
| Add(value) -> Some(enqueueAndContinue(value))
| _ -> None )
and fullQueue() =
scanTimeStart <- sw.ElapsedMilliseconds
inbox.Scan(fun msg ->
scanned <- scanned + 1
match msg with
| AsyncGet(reply) ->
Some(dequeueAndContinue(reply))
| _ -> None )
and runningQueue() = async {
let! msg = inbox.Receive()
scanTimeStart <- sw.ElapsedMilliseconds
match msg with
| AsyncAdd(value, reply) -> return! enqueueAndContinueWithReply(value, reply)
| Add(value) -> return! enqueueAndContinue(value)
| AsyncGet(reply) -> return! dequeueAndContinue(reply) }
and enqueueAndContinueWithReply (value, reply) = async {
reply.Reply()
queue.Enqueue(value)
return! chooseState() }
and enqueueAndContinue (value) = async {
queue.Enqueue(value)
return! chooseState() }
and dequeueAndContinue (reply) = async {
let timestamp = sw.ElapsedMilliseconds
printfn "[AsyncGet] messages cnt/scanned: %d/%d, timestamp/scanTime: %d/%d" inbox.CurrentQueueLength scanned timestamp (timestamp - scanTimeStart)
scanned <- 0
reply.Reply(queue.Dequeue())
return! chooseState() }
and chooseState() =
if queue.Count = 0 then emptyQueue()
elif queue.Count < maxLength then runningQueue()
else fullQueue()
emptyQueue())
let mb = createQueue<int option> 500
let addWithReply v = mb.PostAndAsyncReply(fun ch -> AsyncAdd(v, ch))
let addAndForget v = mb.Post(Add v)
let get() = mb.PostAndAsyncReply(AsyncGet)
[<EntryPoint>]
let main args =
sw.Start()
let enqueue() = async {
for i = 1 to 500 do
//do! ag.AsyncAdd (Some i) // less than a second with do!
addWithReply (Some i) // it takes about 20 seconds without do!
//addAndForget(Some i)
//ag.Add (Some i) // This one takes about 9 seconds
//printfn "=> %d" i
}
async {
do! [ for i = 1 to 100 do yield enqueue() ]
|> Async.Parallel |> Async.Ignore
for i = 1 to 5 do addAndForget None
} |> Async.Start
let rec dequeue() =
async {
let! m = get()
match m with
| Some v ->
//printfn "<= %d" v
return! dequeue()
| None ->
printfn "Done"
}
[ for i = 1 to 5 do yield dequeue() ]
|> Async.Parallel |> Async.Ignore |> Async.RunSynchronously
sw.Stop()
printfn "Totally ellapsed: %dms" sw.ElapsedMilliseconds
0
addWithReply is AsyncAdd. When we run without do! the output is (part of it):
...
[AsyncGet] messages cnt/scanned: 48453/48450, timestamp/scanTime: 3755/6
[AsyncGet] messages cnt/scanned: 48452/48449, timestamp/scanTime: 3758/3
[AsyncGet] messages cnt/scanned: 48451/48448, timestamp/scanTime: 3761/3
[AsyncGet] messages cnt/scanned: 48450/48447, timestamp/scanTime: 3764/3
...
So as you can see, without do! you basically add all 50000 enqueue requests to message queue of mailbox. Dequeue threads are slower here and put their requests only at the end of the messages. Last line of outputstates that we have 48450 message in mailbox, item queue is full (500 items) and in order to free one space we need to scan 48447 messages - because all of them are AsyncAdd, not AsyncGet. scanTime is 2-3ms (on my machine) - approximate time from MailboxProcessor.Scan.
When we add do!, the message queue has different shape (see the output):
[AsyncGet] messages cnt/scanned: 98/96, timestamp/scanTime: 1561/0
[AsyncGet] messages cnt/scanned: 96/96, timestamp/scanTime: 1561/0
[AsyncGet] messages cnt/scanned: 104/96, timestamp/scanTime: 1561/0
[AsyncGet] messages cnt/scanned: 102/96, timestamp/scanTime: 1561/0
The number of messages in message queue ~ # of enqueue threads, because each of them wait now.
What I cannot understand from the experiment yet is when you change AsyncAdd to Add, you still spam the MailboxProcessor:
[AsyncGet] messages cnt/scanned: 47551/47548, timestamp/scanTime: 3069/1
[AsyncGet] messages cnt/scanned: 47550/47547, timestamp/scanTime: 3070/1
[AsyncGet] messages cnt/scanned: 47549/47546, timestamp/scanTime: 3073/3
[AsyncGet] messages cnt/scanned: 47548/47545, timestamp/scanTime: 3077/2
but avg time spent on scan is ~1ms - faster then with AsyncReplyChannel. My thought - this is connected to how AsyncReplyChannel is implemented. It has dependency on ManualResetEvent, so internally there could be another queue of such events per process and each AsyncGet should scan this queue when AsyncReplyChannel is created.
I created a dotnet core application and run the following code of release build. However, the total CPU usage of the PC is around only 20% and process dotnet run takes only 12% (I have eight logical CPUs and I don't see any one of it use 100% either). Isn't the CPU the bottleneck of the code?
open FSharpx.Control
[<EntryPoint>]
let main argv =
let ag = new BlockingQueueAgent<int option>(500)
let enqueue() = async { for i = 0 to 1000 do ag.Add (Some i) }
async {
do! [ for i = 0 to 1000 do yield enqueue() ]
|> Async.Parallel |> Async.Ignore
ag.Add None
} |> Async.Start
let mutable x = 0
let rec dequeue() =
async {
let! m = ag.AsyncGet()
match m with
| Some v ->
//x <- x ^^^ v
for z = 0 to 10000 do x <- x + z
return! dequeue()
| None ->
printfn "Done %d" x
}
[ for i = 0 to 100 do yield dequeue() ]
|> Async.Parallel |> Async.Ignore |> Async.RunSynchronously
0
Here is the source code of BlockingQueueAgent:
https://github.com/fsprojects/FSharpx.Async/blob/master/src/FSharpx.Async/BlockingQueueAgent.fs
Update:
Added more complex code (repaced x <- x ^^^ v). Now it uses a CPU core a lot. Still 13% though. Why it doesn't use multiple core?
You're synchronously enqueueing all of your Add operations before you start dequeuing any messages. This means that when the agent is choosing what to do next it will always Add a new item to the queue if it isn't full. When it is full, it will search for the first AsyncGet operation and process that, but then will immediately Add (synchronously) the next item to the queue before allowing another message to be dequeued. This effectively only allows you to dequeue one message at a time because the agent is always switching back and forth between Add and AsyncGet operations.
If you do an AsyncAdd instead of an Add then both enqueuing and dequeueing can happen asynchronously and you get the desired behaviour, i.e.
let enqueue() = async { for i = 0 to 1000 do do! ag.AsyncAdd (Some i) }
The following code
let doWork n = async {
[1..5]
|> Seq.iter(fun i ->
System.Threading.Thread.Sleep 1000 // Simulate working
printfn "%s runs %i seconds" n i
)
}
let f1 = async {
do! doWork "f1"
return 1
}
async {
let a = f1
do! doWork "main"
let! i = a
} |> Async.RunSynchronously
And it prints the following result. It shows that the two doWork calls run sequentially.
main runs 1 seconds
main runs 2 seconds
main runs 3 seconds
main runs 4 seconds
main runs 5 seconds
f1 runs 1 seconds
f1 runs 2 seconds
f1 runs 3 seconds
f1 runs 4 seconds
f1 runs 5 seconds
However, I want the doWork called by f1 and the main code be running in parallel?
Try it like this:
let computation () = async {
printfn "Start work"
do! Async.Sleep(100)
printfn "Finished work"
}
let s = seq {
for i in 1 .. 10 do
let c = computation()
yield c
}
s |> Async.Parallel |> Async.Ignore |> Async.Start
Async.Parallel converts the Async<unit> list to Async<unit []> which can be run in parallel then.