Limit async calls to at most n per minute - f#

Question

Suppose I have an async that hits some external service:
fetchFoo : Async<string>
In order not to hit the service too hard, I want to rate limit it to n requests per minute.
let fetchFooWithRateLimit : Async<string> = applyRateLimit 6 fetchFoo
If fetchFooWithRateLimit is run more than n times per minute, it will internally wait a little in order delay the underlying call to fetchFoo.
How can I achieve this in F#?

Answer 1

One thing to note about this problem is that in a classic producer-consumer scenario where the producer may outrun the consumer, queuing is inevitable.
A simple approach is to calculate a value of delay for the next item: if the rate limit is hit, delay until the next time slot. The downside is that it might end up using the thread pool as its queue.
With that said, we can use MailboxProcessor as our async queue implementation, as it provides much of what we want out of the box.
let rateLimit fetch period limit =
let now () = DateTimeOffset.Now
let cts = new CancellationTokenSource()
let mailbox =
MailboxProcessor.Start(fun inbox ->
let rec loop nextTime remaining = async {
let diff = int (nextTime - now()).TotalMilliseconds
if remaining = 0 || diff < 0 then
do! Async.Sleep (max diff 0)
return! loop (now() + period) limit
else
let! request = inbox.Receive()
do! fetch request
return! loop nextTime (remaining - 1)
}
loop (now ()) 0
, cts.Token)
{| Post = mailbox.Post; Stop = cts.Cancel |}
The basic idea is to delay de-queuing if we've already exceeded the rate limit.
Test:
let fetch args = async { do printfn "%A %A" DateTime.Now args }
let rl = rateLimit fetch (TimeSpan.FromSeconds 5.0) 5
Observable.interval(TimeSpan.FromSeconds 0.5) |> Observable.subscribe(rl.Post)
Output:
6/4/2020 12:48:19 AM 0L
6/4/2020 12:48:19 AM 1L
6/4/2020 12:48:19 AM 2L
6/4/2020 12:48:19 AM 3L
6/4/2020 12:48:19 AM 4L
6/4/2020 12:48:23 AM 5L
6/4/2020 12:48:23 AM 6L
6/4/2020 12:48:23 AM 7L
6/4/2020 12:48:23 AM 8L
6/4/2020 12:48:23 AM 9L
6/4/2020 12:48:28 AM 10L
6/4/2020 12:48:28 AM 11L
6/4/2020 12:48:28 AM 12L
6/4/2020 12:48:28 AM 13L
6/4/2020 12:48:28 AM 14L
Note: I've used an anonymous record to create a simple API with Post and Stop methods.
If your F# version does not support this yet, just change it to return a tuple.

Answer 2

Basically we need to maintain the moments of the last n executions. Let's name that list lastMoments. When a new execution comes, we calculate the delay based on the current time and the first (oldest) element in lastMoments. Then we update lastMoments - make sure its length is not exceed n.
Below is the code using mutation and lock (you can convert it to using MailboxProcessor if you prefer):
open System
open System.Collections.Generic
let applyRateLimit period n computation =
let lastMoments = LinkedList<DateTime> ()
async {
let delay = lock lastMoments <| fun _ ->
let now = DateTime.Now
let delay =
if lastMoments.Count < n then 0
else period - int (now - lastMoments.Last.Value).TotalMilliseconds
|> max 0
lastMoments.AddLast (LinkedListNode (now.AddMilliseconds (float delay)))
if lastMoments.Count > n then lastMoments.RemoveFirst ()
delay
if delay > 0 then do! Async.Sleep delay
return! computation
}