If you have lots of URLs to call get on, are there any queue patterns in F# with some kind of limit, like let's say 5 or 10 calls at a time before proceeding to the next batch.
let urls = [
"http://example.com/1",
"http://example.com/2",
"http://example.com/3",
....
"http://example.com/100"]
While passing a function to call
let getAsync (url:string) =
async {
let httpClient = new System.Net.Http.HttpClient()
let! response = httpClient.GetAsync(url) |> Async.AwaitTask
response.EnsureSuccessStatusCode () |> ignore
let! content = response.Content.ReadAsStringAsync() |> Async.AwaitTask
return content
}
and then receive a list of all results and any errors since some calls may fail due to any reason like 404 or 500 errors.
First, make your function actually return the error information instead of throwing it as an exception:
let getAsync url : Result<_,_> =
async {
try
...
return (Ok content)
with ex ->
return (Error ex)
}
Then use Async.Parallel to execute them in parallel. This function takes an list<Async> and returns an Async<list>:
let allResults = urls |> List.map getAsync |> Async.Parallel
Related
I'm being a little adventurous with my code for the amount of experience I have with F# and I am a little worried about cross threading issues.
Background:
I have a number of orders where I need to validate the address. Some of the orders can be validated against google maps geocoding API which allows 50/ second. the rest are Australian PO Boxes which we don't have many of - but I need to validate them against a different API that only allows 1 call per second.
I have switched over most of my code from async{} functions to task{} functions and I am assuming to get something on several threads at the same time it needs to be in an async{} function or block and be piped to Async.Parallel
Question: Is this the right way to do this or will it fall over? I am wondering if I am fundamentally thinking about this the wrong way.
Notes:
I am passing a database context into the async function and updating the database within that function
I will call this from a C# ( WPF ) Application and report the progress
Am I going to have cross threading issues?
let validateOrder
(
order: artooProvider.dataContext.``dbo.OrdersEntity``,
httpClient: HttpClient,
ctx: artooProvider.dataContext,
isAuPoBox: bool
) =
async {
// Validate Address
let! addressExceptions = ValidateAddress.validateAddress (order, httpClient, ctx, isAuPoBox) |> Async.AwaitTask
// SaveExceptions
do! ctx.SubmitUpdatesAsync()
// return Exception count
return ""
}
let validateGMapOrders(httpClient: HttpClient, ctx: artooProvider.dataContext, orders: artooProvider.dataContext.``dbo.OrdersEntity`` list) =
async {
let ordersChunked = orders |> List.chunkBySize 50
for fiftyOrders in ordersChunked do
let! tasks =
fiftyOrders
|> List.map (fun (order) -> validateOrder (order, httpClient, ctx, false) )
|> Async.Parallel
do! Async.Sleep(2000)
}
let validateOrders (ctx: artooProvider.dataContext, progress: IProgress<DownloadProgressModel>) =
task {
let unvalidatedOrders =
query {
for orders in ctx.Dbo.Orders do
where (orders.IsValidated.IsNone)
select (orders)
}
|> Seq.toList
let auPoBoxOrders =
unvalidatedOrders
|> List.filter (fun order -> isAUPoBox(order) = true )
let gMapOrders =
unvalidatedOrders
|> List.filter (fun order -> isAUPoBox(order) = false )
let googleHttpClient = new HttpClient()
let auspostHttpclient = Auspost.AuspostApi.getApiClient ()
// Google maps validations
do! validateGMapOrders(googleHttpClient,ctx,gMapOrders)
// PO Box Validations
for position in 0 .. auPoBoxOrders.Length - 1 do
let! result = validateOrder (gMapOrders[position], auspostHttpclient, ctx, true)
do! Task.Delay(1000)
return true
}
When I have had to deal with rate-limited API problems I hide that API behind a MailboxProcessor that maintains an internal time to comply with the rate limit but appears as a normal async API from the outside.
Since you have two API's with different rate limits I'd parameterise the time delay and processing action then create one object for each API.
open System
type Request = string
type Response = string
type RateLimitedProcessor() =
// Initialise 1s in past so ready to start immediately.
let mutable lastCall = DateTime.Now - TimeSpan(0, 0, 1)
let mbox = new MailboxProcessor<Request * AsyncReplyChannel<Response>>((fun mbox ->
let rec f () =
async {
let! (req, reply) = mbox.Receive()
let msSinceCall = (DateTime.Now - lastCall).Milliseconds
// wait 1s between requests
if msSinceCall < 1000 then
do! Async.Sleep (1000 - msSinceCall)
lastCall <- DateTime.Now
reply.Reply "Response"
// Call self recursively to process the next incoming message
return! f()
}
f()
))
do mbox.Start()
member __.Process(req:Request): Async<Response> =
async {
return! mbox.PostAndAsyncReply(fun reply -> req, reply)
}
interface IDisposable with
member this.Dispose() = (mbox :> IDisposable).Dispose()
This is some C# code:
var streamStore = new PostgresStreamStore(new PostgresStreamStoreSettings("Host=localhost;Port=5432;User Id=postgres;Password=123456;Database=postgres"));
await streamStore.CreateSchemaIfNotExists();
I'm trying to call it from F# like this:
let db_connection =
Sql.host "localhost"
|> Sql.port 5432
|> Sql.username "postgres"
|> Sql.password "123456"
|> Sql.database "postgres"
|> Sql.str
let store =
new PostgresStreamStore(PostgresStreamStoreSettings(db_connection))
store.CreateSchemaIfNotExists() |> Async.AwaitTask |> ignore
The code compiles, however the schema in contrast to the C# Version does not a create a schema.
How do I await this Task from store.CreateSchemaIfNotExists?
I'm getting this error message:
`This expression is a function value, i.e. is missing arguments. Its type is unit -> Tasks.Task.
In the C# code, you are using await, so this must be inside an async method. The corresponding thing in F# would be to use F# asynchronous workflows. Inside those, you can use let! which is similar to await. This works with computations of type Async<T> rather than Task<T>. The operation Async.AwaitTask turns Task<T> into Async<T> so that you can access it using let!
let doSomething () = async {
let db_connection =
Sql.host "localhost"
// (other configuration omitted)
let store =
new PostgresStreamStore(PostgresStreamStoreSettings(db_connection))
let! res = store.CreateSchemaIfNotExists() |> Async.AwaitTask
return "whatever" }
I assume that CreateSchemaIfNotExists does not return anything useful, so you can also wait for its completion using do!
do! store.CreateSchemaIfNotExists() |> Async.AwaitTask |> Async.Ignore
An asynchronous computation then needs to be started using Async.Start or Async.RunSynchronously, which is akin to starting a task or blocking using task.RunSynchronously.
Not really sure why my code fails to compile, the error I am having:
Incomplete structured construct at or before this point in expression
let resendErrorsAsync (bus: IBus) (errorQueueName: string) =
async {
let! errorQueue = bus.Advanced.QueueDeclareAsync(errorQueueName) |> Async.AwaitTask
let! message = bus.Advanced.GetMessageAsync(errorQueue) |> Async.AwaitTask
while message <> null do
let utf8Body = Encoding.UTF8.GetString(message.Body)
let error = JsonConvert.DeserializeObject<Error>(utf8Body)
let errorBodyBytes = Encoding.UTF8.GetBytes(error.Message)
let! exchange = bus.Advanced.ExchangeDeclareAsync(error.Exchange, "topic") |> Async.AwaitTask
let! message = bus.Advanced.GetMessageAsync(errorQueue) |> Async.AwaitTask
}
It seems to be related to my two async calls in the while loop, not really sure why though.
You can't end a block with a let!, there has to be a body after it; that's the reason for the syntax error.
The message you're defining on the last line is a different variable to the one tested in the while condition. It seems you want to mutate the message; for that, you have to explicitly create a mutable variable:
let resendErrorsAsync (bus: IBus) (errorQueueName: string) =
async {
let! errorQueue = bus.Advanced.QueueDeclareAsync(errorQueueName) |> Async.AwaitTask
let! msg = bus.Advanced.GetMessageAsync(errorQueue) |> Async.AwaitTask
let mutable message = msg
while message <> null do
let utf8Body = Encoding.UTF8.GetString(message.Body)
let error = JsonConvert.DeserializeObject<Error>(utf8Body)
let errorBodyBytes = Encoding.UTF8.GetBytes(error.Message)
let! exchange = bus.Advanced.ExchangeDeclareAsync(error.Exchange, "topic") |> Async.AwaitTask
let! msg = bus.Advanced.GetMessageAsync(errorQueue) |> Async.AwaitTask
message <- msg
}
What is the idiomatic F# way of handling an asynchronous while loop accumulation?
I'm working with the new (still in preview) Azure Cosmos DB SDK. Querying the database returns a CosmosResultSetIterator<T> which has a HasMoreResults property and a FetchNextSetAsync() method. My straight-up translation of the C# code looks like this:
let private fetchItemsFromResultSet (resultSetIterator: CosmosResultSetIterator<'a>) =
let results = ResizeArray<'a>()
async {
while resultSetIterator.HasMoreResults do
let! response = resultSetIterator.FetchNextSetAsync() |> Async.AwaitTask
results.AddRange(response |> Seq.toArray)
return Seq.toList results
}
I would take a look at the AsyncSeq package. You can use it to create asynchronously computed sequences and then iterate them asynchronously or in parallel. This allows for the async-binding to be inside the sequence and the yield to occur asynchronously, so you don't have to build up an accumulator explicitly.
You can use it to do something like:
open FSharp.Control
let private fetchItemsFromResultSet (resultSetIterator: CosmosResultSetIterator<'a>) =
asyncSeq {
while resultSetIterator.HasMoreResults do
let! response = resultSetIterator.FetchNextSetAsync() |> Async.AwaitTask
yield! response |> AsyncSeq.ofSeq
}
IMHO tail-recursion is preferable to while loops as it's one way to avoid mutation.
For example:
let fetchItemsFromResultSet (resultSetIterator: CosmosResultSetIterator<'a>) =
let rec loop results =
async {
if resultSetIterator.HasMoreResults then
let! vs = resultSetIterator.FetchNextSetAsync () |> Async.AwaitTask
let vs = vs |> Seq.toList
return! loop (vs::results)
else
// List.rev needed because batches are in reverse
return results |> List.rev |> List.concat
}
loop []
Very recently, FSharp.Control.TaskSeq was added to support tasks natively with seqs. The answer here by #Just another metaprogrammer can be rewritten as
#r "nuget: FSharp.Control.TaskSeq"
open FSharp.Control
let private fetchItemsFromResultSet (resultSetIterator: CosmosResultSetIterator<'a>) = taskSeq {
while resultSetIterator.HasMoreResults do
let! response = resultSetIterator.FetchNextSetAsync()
yield! response |> TaskSeq.ofSeq
}
Suppose I have a stream which only allows one request/response at a time but is used in several threads.
Requests/commands should be throttled such that a new request can only occur once
the previous request has been sent and a reply has been received.
The user would be able to do this
let! res = getResponse("longResp")
let! res2 = getResponse("shortResp")
and not really know or care about the throttle.
I have tried with a modified version of Tomas Petricek's Throttling Agent that allows async with return values, but this requires the user to call getResponse("..") |> Enqueue |> w.Post which is a recipe for disaster (in case they forget to do so).
Is there a good/idiomatic way of doing this in F#?
Then make it explicit in your type system that the returned type needs to be unwrapped with another function. So instead of returning an Async<'T> which as you pointed out can be called directly with Async.Start, rather return something like:
type Queuable<'T> = Queuable of Async<'T>
Then getResponse changes to return a Queueable:
let getResponse (s:string) =
let r =
async{
do! write to your stream
return! read from your stream
}
Queuable r
Provide a function that unwraps the Queuable:
let enqueue (Queuable q) = async{
return! processor.PostAndAsyncReply(fun replyChannel -> replyChannel,q)
}
The processor is an agent that simply runs the Async workflow. Something like this:
let processor = new MailboxProcessor<_>(fun inbox ->
let rec Loop() = async {
let! (r:AsyncReplyChannel<_>,job) = inbox.Receive()
let! res = job
r.Reply res
return! Loop()}
Loop())