I have the following interface method:
Task<string[]> GetBlobsFromContainer(string containerName);
and its implementation in C#:
var container = await _containerClient.GetContainer(containerName);
var tasks = container.ListBlobs()
.Cast<CloudBlockBlob>()
.Select(b => b.DownloadTextAsync());
return await Task.WhenAll(tasks);
When I try to rewrite it in F#:
member this.GetBlobsFromContainer(containerName : string) : Task<string[]> =
let task = async {
let! container = containerClient.GetContainer(containerName) |> Async.AwaitTask
return container.ListBlobs()
|> Seq.cast<CloudBlockBlob>
|> Seq.map (fun b -> b.DownloadTextAsync())
|> ??
}
task |> ??
I'm stuck with the last lines.
How to return to Task<string[]> from F# properly?
I had to guess what the type of containerClient is and the closest I found is CloudBlobClient (which does not have getContainer: string -> Task<CloubBlobContainer> but it shouldn't be too hard to adapt). Then, your function might look like as follows:
open System
open System.Threading.Tasks
open Microsoft.WindowsAzure.Storage.Blob
open Microsoft.WindowsAzure.Storage
let containerClient : CloudBlobClient = null
let GetBlobsFromContainer(containerName : string) : Task<string[]> =
async {
let container = containerClient.GetContainerReference(containerName)
return! container.ListBlobs()
|> Seq.cast<CloudBlockBlob>
|> Seq.map (fun b -> b.DownloadTextAsync() |> Async.AwaitTask)
|> Async.Parallel
} |> Async.StartAsTask
I changed the return type to be Task<string[]> instead of Task<string seq> as I suppose you want to keep the interface. Otherwise, I'd suggest to get rid of the Task and use Async in F#-only code.
Will this work?
member this.GetBlobsFromContainer(containerName : string) : Task<string seq> =
let aMap f x = async {
let! a = x
return f a }
let task = async {
let! container = containerClient.GetContainer(containerName) |> Async.AwaitTask
return! container.ListBlobs()
|> Seq.cast<CloudBlockBlob>
|> Seq.map (fun b -> b.DownloadTextAsync() |> Async.AwaitTask)
|> Async.Parallel
|> aMap Array.toSeq
}
task |> Async.StartAsTask
I had to make some assumptions about containerClient etc. so I haven't been able to test this, but at least it compiles.
Related
I have function call
let result = session.Query<Domain.CustomerReadModel>().ToListAsync()
which returns Task<Collections.Generic.IReadOnlyList<Domain.CustomerReadModel>>
How do I "await" this task correctly in F#?
I tried
async {
session.Query<Domain.CustomerReadModel>().ToListAsync() |> ignore
}
as well as
let result = session.Query<Domain.CustomerReadModel>().ToListAsync()
|> Async.AwaitTask
|> Async.RunSynchronously
This seems to compile but I can't use the result and return it:
session.Query<Domain.CustomerReadModel>().ToListAsync()
|> Async.AwaitTask
|> Async.RunSynchronously
Async.AwaitTask takes the task, wraps it in an async computation, and returns you that computation. Once you have it, you can use it with let! or do! or return! just like any other async computation.
async {
let queryAsFsharpAsync = session.Query<Domain.CustomerReadModel>().ToListAsync() |> Async.AwaitTask
let! result = queryAsFsharpAsync
...
}
Or without giving the computation its own name:
async {
let! result = session.Query<Domain.CustomerReadModel>().ToListAsync() |> Async.AwaitTask
...
}
I defined await function like this:
let await (task : Task<'a>) =
task |> Async.AwaitTask |> Async.RunSynchronously
and two tasks like this:
let stringTask = new Task<string>(fun () -> "something")
let unitTask = new Task(fun () -> printf "something")
Calling them like this:
let stringResult = await stringTask
await unitTask
But the second is not generic and I can't call await for it, thus I edited the function argument like this:
let await (task : Task) =
task |> Async.AwaitTask |> Async.RunSynchronously
The problem is now the await stringTask is returning unit instead of string.
How should I write the await function to accept both Task<'a> and Task as the parameter and await it?
Your original definition for await is fine.
The problem is with unitTask which is not generic. Make it generic the same way you did with stringTask:
let unitTask = new Task<unit>(fun () -> printf "something")
If you want to handle both generic Task<'a> and non-generic Task, you have 2 options.
Create 2 await functions (the functional way):
let await (task : Task<'a>) = task |> Async.AwaitTask |> Async.RunSynchronously
let awaitUnit (task : Task ) = task |> Async.AwaitTask |> Async.RunSynchronously
or create 2 Await members for instance on type Task (the OO way):
type Task with
static member Await (task : Task ) = task |> Async.AwaitTask |> Async.RunSynchronously
static member Await (task : Task<'a>) = task |> Async.AwaitTask |> Async.RunSynchronously
By convention members use Pascal case so I used Await instead of await.
Remember to call task.Start() before you call your await.
Async.AwaitTask already has overloads for both Task<'a> and Task, so unless you really need a helper function, you can use it directly:
let stringResult = Async.AwaitTask stringTask |> Async.RunSynchronously
Async.AwaitTask unitTask |> Async.RunSynchronously
And if this is your actual code and you really need to block the thread to wait for those results, you can also just use the Task API directly to save the Task -> Async translation overhead:
let stringResult = stringTask.Result
unitTask.Wait()
You could create a Task -> Task<unit> helper function to convert Task objects you get from the API into Task<unit>. I suspect that this isn't the best way to do it, since it has been a while since I've paid much attention to the differences between F# async and the framework's Task-based asynchrony, but this works in a trivial extension of your example:
open System.Threading
open System.Threading.Tasks
let toUnitTask (t : Task) =
new Task<_>(fun () -> t.Start(); t |> Async.AwaitTask |> Async.RunSynchronously)
let await (task : Task<_>) =
task |> Async.AwaitTask |> Async.RunSynchronously
let stringTask = new Task<string>(fun () -> Thread.Sleep 3000; "something")
let unitTask = new Task(fun () -> Thread.Sleep 2000; printfn "something") |> toUnitTask
stringTask.Start()
unitTask.Start()
let stringResult = await stringTask
await unitTask
I have the following code
open FSharp.Data
let downloadFile link =
......
use os = File.Create(...)
Http.RequestStream(....).ReponseStream.CopyTo(os)
let rec consume() = async {
......
|> Seq.iter (fun x ->
xxx |> Seq.iter(fun link ->
downloadFile link
))
}
I found that the sync downloading makes the code not run concurrently. So I'm trying to do somthing like the following. How to change it to use the FSharp.Data http AsyncRequestStream? Maybe the CopyTo can be async too?
open FSharp.Data
let downloadFile link = async {
......
use os = File.Create(...)
Http.AsyncRequestStream(....).ReponseStream.CopyTo(os) // Error
}
let rec consume() = async {
......
|> Seq.iter (fun x ->
xxx |> Seq.iter(fun link ->
downloadFile link |> Async.Start // do! downloadFile link????
))
}
consume() |> Async.RunSynchronously
Here's a skeleton solution, worthy of all the blank spots in your example:
let downloadFile link =
async {
......
use os = File.Create(...)
let! resp = Http.AsyncRequestStream(....)
return resp.ReponseStream.CopyTo(os)
}
let consume link =
async {
let comps : Async<unit> [] =
xxx
|> Seq.map (fun link -> downloadFile link)
|> Array.ofSeq
return! Async.Parallel comps
}
I think you should read up on asynchronicity and concurrency in general, as well as how to use it in F# in particular. From the OP it seems the whole thing is a bit hazy to you.
Edit: to answer the question in the comment:
With return! (or let!, or do!) you execute the nested workflow asynchronously, then pick up executing the current workflow from that point. That is, everything "below" the do! is put into a continuation that gets called once the thing "after" the do! finishes.
Whereas Async.Start fires up the workflow on (another) background thread and returns immediately without waiting for it to finish.
I'm doing many async web requests and using Async.Parallel. Something like:
xs
|> Seq.map (fun u -> downloadAsync u.Url)
|> Async.Parallel
|> Async.Catch
Some request may throw exceptions, I want to log them and continue with the rest of urls. I found the Async.Catch function, but this stop the computation when the first exception is thrown. I know I can use a try...with expression within the async expression in order to compute the entire list, but, i think, this implies passing a log function to my downloadAsync function changing his type. Is there any other way to catch the exceptions, log them and continue with the rest of urls?
The 'trick' is to move the catch into the map such that catching is parallelized as well:
open System
open System.IO
open System.Net
type T = { Url : string }
let xs = [
{ Url = "http://microsoft.com" }
{ Url = "thisDoesNotExists" } // throws when constructing Uri, before downloading
{ Url = "https://thisDotNotExist.Either" }
{ Url = "http://google.com" }
]
let isAllowedInFileName c =
not <| Seq.contains c (Path.GetInvalidFileNameChars())
let downloadAsync url =
async {
use client = new WebClient()
let fn =
[|
__SOURCE_DIRECTORY__
url |> Seq.filter isAllowedInFileName |> String.Concat
|]
|> Path.Combine
printfn "Downloading %s to %s" url fn
return! client.AsyncDownloadFile(Uri(url), fn)
}
xs
|> Seq.map (fun u -> downloadAsync u.Url |> Async.Catch)
|> Async.Parallel
|> Async.RunSynchronously
|> Seq.iter (function
| Choice1Of2 () -> printfn "Succeeded"
| Choice2Of2 exn -> printfn "Failed with %s" exn.Message)
(*
Downloading http://microsoft.com to httpmicrosoft.com
Downloading thisDoesNotExists to thisDoesNotExists
Downloading http://google.com to httpgoogle.com
Downloading https://thisDotNotExist.Either to httpsthisDotNotExist.Either
Succeeded
Failed with Invalid URI: The format of the URI could not be determined.
Failed with The remote name could not be resolved: 'thisdotnotexist.either'
Succeeded
*)
Here I wrapped the download into another async to capture the Uri construction exception.
Suppose I have an async data source:
let getData() = async { return [ 3.14; 2.72 ] }
I could call it using let! and a temporary label:
let showData1() = async {
let! data = getData()
data
|> Seq.iter (printfn "%A")
}
Or, I could call it (inefficiently!) using Async.RunSynchronously and piping, and without a temporary label:
let showData2() = async {
getData()
|> Async.RunSynchronously
|> Seq.iter (printfn "%A")
}
I like the syntax of showData2 but know that calling Async.RunSynchronously ties up the calling thread.
Is there a syntax, operator, or function defined somewhere that allows me to pipe an Async<'T> into a function that accepts 'T?
It sounds like you want map for Async:
let mapAsync f a = async {
let! v = a
return (f v)
}
then you can do:
let m: Async<unit> = getData() |> mapAsync (Seq.iter (printfn "%A"))