I'm trying to make an asynchronous web request to a URL that will return if the request takes too long. I'm using the F# asynchronous workflow and the System.Net.Http library to do this.
However, I am unable to catch the Task/OperationCancelledExceptions that are raised by the System.Net.Http library in the async workflow. Instead, the exception is raised at the Async.RunSynchronously method, as you can see in this stack trace:
> System.OperationCanceledException: The operation was canceled. at
> Microsoft.FSharp.Control.AsyncBuilderImpl.commit[a](Result`1 res)
> at
> Microsoft.FSharp.Control.CancellationTokenOps.RunSynchronously[a](CancellationToken
> token, FSharpAsync`1 computation, FSharpOption`1 timeout) at
> Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync`1
> computation, FSharpOption`1 timeout, FSharpOption`1 cancellationToken)
> at <StartupCode$FSI_0004>.$FSI_0004.main#()
The code:
#r "System.Net.Http"
open System.Net.Http
open System
let readGoogle () = async {
try
let request = new HttpRequestMessage(HttpMethod.Get, "https://google.co.uk")
let client = new HttpClient()
client.Timeout <- TimeSpan.FromSeconds(0.01) //intentionally low to always fail in this example
let! response = client.SendAsync(request, HttpCompletionOption.ResponseContentRead) |> Async.AwaitTask
return Some response
with
| ex ->
//is never called
printfn "TIMED OUT"
return None
}
//exception is raised here
readGoogle ()
|> Async.RunSynchronously
|> ignore
Cancellation was always different from the error. In your case you can override default behavior of AwaitTask that invokes "cancel continuation" if task is cancelled and handle it differently:
let readGoogle () = async {
try
let request = new HttpRequestMessage(HttpMethod.Get, "https://google.co.uk")
let client = new HttpClient()
client.Timeout <- TimeSpan.FromSeconds(0.01) //intentionally low to always fail in this example
return! (
let t = client.SendAsync(request, HttpCompletionOption.ResponseContentRead)
Async.FromContinuations(fun (s, e, _) ->
t.ContinueWith(fun (t: Task<_>) ->
// if task is cancelled treat it as timeout and process on success path
if t.IsCanceled then s(None)
elif t.IsFaulted then e(t.Exception)
else s(Some t.Result)
)
|> ignore
)
)
with
| ex ->
//is never called
printfn "TIMED OUT"
return None
}
Related
module Main
open System
open System.Threading
open System.Threading.Tasks
open NetMQ
open NetMQ.Sockets
let uri = "ipc://hello-world"
let f (token : CancellationToken) =
use server = new ResponseSocket()
use poller = new NetMQPoller()
poller.Add(server)
printfn "Server is binding to: %s" uri
server.Bind(uri)
printfn <| "Done binding."
use __ = server.ReceiveReady.Subscribe(fun x ->
if token.CanBeCanceled then poller.Stop()
)
use __ = server.SendReady.Subscribe(fun x ->
if token.CanBeCanceled then poller.Stop()
)
poller.Run()
printfn "Server closing."
server.Unbind(uri)
let src = new CancellationTokenSource()
let token = src.Token
let task = Task.Run((fun () -> f token), token)
src.CancelAfter(100)
task.Wait() // Does not trigger.
My failed attempt looks something like this. The problem is that the poller will only check the cancellation token if it gets or sends a message. I guess one way to do it would be to send a special cancel message from the client rather than these tokens, but that would not work if the server gets into a send state.
What would be a reliable way of closing the server in NetMQ?
I have an F# funciton that uses a static instance of HttpClient:
let executeRequest request =
async {
let! response = StaticHttpClient.Instance.SendAsync(request) |> Async.AwaitTask
let! stream = response.Content.ReadAsStreamAsync() |> Async.AwaitTask
return (stream, response.StatusCode)
}
|> Async.RunSynchronously
When the request body is large, the function often throws an AggregateException with inner exception "NotSupportedException: The stream does not support concurrent IO read or write operations."
I wonder why this happens. Looks like there is an attempt to use response stream before the request is stream is fully processed. But why?
I'm trying to import a node module in my fable code. Being new to fable I did expect so problems and understanding the import flow seems to be one of those. I have the below code which compiles fine but fails run time with Cannot read property 'request' of undefined on the line of the printfn statement
module Session =
let inline f (f: 'a->'b->'c->'d) = Func<_,_,_,_> f
[<Import("default","request")>]
type Http =
abstract request : string -> System.Func<obj,obj,obj,unit> -> unit
let http : Http = failwith "js only"
let start () =
http.request "http://dr.dk" (ff (fun error response body ->
printfn "%A" body
))
do
start()
I was able to get your example working in the Fable REPL:
open System
open Fable
open Fable.Core
open Fable.Core.JS
open Fable.Core.JsInterop
type RequestCallback = Func<obj, obj, obj, unit>
type Request = Func<string, RequestCallback, unit>
[<ImportDefault("request")>]
let request : Request = jsNative
let start () =
request.Invoke("http://dr.dk", (fun error response body ->
console.log("error", error)
console.log("response", response)
console.log("body", body)
))
start ()
And here is the JavaScript that it generated:
import request from "request";
import { some } from "fable-library/Option.js";
export function start() {
request("http://dr.dk", (error, response, body) => {
console.log(some("error"), error);
console.log(some("response"), response);
console.log(some("body"), body);
});
}
start();
Note that there are bindings for many modules already. For this particular task, I would suggest using Fable.Fetch. If you want a library that works in the browser and .NET, try Fable.SimpleHttp.
I read the following code from the book Expert f#,
Why the function collectLinks embeds let! html = async { .... } in the outer async block? How about just flat it by removing the inner async?
Same question for the function waitForUrl in urlCollector which has a do! Async.StartChild (async {....}) |> Async.Ignore in an outer async block. How about flat it?
How is the implementation comparing with the one implemented with block queue? https://msdn.microsoft.com/en-us/library/vstudio/hh297096(v=vs.100).aspx Creating a block queue with 5, and en-queue the link to producer.
Code:
open System.Collections.Generic
open System.Net
open System.IO
open System.Threading
open System.Text.RegularExpressions
let limit = 50
let linkPat = "href=\s*\"[^\"h]*(http://[^&\"]*)\""
let getLinks (txt:string) =
[ for m in Regex.Matches(txt,linkPat) -> m.Groups.Item(1).Value ]
// A type that helps limit the number of active web requests
type RequestGate(n:int) =
let semaphore = new Semaphore(initialCount=n, maximumCount=n)
member x.AsyncAcquire(?timeout) =
async { let! ok = Async.AwaitWaitHandle(semaphore,
?millisecondsTimeout=timeout)
if ok then
return
{ new System.IDisposable with
member x.Dispose() =
semaphore.Release() |> ignore }
else
return! failwith "couldn't acquire a semaphore" }
// Gate the number of active web requests
let webRequestGate = RequestGate(5)
// Fetch the URL, and post the results to the urlCollector.
let collectLinks (url:string) =
async { // An Async web request with a global gate
let! html =
async { // Acquire an entry in the webRequestGate. Release
// it when 'holder' goes out of scope
use! holder = webRequestGate.AsyncAcquire()
let req = WebRequest.Create(url,Timeout=5)
// Wait for the WebResponse
use! response = req.AsyncGetResponse()
// Get the response stream
use reader = new StreamReader(response.GetResponseStream())
// Read the response stream (note: a synchronous read)
return reader.ReadToEnd() }
// Compute the links, synchronously
let links = getLinks html
// Report, synchronously
do printfn "finished reading %s, got %d links" url (List.length links)
// We're done
return links }
/// 'urlCollector' is a single agent that receives URLs as messages. It creates new
/// asynchronous tasks that post messages back to this object.
let urlCollector =
MailboxProcessor.Start(fun self ->
// This is the main state of the urlCollector
let rec waitForUrl (visited : Set<string>) =
async { // Check the limit
if visited.Count < limit then
// Wait for a URL...
let! url = self.Receive()
if not (visited.Contains(url)) then
// Start off a new task for the new url. Each collects
// links and posts them back to the urlCollector.
do! Async.StartChild
(async { let! links = collectLinks url
for link in links do
self.Post link }) |> Async.Ignore
// Recurse into the waiting state
return! waitForUrl(visited.Add(url)) }
// This is the initial state.
waitForUrl(Set.empty))
I can think of one reason why async code would call another async block, which is that it lets you dispose of resources earlier - when the nested block completes. To demonstrate this, here is a little helper that prints a message when Dispose is called:
let printOnDispose text =
{ new System.IDisposable with
member x.Dispose() = printfn "%s" text }
The following uses nested async to do something in a nested block and then cleanup the local resources used in the nested block. Then it sleeps some more and cleans up resources used in the outer block:
async {
use bye = printOnDispose "bye from outer block"
let! r = async {
use bye = printOnDispose "bye from nested block"
do! Async.Sleep(1000)
return 1 }
do! Async.Sleep(1000) }
|> Async.Start
Here, the "nested block" resources are disposed of after 1 second and the outer block resources are disposed of after 2 seconds.
There are other cases where nesting async is useful (like returning from an asynchronous block containing try .. with), but I don't think that applies here.
When calling Async.RunSynchronously with a timeout and a CancellationToken, the timeout value seems to be ignored. I can work around this by calling CancelAfter on the CancellationToken, but ideally I'd like to be able to distinguish between exceptions that occur in the workflow, TimeOutExceptions and OperationCanceledExceptions.
I believe the sample code below demonstrates this.
open System
open System.Threading
let work =
async {
let endTime = DateTime.UtcNow.AddMilliseconds(100.0)
while DateTime.UtcNow < endTime do
do! Async.Sleep(10)
Console.WriteLine "working..."
raise ( Exception "worked for more than 100 millis" )
}
[<EntryPoint>]
let main argv =
try
Async.RunSynchronously(work, 50)
with
| e -> Console.WriteLine (e.GetType().Name + ": " + e.Message)
let cts = new CancellationTokenSource()
try
Async.RunSynchronously(work, 50, cts.Token)
with
| e -> Console.WriteLine (e.GetType().Name + ": " + e.Message)
cts.CancelAfter(80)
try
Async.RunSynchronously(work, 50, cts.Token)
with
| e -> Console.WriteLine (e.GetType().Name + ": " + e.Message)
Console.ReadKey(true) |> ignore
0
The outputs the following, showing that the timeout is only effective in the first case (where no CancelationToken is specified)
working...
working...
TimeoutException: The operation has timed out.
working...
working...
working...
working...
working...
working...
working...
Exception: worked for more than 100 millis
working...
working...
working...
working...
working...
working...
OperationCanceledException: The operation was canceled.
Is this the intended behaviour? Is there any way get the behaviour I'm after?
Thanks!
I'm not sure if this is intended behaviour - at least, I do not see any reason why it would be. However, this behaviour is implemented directly in the handling of parameters of RunSynchronously. If you look at the library source code, you can see:
static member RunSynchronously (p:Async<'T>,?timeout,?cancellationToken) =
let timeout,token =
match cancellationToken with
| None -> timeout,(!defaultCancellationTokenSource).Token
| Some token when not token.CanBeCanceled -> timeout, token
| Some token -> None, token
In your case (with both timeout and a cancellation token that can be cancelled), the code goes through the last branch and ignores the timeout. I think this is either a bug or it is something that should be mentioned in the documentation.
As a workaround, you can create a separate CancellationTokenSource to specify the timeout and link it to the main cancellation source so that the caller provides (using CreateLinkedTokenSource). When you get OperationCancelledException, you can then detect whether the source was an actual cancellation or a timeout:
type Microsoft.FSharp.Control.Async with
static member RunSynchronouslyEx(a:Async<'T>, timeout:int, cancellationToken) =
// Create cancellation token that is cancelled after 'timeout'
let timeoutCts = new CancellationTokenSource()
timeoutCts.CancelAfter(timeout)
// Create a combined token that is cancelled either when
// 'cancellationToken' is cancelled, or after a timeout
let combinedCts =
CancellationTokenSource.CreateLinkedTokenSource
(cancellationToken, timeoutCts.Token)
// Run synchronously with the combined token
try Async.RunSynchronously(a, cancellationToken = combinedCts.Token)
with :? OperationCanceledException as e ->
// If the timeout occurred, then we throw timeout exception instead
if timeoutCts.IsCancellationRequested then
raise (new System.TimeoutException())
else reraise()