I have an F# funciton that uses a static instance of HttpClient:
let executeRequest request =
async {
let! response = StaticHttpClient.Instance.SendAsync(request) |> Async.AwaitTask
let! stream = response.Content.ReadAsStreamAsync() |> Async.AwaitTask
return (stream, response.StatusCode)
}
|> Async.RunSynchronously
When the request body is large, the function often throws an AggregateException with inner exception "NotSupportedException: The stream does not support concurrent IO read or write operations."
I wonder why this happens. Looks like there is an attempt to use response stream before the request is stream is fully processed. But why?
Related
I'm having a bit of an issue converting some Microsoft playwright code from C# to F#
Specifically this code: https://playwright.dev/dotnet/docs/navigations#multiple-navigations
// Running action in the callback of waitForNavigation prevents a race
// condition between clicking and waiting for a navigation.
await page.RunAndWaitForNavigationAsync(async () =>
{
// Triggers a navigation with a script redirect.
await page.ClickAsync("a");
}, new PageWaitForNavigationOptions
{
UrlString = "**/login"
});
My F# code is a little separated and specific to my requirements - but here is the attempt so far ( which doesn't work )
let waitNavOptions = new PageWaitForNavigationOptions(UrlRegex=Regex("dashboard|login",RegexOptions.IgnoreCase))
do! Async.AwaitTask(page.Value.RunAndWaitForNavigationAsync(page.Value.ClickAsync("#xl-form-submit"),waitNavOptions))
let waitNavOptions = PageRunAndWaitForNavigationOptions(UrlRegex=Regex("dashboard|login",RegexOptions.IgnoreCase))
do!
page.Value.RunAndWaitForNavigationAsync(
(fun () -> page.Value.ClickAsync("#xl-form-submit")),
waitNavOptions)
|> Async.AwaitTask
|> Async.Ignore
There were a few things to fix here:
Changed PageWaitForNavigationOptions to PageRunAndWaitForNavigationOptions
Change the first method argument to a function returning a task, instead of just a task.
Ignore the Async result at the end so that do! is allowed
There's lower level approach to writing and reading stream for web requests, where you read or write as data comes in, please help with a snippet.
let request = WebRequest.CreateHttp url
request.Method <- "PUT"
async {
request.ContentLength <- (int64) schema.Length
use! requestStream = request.GetRequestStreamAsync() |> Async.AwaitTask
requestStream.Write(Encoding.UTF8.GetBytes(schema), 0, schema.Length)
requestStream.Close()
use! response = request.AsyncGetResponse()
use stream = response.GetResponseStream()
use streamReader = new StreamReader(stream)
let! data = streamReader.ReadToEndAsync() |> Async.AwaitTask
return Ok(data)
}
References used for the above code.
http://www.fssnip.net/7PK/title/Send-async-HTTP-POST-request
Instead of using ReadToEndAsync, use ReadAsync in a while loop. There are plenty of examples in C# about how to use these api's here is a simple one: ReadAsync get data from buffer
Also there are Async based stream apis in FSharp.Core here a code sample: http://www.fssnip.net/nP/title/Async-demo
I'm writing the following code to post to a Web API. However, I got compiler error on the line of client.PostAsJsonAsync. The error message is
Error This expression was expected to have type
Async<'a>
but here has type
Tasks.Task<HttpResponseMessage>
code:
[<CLIMutable>]
type Model = { ..... }
let PostIt params = async {
use client = new HttpClient()
let content = { ..... } // a Model built from params
let! response = client.PostAsJsonAsync("http://...", content) // Error!
return response }
What's the best way to handle Restful API in F#? I'm using Fsharp.Data.
It seems like you need to use Async.AwaitTask:
let! response = Async.AwaitTask (client.PostAsJsonAsync("http://...", content))
Or using the |> operator:
let! response = client.PostAsJsonAsync("http://...", content) |> Async.AwaitTask
If you already have an F# Data reference, you can also do this using the F# Data HTTP utilities, which provides an F#-friendly API for making HTTP requests.
async {
let! response =
Http.AsyncRequest
( "http://httpbin.org/post", httpMethod = "POST",
headers = [ ContentType HttpContentTypes.Json ],
body = TextRequest """ {"test": 42} """)
return response }
F# Data will not automatically serialize data for you though, so the drawback of using these utilities is that you'll need to serialize the data explicitly before making the request.
I read the following code from the book Expert f#,
Why the function collectLinks embeds let! html = async { .... } in the outer async block? How about just flat it by removing the inner async?
Same question for the function waitForUrl in urlCollector which has a do! Async.StartChild (async {....}) |> Async.Ignore in an outer async block. How about flat it?
How is the implementation comparing with the one implemented with block queue? https://msdn.microsoft.com/en-us/library/vstudio/hh297096(v=vs.100).aspx Creating a block queue with 5, and en-queue the link to producer.
Code:
open System.Collections.Generic
open System.Net
open System.IO
open System.Threading
open System.Text.RegularExpressions
let limit = 50
let linkPat = "href=\s*\"[^\"h]*(http://[^&\"]*)\""
let getLinks (txt:string) =
[ for m in Regex.Matches(txt,linkPat) -> m.Groups.Item(1).Value ]
// A type that helps limit the number of active web requests
type RequestGate(n:int) =
let semaphore = new Semaphore(initialCount=n, maximumCount=n)
member x.AsyncAcquire(?timeout) =
async { let! ok = Async.AwaitWaitHandle(semaphore,
?millisecondsTimeout=timeout)
if ok then
return
{ new System.IDisposable with
member x.Dispose() =
semaphore.Release() |> ignore }
else
return! failwith "couldn't acquire a semaphore" }
// Gate the number of active web requests
let webRequestGate = RequestGate(5)
// Fetch the URL, and post the results to the urlCollector.
let collectLinks (url:string) =
async { // An Async web request with a global gate
let! html =
async { // Acquire an entry in the webRequestGate. Release
// it when 'holder' goes out of scope
use! holder = webRequestGate.AsyncAcquire()
let req = WebRequest.Create(url,Timeout=5)
// Wait for the WebResponse
use! response = req.AsyncGetResponse()
// Get the response stream
use reader = new StreamReader(response.GetResponseStream())
// Read the response stream (note: a synchronous read)
return reader.ReadToEnd() }
// Compute the links, synchronously
let links = getLinks html
// Report, synchronously
do printfn "finished reading %s, got %d links" url (List.length links)
// We're done
return links }
/// 'urlCollector' is a single agent that receives URLs as messages. It creates new
/// asynchronous tasks that post messages back to this object.
let urlCollector =
MailboxProcessor.Start(fun self ->
// This is the main state of the urlCollector
let rec waitForUrl (visited : Set<string>) =
async { // Check the limit
if visited.Count < limit then
// Wait for a URL...
let! url = self.Receive()
if not (visited.Contains(url)) then
// Start off a new task for the new url. Each collects
// links and posts them back to the urlCollector.
do! Async.StartChild
(async { let! links = collectLinks url
for link in links do
self.Post link }) |> Async.Ignore
// Recurse into the waiting state
return! waitForUrl(visited.Add(url)) }
// This is the initial state.
waitForUrl(Set.empty))
I can think of one reason why async code would call another async block, which is that it lets you dispose of resources earlier - when the nested block completes. To demonstrate this, here is a little helper that prints a message when Dispose is called:
let printOnDispose text =
{ new System.IDisposable with
member x.Dispose() = printfn "%s" text }
The following uses nested async to do something in a nested block and then cleanup the local resources used in the nested block. Then it sleeps some more and cleans up resources used in the outer block:
async {
use bye = printOnDispose "bye from outer block"
let! r = async {
use bye = printOnDispose "bye from nested block"
do! Async.Sleep(1000)
return 1 }
do! Async.Sleep(1000) }
|> Async.Start
Here, the "nested block" resources are disposed of after 1 second and the outer block resources are disposed of after 2 seconds.
There are other cases where nesting async is useful (like returning from an asynchronous block containing try .. with), but I don't think that applies here.
I'm trying to make an asynchronous web request to a URL that will return if the request takes too long. I'm using the F# asynchronous workflow and the System.Net.Http library to do this.
However, I am unable to catch the Task/OperationCancelledExceptions that are raised by the System.Net.Http library in the async workflow. Instead, the exception is raised at the Async.RunSynchronously method, as you can see in this stack trace:
> System.OperationCanceledException: The operation was canceled. at
> Microsoft.FSharp.Control.AsyncBuilderImpl.commit[a](Result`1 res)
> at
> Microsoft.FSharp.Control.CancellationTokenOps.RunSynchronously[a](CancellationToken
> token, FSharpAsync`1 computation, FSharpOption`1 timeout) at
> Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync`1
> computation, FSharpOption`1 timeout, FSharpOption`1 cancellationToken)
> at <StartupCode$FSI_0004>.$FSI_0004.main#()
The code:
#r "System.Net.Http"
open System.Net.Http
open System
let readGoogle () = async {
try
let request = new HttpRequestMessage(HttpMethod.Get, "https://google.co.uk")
let client = new HttpClient()
client.Timeout <- TimeSpan.FromSeconds(0.01) //intentionally low to always fail in this example
let! response = client.SendAsync(request, HttpCompletionOption.ResponseContentRead) |> Async.AwaitTask
return Some response
with
| ex ->
//is never called
printfn "TIMED OUT"
return None
}
//exception is raised here
readGoogle ()
|> Async.RunSynchronously
|> ignore
Cancellation was always different from the error. In your case you can override default behavior of AwaitTask that invokes "cancel continuation" if task is cancelled and handle it differently:
let readGoogle () = async {
try
let request = new HttpRequestMessage(HttpMethod.Get, "https://google.co.uk")
let client = new HttpClient()
client.Timeout <- TimeSpan.FromSeconds(0.01) //intentionally low to always fail in this example
return! (
let t = client.SendAsync(request, HttpCompletionOption.ResponseContentRead)
Async.FromContinuations(fun (s, e, _) ->
t.ContinueWith(fun (t: Task<_>) ->
// if task is cancelled treat it as timeout and process on success path
if t.IsCanceled then s(None)
elif t.IsFaulted then e(t.Exception)
else s(Some t.Result)
)
|> ignore
)
)
with
| ex ->
//is never called
printfn "TIMED OUT"
return None
}