There's lower level approach to writing and reading stream for web requests, where you read or write as data comes in, please help with a snippet.
let request = WebRequest.CreateHttp url
request.Method <- "PUT"
async {
request.ContentLength <- (int64) schema.Length
use! requestStream = request.GetRequestStreamAsync() |> Async.AwaitTask
requestStream.Write(Encoding.UTF8.GetBytes(schema), 0, schema.Length)
requestStream.Close()
use! response = request.AsyncGetResponse()
use stream = response.GetResponseStream()
use streamReader = new StreamReader(stream)
let! data = streamReader.ReadToEndAsync() |> Async.AwaitTask
return Ok(data)
}
References used for the above code.
http://www.fssnip.net/7PK/title/Send-async-HTTP-POST-request
Instead of using ReadToEndAsync, use ReadAsync in a while loop. There are plenty of examples in C# about how to use these api's here is a simple one: ReadAsync get data from buffer
Also there are Async based stream apis in FSharp.Core here a code sample: http://www.fssnip.net/nP/title/Async-demo
Related
I have an application that calls an API via the below function.
Public Function CreateWebRequest(ByVal strURL As String) As JArray
Try
Dim request As HttpWebRequest = DirectCast(WebRequest.Create(strURL), HttpWebRequest)
request.Method="GET"
request.ContentType = "application/json"
request.Credentials = CredentialCache.DefaultNetworkCredentials
Using response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse)
Dim reader As StreamReader
reader = New StreamReader(response.GetResponseStream())
Dim rawresp As String
rawresp = reader.ReadToEnd()
Dim array As JArray = JArray.Parse(rawresp)
reader.Close()
response.Close()
Return array
End Using
Catch ex As Exception
Dim empty As New JArray
Return empty
End Try
End Function
I noticed recently that sometimes I get a timeout on my server (Taking too much time to respond). Digging more into details, I wanted to check if I have a memory leaking issue with the function I used to call the API.
Is there anything wrong in the above function? Do I need to close any connection or clear any memory that I haven't done?
I have an F# funciton that uses a static instance of HttpClient:
let executeRequest request =
async {
let! response = StaticHttpClient.Instance.SendAsync(request) |> Async.AwaitTask
let! stream = response.Content.ReadAsStreamAsync() |> Async.AwaitTask
return (stream, response.StatusCode)
}
|> Async.RunSynchronously
When the request body is large, the function often throws an AggregateException with inner exception "NotSupportedException: The stream does not support concurrent IO read or write operations."
I wonder why this happens. Looks like there is an attempt to use response stream before the request is stream is fully processed. But why?
I'm writing the following code to post to a Web API. However, I got compiler error on the line of client.PostAsJsonAsync. The error message is
Error This expression was expected to have type
Async<'a>
but here has type
Tasks.Task<HttpResponseMessage>
code:
[<CLIMutable>]
type Model = { ..... }
let PostIt params = async {
use client = new HttpClient()
let content = { ..... } // a Model built from params
let! response = client.PostAsJsonAsync("http://...", content) // Error!
return response }
What's the best way to handle Restful API in F#? I'm using Fsharp.Data.
It seems like you need to use Async.AwaitTask:
let! response = Async.AwaitTask (client.PostAsJsonAsync("http://...", content))
Or using the |> operator:
let! response = client.PostAsJsonAsync("http://...", content) |> Async.AwaitTask
If you already have an F# Data reference, you can also do this using the F# Data HTTP utilities, which provides an F#-friendly API for making HTTP requests.
async {
let! response =
Http.AsyncRequest
( "http://httpbin.org/post", httpMethod = "POST",
headers = [ ContentType HttpContentTypes.Json ],
body = TextRequest """ {"test": 42} """)
return response }
F# Data will not automatically serialize data for you though, so the drawback of using these utilities is that you'll need to serialize the data explicitly before making the request.
I read the following code from the book Expert f#,
Why the function collectLinks embeds let! html = async { .... } in the outer async block? How about just flat it by removing the inner async?
Same question for the function waitForUrl in urlCollector which has a do! Async.StartChild (async {....}) |> Async.Ignore in an outer async block. How about flat it?
How is the implementation comparing with the one implemented with block queue? https://msdn.microsoft.com/en-us/library/vstudio/hh297096(v=vs.100).aspx Creating a block queue with 5, and en-queue the link to producer.
Code:
open System.Collections.Generic
open System.Net
open System.IO
open System.Threading
open System.Text.RegularExpressions
let limit = 50
let linkPat = "href=\s*\"[^\"h]*(http://[^&\"]*)\""
let getLinks (txt:string) =
[ for m in Regex.Matches(txt,linkPat) -> m.Groups.Item(1).Value ]
// A type that helps limit the number of active web requests
type RequestGate(n:int) =
let semaphore = new Semaphore(initialCount=n, maximumCount=n)
member x.AsyncAcquire(?timeout) =
async { let! ok = Async.AwaitWaitHandle(semaphore,
?millisecondsTimeout=timeout)
if ok then
return
{ new System.IDisposable with
member x.Dispose() =
semaphore.Release() |> ignore }
else
return! failwith "couldn't acquire a semaphore" }
// Gate the number of active web requests
let webRequestGate = RequestGate(5)
// Fetch the URL, and post the results to the urlCollector.
let collectLinks (url:string) =
async { // An Async web request with a global gate
let! html =
async { // Acquire an entry in the webRequestGate. Release
// it when 'holder' goes out of scope
use! holder = webRequestGate.AsyncAcquire()
let req = WebRequest.Create(url,Timeout=5)
// Wait for the WebResponse
use! response = req.AsyncGetResponse()
// Get the response stream
use reader = new StreamReader(response.GetResponseStream())
// Read the response stream (note: a synchronous read)
return reader.ReadToEnd() }
// Compute the links, synchronously
let links = getLinks html
// Report, synchronously
do printfn "finished reading %s, got %d links" url (List.length links)
// We're done
return links }
/// 'urlCollector' is a single agent that receives URLs as messages. It creates new
/// asynchronous tasks that post messages back to this object.
let urlCollector =
MailboxProcessor.Start(fun self ->
// This is the main state of the urlCollector
let rec waitForUrl (visited : Set<string>) =
async { // Check the limit
if visited.Count < limit then
// Wait for a URL...
let! url = self.Receive()
if not (visited.Contains(url)) then
// Start off a new task for the new url. Each collects
// links and posts them back to the urlCollector.
do! Async.StartChild
(async { let! links = collectLinks url
for link in links do
self.Post link }) |> Async.Ignore
// Recurse into the waiting state
return! waitForUrl(visited.Add(url)) }
// This is the initial state.
waitForUrl(Set.empty))
I can think of one reason why async code would call another async block, which is that it lets you dispose of resources earlier - when the nested block completes. To demonstrate this, here is a little helper that prints a message when Dispose is called:
let printOnDispose text =
{ new System.IDisposable with
member x.Dispose() = printfn "%s" text }
The following uses nested async to do something in a nested block and then cleanup the local resources used in the nested block. Then it sleeps some more and cleans up resources used in the outer block:
async {
use bye = printOnDispose "bye from outer block"
let! r = async {
use bye = printOnDispose "bye from nested block"
do! Async.Sleep(1000)
return 1 }
do! Async.Sleep(1000) }
|> Async.Start
Here, the "nested block" resources are disposed of after 1 second and the outer block resources are disposed of after 2 seconds.
There are other cases where nesting async is useful (like returning from an asynchronous block containing try .. with), but I don't think that applies here.
I'm trying to download a Fusion table programatically using C# only not .NET. I'm having trouble finding a decent API or C# example code to do so. Is that even possible?
HttpWebRequest request = WebRequest.Create(this.url2 + "?sql=SELECT * FROM 'table id'") as HttpWebRequest;
request.Method = "GET";
request.Headers.Add("Authorization", "GoogleLogin auth=" + this.token);
// Get response
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream
StreamReader reader = new StreamReader(response.GetResponseStream());
// Console application output
return reader.ReadToEnd();
}
What am I doing wrong?