I can't work out why the following code is hanging at the call to GetTotal. I don't seem to be able to debug inside the MailboxProcessor, so it's hard to see what's going on.
module Aggregator
open System
type Message<'T, 'TState> =
| Aggregate of 'T
| GetTotal of AsyncReplyChannel<'TState>
type Aggregator<'T, 'TState>(initialState, f) =
let myAgent = new MailboxProcessor<Message<'T, 'TState>>(fun inbox ->
let rec loop agg =
async {
let! message = inbox.Receive()
match message with
| Aggregate x -> return! loop (f agg x)
| GetTotal replyChannel ->
replyChannel.Reply(agg)
return! loop agg
}
loop initialState
)
member m.Aggregate x = myAgent.Post(Aggregate(x))
member m.GetTotal = myAgent.PostAndReply(fun replyChannel -> GetTotal(replyChannel))
let myAggregator = new Aggregator<int, int>(0, (+))
myAggregator.Aggregate(3)
myAggregator.Aggregate(4)
myAggregator.Aggregate(5)
let totalSoFar = myAggregator.GetTotal
printfn "%d" totalSoFar
Console.ReadLine() |> ignore
It seems to work fine when using an identical MailboxProcessor directly, rather than wrapping in the Aggregator class.
The problem is that you did not start the agent. You can either call Start after you create the agent:
let myAgent = (...)
do myAgent.Start()
Alternatively, you can create the agent using MailboxProcessor<'T>.Start instead of calling the constructor (I usually prefer this option, because it looks more functional):
let myAgent = MailboxProcessor<Message<'T, 'TState>>.Start(fun inbox -> (...) )
I suppose that you couldn't debug the agent, because the code inside agent wasn't actually running. I tried adding printfn "Msg: %A" message right after the call to Receive inside the agent (to print incoming messages for debugging) and I noticed that, after calling Aggregate, no messages were actually received by the agent... (It only blocked after calling GetTotal, which avaits reply)
As a side-note, I would probably turn GetTotal into a method, so you'd call GetTotal(). Properties are re-evaluated each time you access them, so your code does the same thing, but best practices don't recommend using properties that do complex work.
You forgot to start the mailbox:
open System
type Message<'T, 'TState> =
| Aggregate of 'T
| GetTotal of AsyncReplyChannel<'TState>
type Aggregator<'T, 'TState>(initialState, f) =
let myAgent = new MailboxProcessor<Message<'T, 'TState>>(fun inbox ->
let rec loop agg =
async {
let! message = inbox.Receive()
match message with
| Aggregate x -> return! loop (f agg x)
| GetTotal replyChannel ->
replyChannel.Reply(agg)
return! loop agg
}
loop initialState
)
member m.Aggregate x = myAgent.Post(Aggregate(x))
member m.GetTotal = myAgent.PostAndReply(fun replyChannel -> GetTotal(replyChannel))
member m.Start() = myAgent.Start()
let myAggregator = new Aggregator<int, int>(0, (+))
myAggregator.Start()
myAggregator.Aggregate(3)
myAggregator.Aggregate(4)
myAggregator.Aggregate(5)
let totalSoFar = myAggregator.GetTotal
printfn "%d" totalSoFar
Console.ReadLine() |> ignore
Related
I would like to create a chain of expressions and any of them can fail when the computation should just stop.
With Unix pipes it is usually like this:
bash-3.2$ echo && { echo 'a ok'; echo; } && { echo 'b ok'; echo; }
a ok
b ok
When something fails the pipeline stops:
echo && { echo 'a ok'; false; } && { echo 'b ok'; echo; }
a ok
I can handle Optionals but my problem is that I might want to do multiple things in each branch:
let someExternalOperation = callToAnAPI()
match someExternalOperation with
| None -> LogAndStop()
| Some x -> LogAndContinue()
Then I would like to keep going with other API calls and only stop if there is an error.
Is there something like that in F#?
Update1:
What I am trying to do is calling out to external APIs. Each call can fail. Would be nice to try to retry but not required.
You can use the F# Async and Result types together to represent the results of each API Call. You can then use the bind functions for those types to build a workflow in which you only continue processing when the previous calls were successful. In order to make that easier, you can wrap the Async<Result<_,_>> you would be working with for each api call in its own type and build a module around binding those results to orchestrate a chained computation. Here's a quick example of what that would look like:
First, we would lay out the type ApiCallResult to wrap Async and Result, and we would define ApiCallError to represent HTTP error responses or exceptions:
open System
open System.Net
open System.Net.Http
type ApiCallError =
| HttpError of (int * string)
| UnexpectedError of exn
type ApiCallResult<'a> = Async<Result<'a, ApiCallError>>
Next, we would create a module to work with ApiCallResult instances, allowing us to do things like bind, map, and return so that we can process the results of a computation and feed them into the next one.
module ApiCall =
let ``return`` x : ApiCallResult<_> =
async { return Ok x }
let private zero () : ApiCallResult<_> =
``return`` []
let bind<'a, 'b> (f: 'a -> ApiCallResult<'b>) (x: ApiCallResult<'a>) : ApiCallResult<'b> =
async {
let! result = x
match result with
| Ok value ->
return! f value
| Error error ->
return Error error
}
let map f x = x |> bind (f >> ``return``)
let combine<'a> (acc: ApiCallResult<'a list>) (cur: ApiCallResult<'a>) =
acc |> bind (fun values -> cur |> map (fun value -> value :: values))
let join results =
results |> Seq.fold (combine) (zero ())
Then, you would have a module to simply do your API calls, however that works in your real scenario. Here's one that just handles GETs with query parameters, but you could make this more sophisticated:
module Api =
let call (baseUrl: Uri) (queryString: string) : ApiCallResult<string> =
async {
try
use client = new HttpClient()
let url =
let builder = UriBuilder(baseUrl)
builder.Query <- queryString
builder.Uri
printfn "Calling API: %O" url
let! response = client.GetAsync(url) |> Async.AwaitTask
let! content = response.Content.ReadAsStringAsync() |> Async.AwaitTask
if response.IsSuccessStatusCode then
let! content = response.Content.ReadAsStringAsync() |> Async.AwaitTask
return Ok content
else
return Error <| HttpError (response.StatusCode |> int, content)
with ex ->
return Error <| UnexpectedError ex
}
let getQueryParam name value =
value |> WebUtility.UrlEncode |> sprintf "%s=%s" name
Finally, you would have your actual business workflow logic, where you call multiple APIs and feed the results of one into another. In the below example, anywhere you see callMathApi, it is making a call to an external REST API that may fail, and by using the ApiCall module to bind the results of the API call, it only proceeds to the next API call if the previous call was successful. You can declare an operator like >>= to eliminate some of the noise in the code when binding computations together:
module MathWorkflow =
let private (>>=) x f = ApiCall.bind f x
let private apiUrl = Uri "http://api.mathjs.org/v4/" // REST API for mathematical expressions
let private callMathApi expression =
expression |> Api.getQueryParam "expr" |> Api.call apiUrl
let average values =
values
|> List.map (sprintf "%d")
|> String.concat "+"
|> callMathApi
>>= fun sum ->
sprintf "%s/%d" sum values.Length
|> callMathApi
let averageOfSquares values =
values
|> List.map (fun value -> sprintf "%d*%d" value value)
|> List.map callMathApi
|> ApiCall.join
|> ApiCall.map (List.map int)
>>= average
This example uses the Mathjs.org API to compute the average of a list of integers (making one API call to compute the sum, then another to divide by the number of elements), and also allows you to compute the average of the squares of a list of values, by calling the API asynchronously for each element in the list to square it, then joining the results together and computing the average. You can use these functions as follows (I added a printfn to the actual API call so it logs the HTTP requests):
Calling average:
MathWorkflow.average [1;2;3;4;5] |> Async.RunSynchronously
Outputs:
Calling API: http://api.mathjs.org/v4/?expr=1%2B2%2B3%2B4%2B5
Calling API: http://api.mathjs.org/v4/?expr=15%2F5
[<Struct>]
val it : Result<string,ApiCallError> = Ok "3"
Calling averageOfSquares:
MathWorkflow.averageOfSquares [2;4;6;8;10] |> Async.RunSynchronously
Outputs:
Calling API: http://api.mathjs.org/v4/?expr=2*2
Calling API: http://api.mathjs.org/v4/?expr=4*4
Calling API: http://api.mathjs.org/v4/?expr=6*6
Calling API: http://api.mathjs.org/v4/?expr=8*8
Calling API: http://api.mathjs.org/v4/?expr=10*10
Calling API: http://api.mathjs.org/v4/?expr=100%2B64%2B36%2B16%2B4
Calling API: http://api.mathjs.org/v4/?expr=220%2F5
[<Struct>]
val it : Result<string,ApiCallError> = Ok "44"
Ultimately, you may want to implement a custom Computation Builder to allow you to use a computation expression with the let! syntax, instead of explicitly writing the calls to ApiCall.bind everywhere. This is fairly simple, since you already do all the real work in the ApiCall module, and you just need to make a class with the appropriate Bind/Return members:
type ApiCallBuilder () =
member __.Bind (x, f) = ApiCall.bind f x
member __.Return x = ApiCall.``return`` x
member __.ReturnFrom x = x
member __.Zero () = ApiCall.``return`` ()
let apiCall = ApiCallBuilder()
With the ApiCallBuilder, you could rewrite the functions in the MathWorkflow module like this, making them a little easier to read and compose:
let average values =
apiCall {
let! sum =
values
|> List.map (sprintf "%d")
|> String.concat "+"
|> callMathApi
return!
sprintf "%s/%d" sum values.Length
|> callMathApi
}
let averageOfSquares values =
apiCall {
let! squares =
values
|> List.map (fun value -> sprintf "%d*%d" value value)
|> List.map callMathApi
|> ApiCall.join
return! squares |> List.map int |> average
}
These work as you described in the question, where each API call is made independently and the results feed into the next call, but if one call fails the computation is stopped and the error is returned. For example, if you change the URL used in the example calls here to the v3 API ("http://api.mathjs.org/v3/") without changing anything else, you get the following:
Calling API: http://api.mathjs.org/v3/?expr=2*2
[<Struct>]
val it : Result<string,ApiCallError> =
Error
(HttpError
(404,
"<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /v3/</pre>
</body>
</html>
"))
Building on a snippet and answer, would it be possible to return results to the caller from the throttling queue? I've tried PostAndAsyncReply to receive reply on a channel but it's throwing an error if I pipe it with Enqueue. Here's the code.
Appreciate a F# core vanilla based solution around Queue or Mailbox design patterns.
Question
The question is to be able to call functions asynchronously based on the throttle (max 3 at a time), passing each item from the array, wait on the whole queue/array until it's finished while collecting all the results and then return the results to the caller. (Return the results to the caller is what's pending in here)
Callee Code
// Message type used by the agent - contains queueing
// of work items and notification of completion
type ThrottlingAgentMessage =
| Completed
| Enqueue of Async<unit>
/// Represents an agent that runs operations in concurrently. When the number
/// of concurrent operations exceeds 'limit', they are queued and processed later
let throttlingAgent limit =
MailboxProcessor.Start(fun inbox ->
async {
// The agent body is not executing in parallel,
// so we can safely use mutable queue & counter
let queue = System.Collections.Generic.Queue<Async<unit>>()
let running = ref 0
while true do
// Enqueue new work items or decrement the counter
// of how many tasks are running in the background
let! msg = inbox.Receive()
match msg with
| Completed -> decr running
| Enqueue w -> queue.Enqueue(w)
// If we have less than limit & there is some work to
// do, then start the work in the background!
while running.Value < limit && queue.Count > 0 do
let work = queue.Dequeue()
incr running
do! // When the work completes, send 'Completed'
// back to the agent to free a slot
async {
do! work
inbox.Post(Completed)
}
|> Async.StartChild
|> Async.Ignore
})
let requestDetailAsync (url: string) : Async<Result<string, Error>> =
async {
Console.WriteLine ("Simulating request " + url)
try
do! Async.Sleep(1000) // let's say each request takes about a second
return Ok (url + ":body...")
with :? WebException as e ->
return Error {Code = "500"; Message = "Internal Server Error"; Status = HttpStatusCode.InternalServerError}
}
let requestMasterAsync() : Async<Result<System.Collections.Concurrent.ConcurrentBag<_>, Error>> =
async {
let urls = [|
"http://www.example.com/1";
"http://www.example.com/2";
"http://www.example.com/3";
"http://www.example.com/4";
"http://www.example.com/5";
"http://www.example.com/6";
"http://www.example.com/7";
"http://www.example.com/8";
"http://www.example.com/9";
"http://www.example.com/10";
|]
let results = System.Collections.Concurrent.ConcurrentBag<_>()
let agent = throttlingAgent 3
for url in urls do
async {
let! res = requestDetailAsync url
results.Add res
}
|> Enqueue
|> agent.Post
return Ok results
}
Caller Code
[<TestMethod>]
member this.TestRequestMasterAsync() =
match Entity.requestMasterAsync() |> Async.RunSynchronously with
| Ok result -> Console.WriteLine result
| Error error -> Console.WriteLine error
You could use Hopac.Streams for that. With such tool it is pretty trivial:
open Hopac
open Hopac.Stream
open System
let requestDetailAsync url = async {
Console.WriteLine ("Simulating request " + url)
try
do! Async.Sleep(1000) // let's say each request takes about a second
return Ok (url + ":body...")
with :? Exception as e ->
return Error e
}
let requestMasterAsync() : Stream<Result<string,exn>> =
[| "http://www.example.com/1"
"http://www.example.com/2"
"http://www.example.com/3"
"http://www.example.com/4"
"http://www.example.com/5"
"http://www.example.com/6"
"http://www.example.com/7"
"http://www.example.com/8"
"http://www.example.com/9"
"http://www.example.com/10" |]
|> Stream.ofSeq
|> Stream.mapPipelinedJob 3 (requestDetailAsync >> Job.fromAsync)
requestMasterAsync()
|> Stream.iterFun (printfn "%A")
|> queue //prints all results asynchronously
let allResults : Result<string,exn> list =
requestMasterAsync()
|> Stream.foldFun (fun results cur -> cur::results ) []
|> run //fold stream into list synchronously
ADDED
In case you want to use only vanilla FSharp.Core with mailboxes only try this:
type ThrottlingAgentMessage =
| Completed
| Enqueue of Async<unit>
let inline (>>=) x f = async.Bind(x, f)
let inline (>>-) x f = async.Bind(x, f >> async.Return)
let throttlingAgent limit =
let agent = MailboxProcessor.Start(fun inbox ->
let queue = System.Collections.Generic.Queue<Async<unit>>()
let startWork work =
work
>>- fun _ -> inbox.Post Completed
|> Async.StartChild |> Async.Ignore
let rec loop curWorkers =
inbox.Receive()
>>= function
| Completed when queue.Count > 0 ->
queue.Dequeue() |> startWork
>>= fun _ -> loop curWorkers
| Completed ->
loop (curWorkers - 1)
| Enqueue w when curWorkers < limit ->
w |> startWork
>>= fun _ -> loop (curWorkers + 1)
| Enqueue w ->
queue.Enqueue w
loop curWorkers
loop 0)
Enqueue >> agent.Post
It is pretty much the same logic, but slightly optimized to not use queue if there is free worker capacity (just start job and don't bother with queue/dequeue).
throttlingAgent is a function int -> Async<unit> -> unit
Because we don't want client to bother with our internal ThrottlingAgentMessage type.
use like this:
let throttler = throttlingAgent 3
for url in urls do
async {
let! res = requestDetailAsync url
results.Add res
}
|> throttler
Please see last edit.
Apologies for the newbie question. I am trying to implement something in F# using Akka.net. I'm very new to F# and I have only used Akka from Scala. Basically I am trying to implement something that's pretty easy in Scala, namely making an Actor do different things based on the type of message it receives.
My code is below and it's a slight modification of the hello world example lifted from the akka.net website. I believe a first problem with my code is that it does record pattern matching instead of type pattern matching, however I was unable to write a type match one without compilation errors... Any help will be greatly appreciated. Thank you.
open Akka.FSharp
open Actors
open Akka
open Akka.Actor
type Entries = { Entries: List<string>}
let system = ActorSystem.Create "MySystem"
let feedBrowser = spawn system "feedBrowser" <| fun mailbox ->
let rec loop() = actor {
let! msg = mailbox.Receive()
match msg with
| { Entries = entries} -> printf "%A" entries
| _ -> printf "unmatched message %A" msg
return! loop()}
loop()
[<EntryPoint>]
let main argv =
feedBrowser <! "abc" // this should not blow up but it does
system.AwaitTermination()
0
Edit: the error is a runtime one, System.InvalidCastException, unable to cast object of type String to Entries.
Later edit: I got this to work with this change, downcasting to Object:
let feedBrowser = spawn system "feedBrowser" <| fun mailbox ->
let rec loop() = actor {
let! msg = mailbox.Receive()
let msgObj = msg :> Object
match msgObj with
| :? Entries as e -> printfn "matched message %A" e
| _ -> printf "unmatched message %A" msg
return! loop()}
loop()
Now these two lines work correctly
feedBrowser <! "abc"
feedBrowser <! { Entries = ["a"; "b"] }
the first one prints "unmatched message abc" and the second outputs the entries.
Is there a better way of going about this, without the cast? Does akka.net have something specifically for this case?
Thank you.
You should use a Discriminated Union (the Command type in this example). Then you can pattern match its options.
type Entries = { Entries: List<string>}
type Command =
| ListEntries of Entries
| OtherCommand of string
let stack() =
let system = ActorSystem.Create "MySystem"
let feedBrowser = spawn system "feedBrowser" <| fun mailbox ->
let rec loop() = actor {
let! msg = mailbox.Receive()
match msg with
| ListEntries { Entries = entries} -> printf "%A" entries
| OtherCommand s -> printf "%s" s
return! loop() }
loop()
And to send the message you should use:
feedBrowser <! OtherCommand "abc"
feedBrowser <! ListEntries { Entries = ["a"; "b"] }
It's important to say that the send operator has the following signature:
#ICanTell -> obj -> unit
So, if you pass an message with a different type, like a string, it'll raise an exception.
I am using F# mailBoxProcessor to asynchronously process messages received from multiple network ends.
The code works as expected until I added function call getTreasuryYield after inbox.receive().
It gets stuck every time at inbox.receive() after running for a few seconds.
GetTreasuryYield is a quite slow method since it involves database and IO operations, but I
still do not understand how it gets stuck.
Any HELP will be appreciated.
let start rvSetting (agent:Agent<Message>) messageSelector=
try
TIBCO.Rendezvous.Environment.Open()
let _transport = new NetTransport(rvSetting.rvService, rvSetting.rvNetwork, rvSetting.rvDaemon)
let _listener = new Listener(TIBCO.Rendezvous.Queue.Default, _transport, rvSetting.rvSubject, null)
_listener.MessageReceived.Add(fun args->
printfn "before sent"
if messageSelector(args.Message) then
printfn "Message sent to agent: %A" args.Message
agent.Post(args.Message))
let rec dispatch() =
async{
try
TIBCO.Rendezvous.Queue.Default.Dispatch()
return! dispatch()
with
| e -> _log.Error(e.ToString())
}
Async.Start(dispatch())
with
|e -> printfn "%A" e.Message
_log.Error(e.Message)
let agent = new Agent<Message>(fun inbox ->
let rec loop() =
async{
let! (m : Message) = inbox.Receive()
// This line causes the problem
printfn "%A" (getTreasuryYieldFromMessage m)
Async.Start(treasuryAction m)
return! loop()
}
loop())
agent.Error.Add raise
[<EntryPoint>]
let main argv =
//start rvCorporate agent (fun x -> true)
agent.Start()
start rvTreasury agent treasurySelector
Console.ReadLine() |> ignore
0
Given the following agent, which is a simple cache mechanism:
type CacheMsg<'a,'b> = Add of 'a * 'b | ForceFlush
type CacheAgent<'a, 'b when 'a : comparison>(size:int, flushCont:Map<'a, 'b> -> unit) =
let agent = MailboxProcessor.Start(fun inbox ->
let rec loop (cache : Map<'a, 'b>) = async {
let inline flush() =
flushCont cache
loop Map.empty
if cache.Count > size then return! flush()
let! msg = inbox.Receive()
match msg with
| Add (key, value) ->
if cache.ContainsKey key then
return! loop cache
else return! loop (cache.Add(key, value))
| ForceFlush -> return! flush() }
loop Map.empty)
member x.AddIfNotExists key value = Add(key,value) |> agent.Post
member x.ForceFlush() = agent.Post ForceFlush
This agent will keep taking up memory (seems like the memory is not freed when the flushCont has been called).
Given the same code, but with a minor change:
type CacheMsg<'a,'b> = Add of 'a * 'b | ForceFlush
type CacheAgent<'a, 'b when 'a : comparison>(size:int, flushCont:Map<'a, 'b> -> unit) =
let agent = MailboxProcessor.Start(fun inbox ->
let rec loop (cache : Map<'a, 'b>) = async {
let inline flush() =
flushCont cache
loop Map.empty
let! msg = inbox.Receive()
match msg with
| Add (key, value) ->
if cache.ContainsKey key then
return! loop cache
else
let newCache = cache.Add(key, value)
if newCache.Count > size then
return! flush()
else return! loop (cache.Add(key, value))
| ForceFlush -> return! flush() }
loop Map.empty)
member x.AddIfNotExists key value = Add(key,value) |> agent.Post
member x.ForceFlush() = agent.Post ForceFlush
I have moved the expression that decides when to flush, into the union case Add. This results in the memory is freed as expected.
What's wrong about the first approach, since it leaks memory?
The first version isn't tail recursive.
It's not tail recursive, because this expression isn't the last expression in the function:
if cache.Count > size then return! flush()
After that expression, you call
let! msg = inbox.Receive()
so the flush() call isn't the last thing happening. After the recursive call implicit in flush has completed, the execution will need to return to the next expression, where you invoke inbox.Receive(). That means that the context will have to keep the previous invocation on the stack, because the recursion isn't in a tail position: there's still more work to do.
In the second example, all calls to flush and loop are in tail positions.
If you're coming from a C# background, you'd be inclined to think that return! flush() exits the function, but that's not really the case here. The only reason
if cache.Count > size then return! flush()
even compiles without a corresponding else branch is because the expression returns unit. This means that the code inside the then branch doesn't truly exit the function - it just performs the work in the branch (in this case flush()), and then continues executing the subsequent expressions.