I would like to create a chain of expressions and any of them can fail when the computation should just stop.
With Unix pipes it is usually like this:
bash-3.2$ echo && { echo 'a ok'; echo; } && { echo 'b ok'; echo; }
a ok
b ok
When something fails the pipeline stops:
echo && { echo 'a ok'; false; } && { echo 'b ok'; echo; }
a ok
I can handle Optionals but my problem is that I might want to do multiple things in each branch:
let someExternalOperation = callToAnAPI()
match someExternalOperation with
| None -> LogAndStop()
| Some x -> LogAndContinue()
Then I would like to keep going with other API calls and only stop if there is an error.
Is there something like that in F#?
Update1:
What I am trying to do is calling out to external APIs. Each call can fail. Would be nice to try to retry but not required.
You can use the F# Async and Result types together to represent the results of each API Call. You can then use the bind functions for those types to build a workflow in which you only continue processing when the previous calls were successful. In order to make that easier, you can wrap the Async<Result<_,_>> you would be working with for each api call in its own type and build a module around binding those results to orchestrate a chained computation. Here's a quick example of what that would look like:
First, we would lay out the type ApiCallResult to wrap Async and Result, and we would define ApiCallError to represent HTTP error responses or exceptions:
open System
open System.Net
open System.Net.Http
type ApiCallError =
| HttpError of (int * string)
| UnexpectedError of exn
type ApiCallResult<'a> = Async<Result<'a, ApiCallError>>
Next, we would create a module to work with ApiCallResult instances, allowing us to do things like bind, map, and return so that we can process the results of a computation and feed them into the next one.
module ApiCall =
let ``return`` x : ApiCallResult<_> =
async { return Ok x }
let private zero () : ApiCallResult<_> =
``return`` []
let bind<'a, 'b> (f: 'a -> ApiCallResult<'b>) (x: ApiCallResult<'a>) : ApiCallResult<'b> =
async {
let! result = x
match result with
| Ok value ->
return! f value
| Error error ->
return Error error
}
let map f x = x |> bind (f >> ``return``)
let combine<'a> (acc: ApiCallResult<'a list>) (cur: ApiCallResult<'a>) =
acc |> bind (fun values -> cur |> map (fun value -> value :: values))
let join results =
results |> Seq.fold (combine) (zero ())
Then, you would have a module to simply do your API calls, however that works in your real scenario. Here's one that just handles GETs with query parameters, but you could make this more sophisticated:
module Api =
let call (baseUrl: Uri) (queryString: string) : ApiCallResult<string> =
async {
try
use client = new HttpClient()
let url =
let builder = UriBuilder(baseUrl)
builder.Query <- queryString
builder.Uri
printfn "Calling API: %O" url
let! response = client.GetAsync(url) |> Async.AwaitTask
let! content = response.Content.ReadAsStringAsync() |> Async.AwaitTask
if response.IsSuccessStatusCode then
let! content = response.Content.ReadAsStringAsync() |> Async.AwaitTask
return Ok content
else
return Error <| HttpError (response.StatusCode |> int, content)
with ex ->
return Error <| UnexpectedError ex
}
let getQueryParam name value =
value |> WebUtility.UrlEncode |> sprintf "%s=%s" name
Finally, you would have your actual business workflow logic, where you call multiple APIs and feed the results of one into another. In the below example, anywhere you see callMathApi, it is making a call to an external REST API that may fail, and by using the ApiCall module to bind the results of the API call, it only proceeds to the next API call if the previous call was successful. You can declare an operator like >>= to eliminate some of the noise in the code when binding computations together:
module MathWorkflow =
let private (>>=) x f = ApiCall.bind f x
let private apiUrl = Uri "http://api.mathjs.org/v4/" // REST API for mathematical expressions
let private callMathApi expression =
expression |> Api.getQueryParam "expr" |> Api.call apiUrl
let average values =
values
|> List.map (sprintf "%d")
|> String.concat "+"
|> callMathApi
>>= fun sum ->
sprintf "%s/%d" sum values.Length
|> callMathApi
let averageOfSquares values =
values
|> List.map (fun value -> sprintf "%d*%d" value value)
|> List.map callMathApi
|> ApiCall.join
|> ApiCall.map (List.map int)
>>= average
This example uses the Mathjs.org API to compute the average of a list of integers (making one API call to compute the sum, then another to divide by the number of elements), and also allows you to compute the average of the squares of a list of values, by calling the API asynchronously for each element in the list to square it, then joining the results together and computing the average. You can use these functions as follows (I added a printfn to the actual API call so it logs the HTTP requests):
Calling average:
MathWorkflow.average [1;2;3;4;5] |> Async.RunSynchronously
Outputs:
Calling API: http://api.mathjs.org/v4/?expr=1%2B2%2B3%2B4%2B5
Calling API: http://api.mathjs.org/v4/?expr=15%2F5
[<Struct>]
val it : Result<string,ApiCallError> = Ok "3"
Calling averageOfSquares:
MathWorkflow.averageOfSquares [2;4;6;8;10] |> Async.RunSynchronously
Outputs:
Calling API: http://api.mathjs.org/v4/?expr=2*2
Calling API: http://api.mathjs.org/v4/?expr=4*4
Calling API: http://api.mathjs.org/v4/?expr=6*6
Calling API: http://api.mathjs.org/v4/?expr=8*8
Calling API: http://api.mathjs.org/v4/?expr=10*10
Calling API: http://api.mathjs.org/v4/?expr=100%2B64%2B36%2B16%2B4
Calling API: http://api.mathjs.org/v4/?expr=220%2F5
[<Struct>]
val it : Result<string,ApiCallError> = Ok "44"
Ultimately, you may want to implement a custom Computation Builder to allow you to use a computation expression with the let! syntax, instead of explicitly writing the calls to ApiCall.bind everywhere. This is fairly simple, since you already do all the real work in the ApiCall module, and you just need to make a class with the appropriate Bind/Return members:
type ApiCallBuilder () =
member __.Bind (x, f) = ApiCall.bind f x
member __.Return x = ApiCall.``return`` x
member __.ReturnFrom x = x
member __.Zero () = ApiCall.``return`` ()
let apiCall = ApiCallBuilder()
With the ApiCallBuilder, you could rewrite the functions in the MathWorkflow module like this, making them a little easier to read and compose:
let average values =
apiCall {
let! sum =
values
|> List.map (sprintf "%d")
|> String.concat "+"
|> callMathApi
return!
sprintf "%s/%d" sum values.Length
|> callMathApi
}
let averageOfSquares values =
apiCall {
let! squares =
values
|> List.map (fun value -> sprintf "%d*%d" value value)
|> List.map callMathApi
|> ApiCall.join
return! squares |> List.map int |> average
}
These work as you described in the question, where each API call is made independently and the results feed into the next call, but if one call fails the computation is stopped and the error is returned. For example, if you change the URL used in the example calls here to the v3 API ("http://api.mathjs.org/v3/") without changing anything else, you get the following:
Calling API: http://api.mathjs.org/v3/?expr=2*2
[<Struct>]
val it : Result<string,ApiCallError> =
Error
(HttpError
(404,
"<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /v3/</pre>
</body>
</html>
"))
Related
I have the following code:
let exchangeInfo, marginBrackets, accountInfo =
async {
let failOrExtract = function
| Ok d -> d
| Error (e: ExchangeError) -> e.Describe |> tee logger.Fatal |> failwith
let pf = processResult >> failOrExtract
let! e = rest.FuturesUsdt.System.GetExchangeInfoAsync() |> Async.AwaitTask
let! b = rest.FuturesUsdt.GetBracketsAsync() |> Async.AwaitTask
let! i = rest.FuturesUsdt.Account.GetAccountInfoAsync() |> Async.AwaitTask
return pf e, pf b, pf i
} |> Async.RunSynchronously
In short, it calls 3 async rest functions, then sends them through failOrExtract which is made of 2 parts: the processResult part that returns a Result<'a, ExchangeError> type, with 'a different for each of these calls, and then failOrExtract that either fails or returns the data field.
this will not compile, the compiler says:
[FS0001] The type 'BinanceFuturesUsdtExchangeInfo' does not match the type 'System.Collections.Generic.IEnumerable<BinanceFuturesSymbolBracket>'```
It looks like the failOrExtract function was generic on the first use, but not afterwards since pf e compiles, but pf b and pf i have the same error.
why am I getting this error?
I think you're running into the F# "value restriction", although the compiler isn't being clear about it. The problem here is that pf is defined as a value, not a function, so the compiler is inferring that it has a concrete, non-generic type. In order to give pf a generic signature, try changing it to this instead:
let pf x = x |> processResult |> failOrExtract
I'm using quite a lot this piece of code:
let inline (||>) (a: 'a option) (b: 'a -> unit) = if a.IsSome then b a.Value
so I can do things like
myData ||> DoSomethingWithIt
without having to test if myData is Some or None since there are many functions that don't generally need to test for an option. This avoid to put the test in the function itself.
I would like to extend this to methods of a type where I could do like C#'s:
myData?.DoSomethingWithIt
essentially replacing:
if myData.IsSome then myData.Value.DoSomethingWithIt
with some syntactic sugar.
but I have no idea how I could do the operator so that it allows to get access to the type's method in the expression. Is that possible in F#?
I'm also open to learn about why it could be a bad idea if it is :)
Depending on your return type of DoSomethingWithIt the F# library offers a few standard functions for working with Options that can be turned into operators.
let x = Some 1
let aPrinter a = printfn "%i" a
let add1 a = a + 1
let (|?>) opt f = Option.iter f opt
let (|??>) opt f = Option.map f opt
x |?> aPrinter
let y = x |??> add1
You can also consider redefining your DoSomethingWithIt to work with an option by partial application.
let DoSomethingWithIt' = Option.iter DoSomethingWithIt
let something' = Option.iter (fun (b:B) -> b.DoSomethingWithIt()) //For instance methods
That may end up being a lot of work depending how many functions you are dealing with.
Ultimately you shouldn't try to hide the fact you are working with Options. By making something an Option you are telling the compiler that you aren't sure whether it exists or not. It is trying to help you by forcing you to deal with the None case. If there are lots of cases in your code where you know your Option is Some then there's probably a larger architectural issue in your code and you should try to lift all your Option<'T> to just T prior to doing work with them. e.g.:
let lift xs =
[
for x in xs do
match x with
| Some x -> yield x
| None -> ()
]
Have a look at Option.iter. It has the same signature as your operator.
There is no analogical syntax for such constructions but F# have alternatives.
The easiest way is to use FSharpx.Extras library and FSharpx.Option.maybe computation expression which will allow you to use Option related operations.
open FSharpx.Option
let a = Some 1
let b = maybe {
let! v = a
return v + 3
} // b is (Some 4)
let c : int option = None
let d = maybe {
let! v = c
return v + 3 // this line won't be reached
} // d is None
I believe that the ?. operator in c# is a syntactic sugar that hides the if statement checking for null before invoking a member of the type. Even if you could make it work the way you plan, I feel that it would go against the FP principles and could cause more problems down the line.
The Option module contains probably most of what you need already. The iter function allows to call a function on the value of the Option if that value is present (Some).
If you have situation that your input parametes can be nulls, but not options, you can use the Option.ofObj function that will convert the parameter to an Option with Some if the parameter is not null, else None.
So assuming that your function DoSomethingWithit accepts a string and returns unit:
let DoSomethingWithIt = //(string -> unit)
printf "%s; "
You can use this more verbose syntax to (for example) iterate over nullable values in your list:
let lst = [ "data"; "data 2"; null; "data3" ]
lst
|> List.iter (fun v -> v |> Option.ofObj |> Option.iter DoSomethingWithIt)
Alternatively you can compose the Optioni.ofObj and Option.iter DoSomethingWithIt functions and do something like
let SafeDoSomethingWithIt = //(string -> unit)
Option.ofObj >> Option.iter DoSomethingWithIt
This gives you safe invocation:
let lst2 = [ "data"; "data 2"; null; "data3" ]
lst2
|> List.iter SafeDoSomethingWithIt
You can generalize the combination of the functions returning unit (but not only)
let makeSafe fn =
Option.ofObj >> Option.iter fn
Then you can create a series of safe functions:
let SafeDoSomethingWithIt = makeSafe DoSomethingWithIt
let safePrint = makeSafe (printf "%s; ")
//...etc
Then this still works:
lst2
|> List.iter SafeDoSomethingWithIt
lst2
|> List.iter safePrint
You can also write a wrapper for functions returning values using Option.bind function.
let makeSafeReturn fn = //(string -> string option)
Option.ofObj >> Option.bind fn
I am using the csv type provider to collect some data from a series of files I have on Azure blob storage:
#r "../packages/FSharp.Data.2.0.9/lib/portable-net40+sl5+wp8+win8/FSharp.Data.dll"
open FSharp.Data
type censusDataContext = CsvProvider<"https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/AK.TXT">
type stateCodeContext = CsvProvider<"https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/states.csv">
let stateCodes = stateCodeContext.Load("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/states.csv");
let fetchStateData (stateCode:string)=
let uri = System.String.Format("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/{0}.TXT",stateCode)
censusDataContext.Load(uri).Rows
let usaData = stateCodes.Rows
|> Seq.collect(fun r -> fetchStateData(r.Abbreviation))
|> Seq.length
I now want to run these async and I am running into a problem with AsyncLoad:
let fetchStateDataAsync(stateCode:string)=
async{
let uri = System.String.Format("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/{0}.TXT",stateCode)
let! stateData = censusDataContext.AsyncLoad(uri)
return stateData.Rows
}
let usaData = stateCodes.Rows
|> Seq.collect(fun r -> fetchStateDataAsync(r.Abbreviation))
|> Seq.length
The error message is
The type 'Async<seq<CsvProvider<...>.Row>>' is not compatible with the type 'seq<'a>'
Forgive my lack of async knowledge, but do I have to use something other than Seq.Collect when applying async functions?
Thanks in advance
The problem is that turning code to asynchronous (by wrapping it in the async { .. } block) changes the result from seq<Row> to Async<seq<Row>> - that is, you now get an asynchronous computation that will eventually complete and return the sequence.
To fix this, you need to somehow start the computation and wait for the result. There is a number of choices - like running one by one sequentially. Probably the easiest option (and maybe the best - depending on what you want to do) is to run the computations in parallel:
let getAll =
stateCodes.Rows
|> Seq.map(fun r -> fetchStateDataAsync(r.Abbreviation))
|> Async.Parallel
This gives you an asynchronous computation that runs all the downloads and returns an array of results. You can run this synchronously (and block) and get the results:
getAll |> Async.RunSynchronously
|> Seq.collect id
|> Seq.length
If you want to run the downloads asynchronously in the background you can do that to, but you need to specify what to do with the result. For example:
async {
let! all = getAll
all |> Seq.collect id |> Seq.length |> printfn "Length %d" }
|> Async.Start
I have a function that returns a string[].
let asyncScrape url allParameters =
allParameters
|> Seq.map(fun v ->
yearAndClassResultsAsync url v)
|> Async.Parallel
|> Async.RunSynchronously
I want to iterate through that string array, sending each string to a method called resultsBody (that returns a seq), and then finally returning an single sequence that is the concatenation of the results from resultsBody.
I tried doing something like below, but I'm rather lost as it returns:
seq<string[]>[]
and I just want a single combined
seq<string[]>
My attempts so far:
let parseSite html =
Array.mapi (fun s -> resultsBody) html
Simplified, I think your problem is that you have a nested sequence of strings and you want to get as much parallelism as possible, rather than just at the innermost layer of the nesting.
One way you can do this is to also nest the Async.Parallel calls before calling Async.RunSynchronously. Here's a simple example of the technique:
let squareInt n = async { return n*n }
let inParallel (seqOfseqOfInts : seq<seq<int>>) =
seqOfseqOfInts
|> Seq.map // deal with each inner seq of ints
(fun (seqOfInts : seq<int>) ->
seqOfInts
|> Seq.map squareInt
|> Async.Parallel // this gives us Async<int[]>
) // this gives us seq<Async<int[]>>
|> Async.Parallel // this gives us Async<int[][]>
|> Async.RunSynchronously // this gives us int[][]
I can't work out why the following code is hanging at the call to GetTotal. I don't seem to be able to debug inside the MailboxProcessor, so it's hard to see what's going on.
module Aggregator
open System
type Message<'T, 'TState> =
| Aggregate of 'T
| GetTotal of AsyncReplyChannel<'TState>
type Aggregator<'T, 'TState>(initialState, f) =
let myAgent = new MailboxProcessor<Message<'T, 'TState>>(fun inbox ->
let rec loop agg =
async {
let! message = inbox.Receive()
match message with
| Aggregate x -> return! loop (f agg x)
| GetTotal replyChannel ->
replyChannel.Reply(agg)
return! loop agg
}
loop initialState
)
member m.Aggregate x = myAgent.Post(Aggregate(x))
member m.GetTotal = myAgent.PostAndReply(fun replyChannel -> GetTotal(replyChannel))
let myAggregator = new Aggregator<int, int>(0, (+))
myAggregator.Aggregate(3)
myAggregator.Aggregate(4)
myAggregator.Aggregate(5)
let totalSoFar = myAggregator.GetTotal
printfn "%d" totalSoFar
Console.ReadLine() |> ignore
It seems to work fine when using an identical MailboxProcessor directly, rather than wrapping in the Aggregator class.
The problem is that you did not start the agent. You can either call Start after you create the agent:
let myAgent = (...)
do myAgent.Start()
Alternatively, you can create the agent using MailboxProcessor<'T>.Start instead of calling the constructor (I usually prefer this option, because it looks more functional):
let myAgent = MailboxProcessor<Message<'T, 'TState>>.Start(fun inbox -> (...) )
I suppose that you couldn't debug the agent, because the code inside agent wasn't actually running. I tried adding printfn "Msg: %A" message right after the call to Receive inside the agent (to print incoming messages for debugging) and I noticed that, after calling Aggregate, no messages were actually received by the agent... (It only blocked after calling GetTotal, which avaits reply)
As a side-note, I would probably turn GetTotal into a method, so you'd call GetTotal(). Properties are re-evaluated each time you access them, so your code does the same thing, but best practices don't recommend using properties that do complex work.
You forgot to start the mailbox:
open System
type Message<'T, 'TState> =
| Aggregate of 'T
| GetTotal of AsyncReplyChannel<'TState>
type Aggregator<'T, 'TState>(initialState, f) =
let myAgent = new MailboxProcessor<Message<'T, 'TState>>(fun inbox ->
let rec loop agg =
async {
let! message = inbox.Receive()
match message with
| Aggregate x -> return! loop (f agg x)
| GetTotal replyChannel ->
replyChannel.Reply(agg)
return! loop agg
}
loop initialState
)
member m.Aggregate x = myAgent.Post(Aggregate(x))
member m.GetTotal = myAgent.PostAndReply(fun replyChannel -> GetTotal(replyChannel))
member m.Start() = myAgent.Start()
let myAggregator = new Aggregator<int, int>(0, (+))
myAggregator.Start()
myAggregator.Aggregate(3)
myAggregator.Aggregate(4)
myAggregator.Aggregate(5)
let totalSoFar = myAggregator.GetTotal
printfn "%d" totalSoFar
Console.ReadLine() |> ignore