I don't yet understand why my addLink function does not get invoked.
I have the following code:
let links = source |> getLinks
let linkIds = links |> Seq.map addLink // addLink never gets executed
At first, I thought that the links value was empty.
However, it's not. I verified that it was populated by calling the following:
let count = Seq.length links // Returns over 100 items
Note:
The only way I have been able to execute the function is by first executing:
let count = linkIds |> Seq.length // Call this after performing map
Why do I need to do that just for my function to be called?
Appendix:
let addLink (info:Link) =
let commandFunc (command: SqlCommand) =
command |> addWithValue "#ProfileId" info.ProfileId
|> addWithValue "#Title" info.Title
|> addWithValue "#Description" info.Description
|> addWithValue "#Url" info.Url
|> addWithValue "#ContentTypeId" (info.ContentType |> contentTypeToId)
|> addWithValue "#IsFeatured" info.IsFeatured
|> addWithValue "#Created" DateTime.Now
commandFunc |> execute connectionString addLinkSql
[<CLIMutable>]
type Link = {
Id: int
ProfileId: string
Title: String
Description: String
Url: string
Topics: Topic list
ContentType: string
IsFeatured: bool
}
Here's the source code.
Seq<'a> is the same as IEnumerable<'a>, which is lazy. This means that each element in the sequence is only evaluated as it's needed. If you want to ensure that side-effects are called, then you could could convert this to a concrete data structure like list by adding Seq.toList.
However, it's better and usually possible to write code that performs side-effects separately to code that returns values. You could calculate the linkIds with one function and perform the side effects with another function using Seq.iter, which forces full evaluation of the sequence.
Related
I have some code, 3 functions
ParseTemplate, ParseTemplates -> These two can be combined into one and will have to be to make this work I think.
And
loadTemplate
let ParseTemplate (template: Match) =
let templateName = template.Groups.[1] |> string
loadTemplate templateName
let ParseTemplates (string: string) =
Regex.Replace(string, "\[tpl\:(.*?)\]", MatchEvaluator ParseTemplate)
let rec loadTemplate templateName =
templateName
|> getTemplateFilePath
|> File.ReadAllText
|> ParseVariables
|> ParseArrays
|> ParseLanguageVariablesWithReplacements
|> ParseSimpleLanguageVariables
|> ParseTemplates
The problem is where to position these in the file/how to structure them differently, because as it is there's always a function calling a function below it (and therefore is not defined).
In this case
I like the loadTemplate function as it exists - it's very clean and readable and I'd rather avoid using lambda functions inside of it if possible. The problem is it calls ParseTemplates which can then call loadTemplate recursively.
Best way to approach?
I would like to create a chain of expressions and any of them can fail when the computation should just stop.
With Unix pipes it is usually like this:
bash-3.2$ echo && { echo 'a ok'; echo; } && { echo 'b ok'; echo; }
a ok
b ok
When something fails the pipeline stops:
echo && { echo 'a ok'; false; } && { echo 'b ok'; echo; }
a ok
I can handle Optionals but my problem is that I might want to do multiple things in each branch:
let someExternalOperation = callToAnAPI()
match someExternalOperation with
| None -> LogAndStop()
| Some x -> LogAndContinue()
Then I would like to keep going with other API calls and only stop if there is an error.
Is there something like that in F#?
Update1:
What I am trying to do is calling out to external APIs. Each call can fail. Would be nice to try to retry but not required.
You can use the F# Async and Result types together to represent the results of each API Call. You can then use the bind functions for those types to build a workflow in which you only continue processing when the previous calls were successful. In order to make that easier, you can wrap the Async<Result<_,_>> you would be working with for each api call in its own type and build a module around binding those results to orchestrate a chained computation. Here's a quick example of what that would look like:
First, we would lay out the type ApiCallResult to wrap Async and Result, and we would define ApiCallError to represent HTTP error responses or exceptions:
open System
open System.Net
open System.Net.Http
type ApiCallError =
| HttpError of (int * string)
| UnexpectedError of exn
type ApiCallResult<'a> = Async<Result<'a, ApiCallError>>
Next, we would create a module to work with ApiCallResult instances, allowing us to do things like bind, map, and return so that we can process the results of a computation and feed them into the next one.
module ApiCall =
let ``return`` x : ApiCallResult<_> =
async { return Ok x }
let private zero () : ApiCallResult<_> =
``return`` []
let bind<'a, 'b> (f: 'a -> ApiCallResult<'b>) (x: ApiCallResult<'a>) : ApiCallResult<'b> =
async {
let! result = x
match result with
| Ok value ->
return! f value
| Error error ->
return Error error
}
let map f x = x |> bind (f >> ``return``)
let combine<'a> (acc: ApiCallResult<'a list>) (cur: ApiCallResult<'a>) =
acc |> bind (fun values -> cur |> map (fun value -> value :: values))
let join results =
results |> Seq.fold (combine) (zero ())
Then, you would have a module to simply do your API calls, however that works in your real scenario. Here's one that just handles GETs with query parameters, but you could make this more sophisticated:
module Api =
let call (baseUrl: Uri) (queryString: string) : ApiCallResult<string> =
async {
try
use client = new HttpClient()
let url =
let builder = UriBuilder(baseUrl)
builder.Query <- queryString
builder.Uri
printfn "Calling API: %O" url
let! response = client.GetAsync(url) |> Async.AwaitTask
let! content = response.Content.ReadAsStringAsync() |> Async.AwaitTask
if response.IsSuccessStatusCode then
let! content = response.Content.ReadAsStringAsync() |> Async.AwaitTask
return Ok content
else
return Error <| HttpError (response.StatusCode |> int, content)
with ex ->
return Error <| UnexpectedError ex
}
let getQueryParam name value =
value |> WebUtility.UrlEncode |> sprintf "%s=%s" name
Finally, you would have your actual business workflow logic, where you call multiple APIs and feed the results of one into another. In the below example, anywhere you see callMathApi, it is making a call to an external REST API that may fail, and by using the ApiCall module to bind the results of the API call, it only proceeds to the next API call if the previous call was successful. You can declare an operator like >>= to eliminate some of the noise in the code when binding computations together:
module MathWorkflow =
let private (>>=) x f = ApiCall.bind f x
let private apiUrl = Uri "http://api.mathjs.org/v4/" // REST API for mathematical expressions
let private callMathApi expression =
expression |> Api.getQueryParam "expr" |> Api.call apiUrl
let average values =
values
|> List.map (sprintf "%d")
|> String.concat "+"
|> callMathApi
>>= fun sum ->
sprintf "%s/%d" sum values.Length
|> callMathApi
let averageOfSquares values =
values
|> List.map (fun value -> sprintf "%d*%d" value value)
|> List.map callMathApi
|> ApiCall.join
|> ApiCall.map (List.map int)
>>= average
This example uses the Mathjs.org API to compute the average of a list of integers (making one API call to compute the sum, then another to divide by the number of elements), and also allows you to compute the average of the squares of a list of values, by calling the API asynchronously for each element in the list to square it, then joining the results together and computing the average. You can use these functions as follows (I added a printfn to the actual API call so it logs the HTTP requests):
Calling average:
MathWorkflow.average [1;2;3;4;5] |> Async.RunSynchronously
Outputs:
Calling API: http://api.mathjs.org/v4/?expr=1%2B2%2B3%2B4%2B5
Calling API: http://api.mathjs.org/v4/?expr=15%2F5
[<Struct>]
val it : Result<string,ApiCallError> = Ok "3"
Calling averageOfSquares:
MathWorkflow.averageOfSquares [2;4;6;8;10] |> Async.RunSynchronously
Outputs:
Calling API: http://api.mathjs.org/v4/?expr=2*2
Calling API: http://api.mathjs.org/v4/?expr=4*4
Calling API: http://api.mathjs.org/v4/?expr=6*6
Calling API: http://api.mathjs.org/v4/?expr=8*8
Calling API: http://api.mathjs.org/v4/?expr=10*10
Calling API: http://api.mathjs.org/v4/?expr=100%2B64%2B36%2B16%2B4
Calling API: http://api.mathjs.org/v4/?expr=220%2F5
[<Struct>]
val it : Result<string,ApiCallError> = Ok "44"
Ultimately, you may want to implement a custom Computation Builder to allow you to use a computation expression with the let! syntax, instead of explicitly writing the calls to ApiCall.bind everywhere. This is fairly simple, since you already do all the real work in the ApiCall module, and you just need to make a class with the appropriate Bind/Return members:
type ApiCallBuilder () =
member __.Bind (x, f) = ApiCall.bind f x
member __.Return x = ApiCall.``return`` x
member __.ReturnFrom x = x
member __.Zero () = ApiCall.``return`` ()
let apiCall = ApiCallBuilder()
With the ApiCallBuilder, you could rewrite the functions in the MathWorkflow module like this, making them a little easier to read and compose:
let average values =
apiCall {
let! sum =
values
|> List.map (sprintf "%d")
|> String.concat "+"
|> callMathApi
return!
sprintf "%s/%d" sum values.Length
|> callMathApi
}
let averageOfSquares values =
apiCall {
let! squares =
values
|> List.map (fun value -> sprintf "%d*%d" value value)
|> List.map callMathApi
|> ApiCall.join
return! squares |> List.map int |> average
}
These work as you described in the question, where each API call is made independently and the results feed into the next call, but if one call fails the computation is stopped and the error is returned. For example, if you change the URL used in the example calls here to the v3 API ("http://api.mathjs.org/v3/") without changing anything else, you get the following:
Calling API: http://api.mathjs.org/v3/?expr=2*2
[<Struct>]
val it : Result<string,ApiCallError> =
Error
(HttpError
(404,
"<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /v3/</pre>
</body>
</html>
"))
I am using the csv type provider to collect some data from a series of files I have on Azure blob storage:
#r "../packages/FSharp.Data.2.0.9/lib/portable-net40+sl5+wp8+win8/FSharp.Data.dll"
open FSharp.Data
type censusDataContext = CsvProvider<"https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/AK.TXT">
type stateCodeContext = CsvProvider<"https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/states.csv">
let stateCodes = stateCodeContext.Load("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/states.csv");
let fetchStateData (stateCode:string)=
let uri = System.String.Format("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/{0}.TXT",stateCode)
censusDataContext.Load(uri).Rows
let usaData = stateCodes.Rows
|> Seq.collect(fun r -> fetchStateData(r.Abbreviation))
|> Seq.length
I now want to run these async and I am running into a problem with AsyncLoad:
let fetchStateDataAsync(stateCode:string)=
async{
let uri = System.String.Format("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/{0}.TXT",stateCode)
let! stateData = censusDataContext.AsyncLoad(uri)
return stateData.Rows
}
let usaData = stateCodes.Rows
|> Seq.collect(fun r -> fetchStateDataAsync(r.Abbreviation))
|> Seq.length
The error message is
The type 'Async<seq<CsvProvider<...>.Row>>' is not compatible with the type 'seq<'a>'
Forgive my lack of async knowledge, but do I have to use something other than Seq.Collect when applying async functions?
Thanks in advance
The problem is that turning code to asynchronous (by wrapping it in the async { .. } block) changes the result from seq<Row> to Async<seq<Row>> - that is, you now get an asynchronous computation that will eventually complete and return the sequence.
To fix this, you need to somehow start the computation and wait for the result. There is a number of choices - like running one by one sequentially. Probably the easiest option (and maybe the best - depending on what you want to do) is to run the computations in parallel:
let getAll =
stateCodes.Rows
|> Seq.map(fun r -> fetchStateDataAsync(r.Abbreviation))
|> Async.Parallel
This gives you an asynchronous computation that runs all the downloads and returns an array of results. You can run this synchronously (and block) and get the results:
getAll |> Async.RunSynchronously
|> Seq.collect id
|> Seq.length
If you want to run the downloads asynchronously in the background you can do that to, but you need to specify what to do with the result. For example:
async {
let! all = getAll
all |> Seq.collect id |> Seq.length |> printfn "Length %d" }
|> Async.Start
I have a function that returns a string[].
let asyncScrape url allParameters =
allParameters
|> Seq.map(fun v ->
yearAndClassResultsAsync url v)
|> Async.Parallel
|> Async.RunSynchronously
I want to iterate through that string array, sending each string to a method called resultsBody (that returns a seq), and then finally returning an single sequence that is the concatenation of the results from resultsBody.
I tried doing something like below, but I'm rather lost as it returns:
seq<string[]>[]
and I just want a single combined
seq<string[]>
My attempts so far:
let parseSite html =
Array.mapi (fun s -> resultsBody) html
Simplified, I think your problem is that you have a nested sequence of strings and you want to get as much parallelism as possible, rather than just at the innermost layer of the nesting.
One way you can do this is to also nest the Async.Parallel calls before calling Async.RunSynchronously. Here's a simple example of the technique:
let squareInt n = async { return n*n }
let inParallel (seqOfseqOfInts : seq<seq<int>>) =
seqOfseqOfInts
|> Seq.map // deal with each inner seq of ints
(fun (seqOfInts : seq<int>) ->
seqOfInts
|> Seq.map squareInt
|> Async.Parallel // this gives us Async<int[]>
) // this gives us seq<Async<int[]>>
|> Async.Parallel // this gives us Async<int[][]>
|> Async.RunSynchronously // this gives us int[][]
I'm trying to write an F# type that wraps an underlying collection of Map<int64, customType> such that I can add items to it:
type public CandleContainer (seedCandles:System.Collections.Generic.IEnumerable<Candle>) =
let candles : Map<int64, Candle> =
seedCandles
|> Seq.map (fun x -> x.Start.Ticks, x)
|> Map.ofSeq
let candleInterval = TimeSpan.FromMinutes(1.0)
member public x.AddCandle (candle:Candle) =
candles = candles.Add(candle.Start.Ticks, candle)
member public x.GetList () : List<Candle> =
candles
|> Map.toSeq
|> Seq.map (fun (key, value) -> value)
|> Seq.toList
The problem that I'm getting is in the AddCandle method. I understand that Map.Add returns a new map with the value added and the compiler isn't complaining that I'm trying to overwrite the value of candles. However, whenever I try adding a new Candle using this method x.GetList returns an empty list. Is there a way to overwrite the old value of candles with the result of the Map.Add? Or do I just make candles a mutable value and overwrite it?
What you're looking for is the mutable keyword and <- operator.
Change let candles ... to be
let mutable candles : Map<int64, Candle>
and candles = ... to be
candles <- candles.Add(candle.Start.Ticks, candle)
. Also note that your previous use of = where you thought replacement was occurring was actually performing an equality check, making the return type of that method bool.