Add calculated key to collection - f#

Please consider this dataset, composed by man and woman, and that I filter in a second moment according to few variables:
type ls = JsonProvider<"...">
let dt = ls.GetSamples()
let dt2 =
dt |> Seq.filter (fun c -> c.Sex = "male" && c.Height > Some 150)
dt2
[{"sex":"male","height":180,"weight":85},
{"sex":"male","height":160" "weight":60},
{"sex":"male","height":180,"weight":85}]
Lets suppose that I would like to add a fourth key "body mass index" or "bmi", and that its value is roughly given by "weight"/"height". Hence I expect:
[{"sex":"male","height":180,"weight":85, "bmi":(180/85)},
{"sex":"male","height":160" "weight":60, "bmi":(160/60},
{"sex":"male","height":180,"weight":85, "bmi":(180/85)}]
I thought that map.Add may help.
let dt3 = dt2.Add("bmi", (dt2.Height/dt2.Weight))
Unfortunately, it returns an error:
error FS0039: The field, constructor or member 'Add' is not defined
I am sure there are further errors in my code, but without this function I cannot actually look for them. Am I, at least, approaching the problem correctly?

Creating modified versions of the JSON is sadly one thing that the F# Data type provider does not make particularly easy. What makes that hard is the fact that we can infer the type from the source JSON, but we cannot "predict" what kind of fields people might want to add.
To do this, you'll need to access the underlying representation of the JSON value and operate on that. For example:
type ls = JsonProvider<"""
[{"sex":"male","height":180,"weight":85},
{"sex":"male","height":160,"weight":60},
{"sex":"male","height":180,"weight":85}]""">
let dt = ls.GetSamples()
let newJson =
dt
|> Array.map (fun recd ->
// To do the calculation, you can access the fields via inferred types
let bmi = float recd.Height / float recd.Weight
// But now we need to look at the underlying value, check that it is
// a record and extract the properties, which is an array of key-value pairs
match recd.JsonValue with
| JsonValue.Record props ->
// Append the new property to the existing properties & re-create record
Array.append [| "bmi", JsonValue.Float bmi |] props
|> JsonValue.Record
| _ -> failwith "Unexpected format" )
// Re-create a new JSON array and format it as JSON
JsonValue.Array(newJson).ToString()

Related

F# CsvTypeProvider - mapping function from inferred types

I am using the CsvTypeProvider to map data from CSV files into my own data structures. This works splendidly, except for that I have to repeat the mapping function every time:
type GamesFile = CsvProvider<"./data/15.csv">
let games15 = GamesFile.Load("./data/15.csv").Rows |> Seq.map ( fun c -> { Division = c.Div; Date = c.Date; HomeScore = c.HomeScore; AwayScore = c.AwayScore })
let games16 = GamesFile.Load("./data/16.csv").Rows |> Seq.map ( fun c -> { Division = c.Div; Date = c.Date; HomeScore = c.HomeScore; AwayScore = c.AwayScore })
When I try moving it to a function, I am told that "Lookup on object of indeterminate type based on information prior to this program point. A type annotation may be needed prior to this program point to constrain the type of the object. This may allow the lookup to be resolved."
This makes sense, but how can I tell a mapping function what type it is when the type is inferred from the contents of the CSV? How is this normally solved?
The type provider generates a type representing the row and it exposes it as a nested type of the main provided type - in your case aliased as GamesFile.
This is not entirely obvious, because the editors will show tooltip with something like
CsvFile<...>.Row so it does not show the name of the alias, but it suggests Row is a nested type. To use the type in your code, you can just write GamesFile.Row, so you need something like this:
type GamesFile = CsvProvider<"./data/15.csv">
let mapRows (rows:seq<GamesFile.Row>) =
rows |> Seq.map (fun c ->
{ Division = c.Div; Date = c.Date; HomeScore = c.HomeScore; AwayScore = c.AwayScore })
let games15 = GamesFile.Load("./data/15.csv").Rows |> mapRows
let games16 = GamesFile.Load("./data/16.csv").Rows |> mapRows

How to define a proper data structure for a parameter set

I need a data structure for the following:
In a device that has memory slots, each of the slots has a set of parameters. These parameters have different types. The list of possible parameters is fixed, so there is no need for generic flexibility à la »Support of arbitrary parameters without change«. Also, for each parameter, the structure of the contents is known. Typical use cases are the retrieval and modification of one specific parameter as well as a transformation of the complete parameter set into a different (but already defined) data structure.
The natural choice of F# data structure would be a sum type like this:
type SomeParameterContentType = { Field1 : string, Field2 : int }
type SomeOtherParameterContentType = { Other1 : bool option, Other2 : double }
type Parameter =
| SomeParameter of SomeParameterContentType
| SomeOtherParameter of SomeOtherParameterContentType
This way I could create a set and store the parameters there with a very nice data structure. The question here is: Given this idea, how would looking for a specific parameter look like? I don't know of any way to specify a predicate for a find-function for sets. It would be possible to define another sum type listing just the Parameter Types without their contents using this as key for a Dictionary but I don't like this idea too much. Using strings instead of the second sum type doesn't make things better as it still would require providing the list of possible parameters twice.
Does anyone have a better idea?
Thx
--Mathias.
Sounds like all you want is a tryFind for a Set:
module Set =
let tryFind p =
Set.toList >> List.tryFind p
Usage:
let mySet = Set.ofList [1;2;3;4;5]
let m = mySet |> Set.tryFind (fun t -> t = 2)
val m : int option = Some 2
Usage with your Types:
let yourSet = Set.ofList [SomeParameter {Field1="hello";Field2=3}]
let mYours = yourSet |> Set.tryFind (fun t -> match t with
|SomeParameter p -> true
|SomeOtherParameter p -> false)
val mYours : Parameter option = Some (SomeParameter {Field1 = "hello";
Field2 = 3;})

Type annotation for using a F# TypeProvider type e.g. FSharp.Data.JsonProvider<...>.DomainTypes.Url

I'm using the FSharp.Data.JsonProvider to read Twitter Tweets.
Playing with this sample code
https://github.com/tpetricek/Documents/tree/master/Samples/Twitter.API
I want to expand the urls in the tweet with
let expandUrl (txt:string) (url:Search.DomainTypes<...>.DomainTypes.Url) =
txt.Replace( url.Url, url.ExpandedUrl )
This results in Error:
Lookup on object of indeterminate type based on information prior to this program point.
A type annotation may be needed prior to this program point to constrain the type of the object.
My problem is how to define the TypeProvider Type for url in the expandUrl function above?
The type inferance shows me this
val urls : FSharp.Data.JsonProvider<...>.DomainTypes.Url []
but this is not accepted in the type declaration. I assume "<...>" is not F# synatx.
How to do a type annotation for using a TypeProvider type e.g. FSharp.Data.JsonProvider<...>.DomainTypes.Url ?
Here is the complete code snippet:
open TwitterAPI // github.com/tpetricek/Documents/tree/master/Samples/Twitter.API
let twitter = TwitterAPI.TwitterContext( _consumerKey, _consumerSecret, _accessToken, _accessTokenSecret )
let query = "water"
let ts = Twitter.Search.Tweets(twitter, Utils.urlEncode query, count=100)
let ret =
[ for x in ts.Statuses do
// val urls : FSharp.Data.JsonProvider<...>.DomainTypes.Url []
let urls = x.Entities.Urls
// fully declarated to help the type inference at expandUrl
let replace (txt:string) (oldValue:string) (newValue:string) =
txt.Replace( oldValue, newValue)
// Error:
// Lookup on object of indeterminate type based on information prior to this program point.
// A type annotation may be needed prior to this program point to constrain the type of the object.
// This may allow the lookup to be resolved.
let expandUrl (txt:string) (url:FSharp.Data.JsonProvider<_>.DomainTypes.Url) =
replace txt url.Url url.ExpandedUrl
let textWithExpandedUrls = Array.fold expandUrl x.Text urls
yield textWithExpandedUrls
]
When you call Twitter.Search.Tweets (https://github.com/tpetricek/Documents/blob/master/Samples/Twitter.API/Twitter.fs#L284), the return type of that is one of the domain types of TwitterTypes.SearchTweets, which is a type alias for JsonProvider<"references\\search_tweets.json"> (https://github.com/tpetricek/Documents/blob/master/Samples/Twitter.API/Twitter.fs#L183).
Although in the tooltip it shows up as JsonProvider<...>.DomainTypes.Url, you'll have to use the type alias TwitterTypes.SearchTweets.DomainTypes.Url
I had a similar problem trying to figure out how to use the FSharp.Data HtmlProvider.
I am using Wikipedia to get information about USA presidents. The HtmlProvider does a great job of discovering the various tables in that webpage, but I wanted to extract the logic for processing a row of "president data" into a separate function called processRow.
And the problem was trying to work out what the type of such a row is for processRow's parameter row. The following code does the trick:
#load "Scripts\load-references.fsx"
open FSharp.Data
let presidents = new HtmlProvider<"https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States">()
let ps = presidents.Tables.``List of presidents``
ps.Headers |> Option.map (fun hs -> for h in hs do printf "%s " h)
printfn ""
type Presidents = ``HtmlProvider,Sample="https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States"``.ListOfPresidents
let processRow (row:Presidents.Row) =
printfn "%d %s" row.``№`` row.President2
ps.Rows |> Seq.iter processRow
I did not type in the long type alias for Presidents, I used Visual Studio auto-completion by guessing that the type for List of presidents would be discoverable from something starting with Html, and it was, complete with the four single back quotes.

Verifying a set of objects have been mapped correctly

I'm looking for a clean set of ways to manage Test Specific Equality in F# unit tests. 90% of the time, the standard Structural Equality fits the bill and I can leverage it with unquote to express the relation between my result and my expected.
TL;DR "I can't find a clean way to having a custom Equality function for one or two properties in a value which 90% of is well served by Structural Equality, does F# have a way to match an arbitrary record with custom Equality for just one or two of its fields?"
Example of a general technique that works for me
When verifying a function that performs a 1:1 mapping of a datatype to another, I'll often extract matching tuples from both sides of in some cases and compare the input and output sets. For example, I have an operator:-
let (====) x y = (x |> Set.ofSeq) = (y |> Set.ofSeq)
So I can do:
let inputs = ["KeyA",DateTime.Today; "KeyB",DateTime.Today.AddDays(1); "KeyC",DateTime.Today.AddDays(2)]
let trivialFun (a:string,b) = a.ToLower(),b
let expected = inputs |> Seq.map trivialFun
let result = inputs |> MyMagicMapper
test <# expected ==== actual #>
This enables me to Assert that each of my inputs has been mapped to an output, without any superfluous outputs.
The problem
The problem is when I want to have a custom comparison for one or two of the fields.
For example, if my DateTime is being passed through a slightly lossy serialization layer by the SUT, I need a test-specific tolerant DateTime comparison. Or maybe I want to do a case-insensitive verification for a string field
Normally, I'd use Mark Seemann's SemanticComparison library's Likeness<Source,Destination> to define a Test Specific equality, but I've run into some roadblocks:
tuples: F# hides .ItemX on Tuple so I can't define the property via a .With strongly typed field name Expression<T>
record types: TTBOMK these are sealed by F# with no opt-out so SemanticComparison can't proxy them to override Object.Equals
My ideas
All I can think of is to create a generic Resemblance proxy type that I can include in a tuple or record.
Or maybe using pattern matching (Is there a way I can use that to generate an IEqualityComparer and then do a set comparison using that?)
Alternate failing test
I'm also open to using some other function to verify the full mapping (i.e. not abusing F# Set or involving too much third party code. i.e. something to make this pass:
let sut (a:string,b:DateTime) = a.ToLower(),b + TimeSpan.FromTicks(1L)
let inputs = ["KeyA",DateTime.Today; "KeyB",DateTime.Today.AddDays(1.0); "KeyC",DateTime.Today.AddDays(2.0)]
let toResemblance (a,b) = TODO generate Resemblance which will case insensitively compare fst and tolerantly compare snd
let expected = inputs |> List.map toResemblance
let result = inputs |> List.map sut
test <# expected = result #>
Firstly, thanks to all for the inputs. I was largely unaware of SemanticComparer<'T> and it definitely provides a good set of building blocks for building generalized facilities in this space. Nikos' post gives excellent food for thought in the area too. I shouldn't have been surprised Fil exists too - #ptrelford really does have a lib for everything (the FSharpValue point is also v valuable)!
We've thankfully arrived at a conclusion to this. Unfortunately it's not a single all-encompassing tool or technique, but even better, a set of techniques that can be used as necessary in a given context.
Firstly, the issue of ensuring a mapping is complete is really an orthogonal concern. The question refers to an ==== operator:-
let (====) x y = (x |> Set.ofSeq) = (y |> Set.ofSeq)
This is definitely the best default approach - lean on Structural Equality. One thing to note is that, being reliant on F# persistent sets, it requires your type to support : comparison (as opposed to just : equality).
When doing set comparisons off the proven Structural Equality path, a useful technique is to use HashSet<T> with a custom IEqualityComparer:-
[<AutoOpen>]
module UnorderedSeqComparisons =
let seqSetEquals ec x y =
HashSet<_>( x, ec).SetEquals( y)
let (==|==) x y equals =
let funEqualityComparer = {
new IEqualityComparer<_> with
member this.GetHashCode(obj) = 0
member this.Equals(x,y) =
equals x y }
seqSetEquals funEqualityComparer x y
the equals parameter of ==|== is 'a -> 'a -> bool which allows one to use pattern matching to destructure args for the purposes of comparison. This works well if either the input or the result side are naturally already tuples. Example:
sut.Store( inputs)
let results = sut.Read()
let expecteds = seq { for x in inputs -> x.Name,x.ValidUntil }
test <# expecteds ==|== results
<| fun (xN,xD) (yN,yD) ->
xF=yF
&& xD |> equalsWithinASecond <| yD #>
While SemanticComparer<'T> can do a job, it's simply not worth bothering for tuples with when you have the power of pattern matching. e.g. Using SemanticComparer<'T>, the above test can be expressed as:
test <# expecteds ==~== results
<| [ funNamedMemberComparer "Item2" equalsWithinASecond ] #>
using the helper:
[<AutoOpen>]
module MemberComparerHelpers =
let funNamedMemberComparer<'T> name equals = {
new IMemberComparer with
member this.IsSatisfiedBy(request: PropertyInfo) =
request.PropertyType = typedefof<'T>
&& request.Name = name
member this.IsSatisfiedBy(request: FieldInfo) =
request.FieldType = typedefof<'T>
&& request.Name = name
member this.GetHashCode(obj) = 0
member this.Equals(x, y) =
equals (x :?> 'T) (y :?> 'T) }
let valueObjectMemberComparer() = {
new IMemberComparer with
member this.IsSatisfiedBy(request: PropertyInfo) = true
member this.IsSatisfiedBy(request: FieldInfo) = true
member this.GetHashCode(obj) = hash obj
member this.Equals(x, y) =
x.Equals( y) }
let (==~==) x y mcs =
let ec = SemanticComparer<'T>( seq {
yield valueObjectMemberComparer()
yield! mcs } )
seqSetEquals ec x y
All of the above is best understood by reading Nikos Baxevanis' post NOW!
For types or records, the ==|== technique can work (except critically you lose Likeness<'T>s verifying coverage of fields). However the succinctness can make it a valuable tool for certain sorts of tests :-
sut.Save( inputs)
let expected = inputs |> Seq.map (fun x -> Mapped( base + x.ttl, x.Name))
let likeExpected x = expected ==|== x <| (fun x y -> x.Name = y.Name && x.ValidUntil = y.ValidUntil)
verify <# repo.Store( is( likeExpected)) #> once

error with f# generic follow Expert Fsharp book example

I'm reading Expert F# book and I found this code
open System.Collections.Generic
let divideIntoEquivalenceClasses keyf seq =
// The dictionary to hold the equivalence classes
let dict = new Dictionary<'key,ResizeArray<'T>>()
// Build the groupings
seq |> Seq.iter (fun v ->
let key = keyf v
let ok,prev = dict.TryGetValue(key)
if ok then prev.Add(v)
else let prev = new ResizeArray<'T>()
dict.[key] <- prev
prev.Add(v))
dict |> Seq.map (fun group -> group.Key, Seq.readonly group.Value)
and the example use:
> divideIntoEquivalenceClasses (fun n -> n % 3) [ 0 .. 10 ];;
val it : seq<int * seq<int>>
= seq [(0, seq [0; 3; 6; 9]); (1, seq [1; 4; 7; 10]); (2, seq [2; 5; 8])]
first for me this code is really ugly, even if this is safe, It looks more similar to imperative languages than to functional lang..specially compared to clojure. But the problem is not this...I'm having problems with the Dictionary definition
when I type this:
let dict = new Dictionary<'key,ResizeArray<'T>>();;
I get this:
pruebafs2a.fs(32,5): error FS0030: Value restriction. The value 'dict' has been inferred to have generic type
val dict : Dictionary<'_key,ResizeArray<'_T>> when '_key : equality
Either define 'dict' as a simple data term, make it a function with explicit arguments or, if you do not intend for it to be generic, add a type annotation.
is It ok?...
thanks so much
improve question:
Ok I've been reading about value restriction and I found this helpfull information
In particular, only function definitions and simple immutable data
expressions are automatically generalized
...ok..this explains why
let dict = new Dictionary<'key,ResizeArray<'T>>();;
doesn't work...and show 4 different techniques, although in my opinion they only resolve the error but aren't solutions for use generic code:
Technique 1: Constrain Values to Be Nongeneric
let empties : int list [] = Array.create 100 []
Technique 3: Add Dummy Arguments to Generic Functions When Necessary
let empties () = Array.create 100 []
let intEmpties : int list [] = empties()
Technique 4: Add Explicit Type Arguments When Necessary (similar to tec 3)
let emptyLists = Seq.init 100 (fun _ -> [])
> emptyLists<int>;;
val it : seq<int list> = seq [[]; []; []; []; ...]
----- and the only one than let me use real generic code ------
Technique 2: Ensure Generic Functions Have Explicit Arguments
let mapFirst = List.map fst //doesn't work
let mapFirst inp = List.map fst inp
Ok, in 3 of 4 techniques I need resolve the generic code before can work with this...now...returning to book example...when the compile knows the value for 'key and 'T
let dict = new Dictionary<'key,ResizeArray<'T>>()
in the scope the code is very generic for let key be any type, the same happen with 'T
and the biggest dummy question is :
when I enclose the code in a function (technique 3):
let empties = Array.create 100 [] //doesn't work
let empties () = Array.create 100 []
val empties : unit -> 'a list []
I need define the type before begin use it
let intEmpties : int list [] = empties()
for me (admittedly I'm a little dummy with static type languages) this is not real generic because it can't infer the type when I use it, I need define the type and then pass values (not define its type based in the passed values) exist other way define type without be so explicit..
thanks so much..really appreciate any help
This line
let dict = new Dictionary<'key,ResizeArray<'T>>();;
fails because when you type the ;; the compiler doesn't know what 'key and 'T are. As the error message states you need to add a type annotation, or allow the compiler to infer the type by using it later or make it a function
Examples
Type annotation change
let dict = new Dictionary<int,ResizeArray<int>>();;
Using types later
let dict = new Dictionary<'key,ResizeArray<'T>>()
dict.[1] <- 2
using a function
let dict() = new Dictionary<'key,ResizeArray<'T>>();;
This actually doesn't cause an issue when it's defined all together. That is, select the entire block that you posted and send it to FSI in one go. I get this:
val divideIntoEquivalenceClasses :
('T -> 'key) -> seq<'T> -> seq<'key * seq<'T>> when 'key : equality
However, if you type these individually into FSI then as John Palmer says there is not enough information in that isolated line for the interpreter to determine the type constraints. John's suggestions will work, but the original code is doing it correctly - defining the variable and using it in the same scope so that the types can be inferred.
for me this code is really ugly, even if this is safe, It looks more similar to imperative languages than to functional lang.
I agree completely – it's slightly tangential to your direct question, but I think a more idiomatic (functional) approach would be:
let divideIntoEquivalenceClasses keyf seq =
(System.Collections.Generic.Dictionary(), seq)
||> Seq.fold (fun dict v ->
let key = keyf v
match dict.TryGetValue key with
| false, _ -> dict.Add (key, ResizeArray(Seq.singleton v))
| _, prev -> prev.Add v
dict)
|> Seq.map (function KeyValue (k, v) -> k, Seq.readonly v)
This allows sufficient type inference to obviate the need for your question in the first place.
The workarounds proposed by the other answers are all good. Just to clarify based on your latest updates, let's consider two blocks of code:
let empties = Array.create 100 []
as opposed to:
let empties = Array.create 100 []
empties.[0] <- [1]
In the second case, the compiler can infer that empties : int list [], because we are inserting an int list into the array in the second line, which constrains the element type.
It sounds like you'd like the compiler to infer a generic value empties : 'a list [] in the first case, but this would be unsound. Consider what would happen if the compiler did that and we then entered the following two lines in another batch:
empties.[0] <- [1] // treat 'a list [] as int list []
List.iter (printfn "%s") empties.[0] // treat 'a list [] as string list []
Each of these lines unifies the generic type parameter 'a with a different concrete type (int and string). Either of these unifications is fine in isolation, but they are incompatible with each other and would result in treating the int value 1 inserted by the first line as a string when the second line is executed, which is clearly a violation of type safety.
Contrast this with an empty list, which really is generic:
let empty = []
Then in this case, the compiler does infer empty : 'a list, because it's safe to treat empty as a list of different types in different locations in your code without ever impacting type safety:
let l1 : int list = empty
let l2 : string list = empty
let l3 = 'a' :: empty
In the case where you make empties the return value of a generic function:
let empties() = Array.create 100 []
it is again safe to infer a generic type, since if we try our problematic scenario from before:
empties().[0] <- [1]
List.iter (printfn "%s") (empties().[0])
we are creating a new array on each line, so the types can be different without breaking the type system.
Hopefully this helps explain the reasons behind the limitation a bit more.

Resources