I have a Deedle DataFrame of type Frame<int,string> that contains some missing values. I would like to convert the missing values into empty strings "". I tried to use the valueOr function but that did not help. Is there a way to do this?
Here is my DataFrame:
let s1 = Series.ofOptionalObservations [ 1 => Some("A"); 2 => None ]
let s2 = Series.ofOptionalObservations [ 1 => Some("B"); 2 => Some("C") ]
let df = Frame.ofColumns ["A", s1; "BC", s2]
Typing df;; in FSI yields some information including
ColumnTypes = seq [System.String; System.String];. So the values of df are of type string and not string option.
This is the function valueOr:
let valueOr (someDefault: 'a) (xo: 'a option) : 'a =
match xo with
| Some v -> v
| None -> someDefault
I defined an auxiliary function emptyFoo as:
let emptyFoo = valueOr ""
The signature of emptyFoo is string option -> string. This means emptyFoo should not be acceptable to the compiler in the following command:
let df' = Frame.mapValues emptyFoo df
This is because the values of df are of type string and not string option.
Still, the compiler does not complain and the code runs. However, df' still has a missing value.
Is there a way to transform the missing value into the empty string?
The Deedle documentation for Frame.mapValues:
Builds a new data frame whose values are the results of applying the specified function on these values, but only for those columns which can be converted to the appropriate type for input to the mapping function
So the mapping does nothing because strings are found, rather than string options.
I noticed another function that seems to do exactly what you want.
let df' = Frame.fillMissingWith "" df
The key thing I noticed was that Deedle shows those missing values as <missing>, suggesting that it uses it's own representation (as opposed to option for example). With that knowledge I guessed that the library would provide some way of manipulating missing values, so I explored the API by doing Frame. in my IDE and browsing the list of available functions and their documentation.
Related
I need to call bool.TryParse in F# and I'm having a little difficulty with the compiler / Ionide.
Ex.
let value = true
let parsable = bool.TryParse("True", &value)
There error I'm seeing in Ionide for VSCode looks like the following.
val value : bool Full name:
value
Assembly: example3
The type 'ByRefKinds.InOut' does not match the type 'ByRefKinds.In'F#
Compiler(1)
This is the first example that I've used call by ref keywords and syntax in F# so I'm not sure what I'm doing wrong and the byrefs documentation doesn't seem to be of much help on understanding this particular scenario.
It turns out that the main thing I was missing in my example was that value wasn't mutable as it would need to be when called in this context.
Working example:
let mutable value = true
let parsable = bool.TryParse("True", &value)
This solution came to me after reading a slightly related GitHub issue related to Span<T> and then also playing around in dotnet fsi which at least gave me the following clue.
let parsable = bool.TryParse("True", &value);;
-------------------------------------^^^^^^
stdin(34,38): error FS3230: A value defined in a module must be mutable in order to take its address, e.g. 'let mutable x = ...'
As it turns out however F# also seems to have sntaxtic sugar around the Try-Parse pattern which I began to recall after playing around with this example further. This reduced into the below alternative solution to calling bool.TryParse.
let value = true
let (parsable, _) = bool.TryParse "True"
Alternatively without having to bind parsable and to specify the default value within a single expression the following example may be more elegant.
let value = match bool.TryParse "True" with
| true, value -> value
| false, _ -> true
Perhaps there are pros and cons to the different ways of calling bool.TryParse and other Try-Parse methods but the important thing is that I found some solutions that work and get me past the initial stumbling blocks with regard to lack of F# documentation on the subject.
As you have already discovered, the error can be fixed by making the 'value' a mutable variable.
Saying that, the last solution solution you shared (using pattern matching) looks the nicest and safest.
If you happen to do many of these parsing operations, I would suggest extracting some utility functions. That will avoid having to correctly pattern match on (true, value) rather than (false, value) by mistake.
Sample code:
open System
module Option =
let ofTry<'a> (res:(bool*'a)) : 'a option =
match res with
| true, res -> Some res
| false, _ -> None
module Result =
let ofTry<'a,'b> (err:'b) (res:(bool*'a)) : Result<'a, 'b> =
res
|> csharpToOption
|> Option.toResult err
module Samples =
let withPatternMatch =
match Int32.TryParse "5" with
| true, value -> value |> Some
| false, _ -> None
let withOptionHelper = Int32.TryParse "5" |> Option.ofTry
let withResultHelper = Int32.TryParse "5" |> Result.ofTry "Not a number"
While writing some code yesterday, I ran into two odd problems, which neither me nor my functional programming oriented friend could figure out. We have looked at it for quite some time, and researched it on the net, but we were not able to find any answers anywhere, so here goes:
The issue is that in this code:
First weird problem:
let outer1 (bs : byte array) =
let rec inner (bs : byte array) (bacc : byte array) (i : int) =
match i with
| bs.Length -> bacc // <--- Error: bs is not recognized. Why?
| _ -> bacc.[i] <- bs.[i]
inner bs bacc (i + 1)
inner bs (Array.zeroCreate bs.Length) 0
The problem here is: FS0039: The namespace or module 'bs' is not defined.
How can this be? bs is in the function signature after all. Moreover, defining a new value with let bsLength = bs.Length works right before the match. But by doing so I see a new oddity:
let outer2 (bs : byte array) =
let rec inner (bs : byte array) (bacc : byte array) (i : int) =
let bsLength = bs.Length
match i with
| bsLength -> bacc
| _ -> bacc.[i] <- bs.[i] // <--- Warning: Rule never matched. Why?
inner bs bacc (i + 1)
inner bs (Array.zeroCreate bs.Length) 0
Here the problem is a warning that says: warning FS0026: This rule will never be matched.
I don't get that at all. i and the length of the array has no relation to each other. If I write an integer (for instance 10) instead of bsLength, the warning disappears.
Both your problems stem from the expectation that pattern matching allows using values and literals interchangeably. No, it does not. Pattern Matching (F#) topic on MSDN gives a good overview of supported pattern types and precedence rules of their application. The major principle simplifying a lengthy description there is: you cannot match a value unless this value is a literal, or identifier (a case value of a discriminated union, an exception label, or an active pattern case).
In your first problem point compiler treats bs.Length not as a property Length of array bs as you expect, but as a literal or identifier Length from non-existing module or namespace bs; as John Palmer pointed in his answer you may achieve the expected behavior by using variable pattern with a guard statement. A sample of legitimate use of the pattern matching expression resembling yours would be:
module bs =
[<Literal>]
let Length = 100
//.............................
let v = 100;
let s = match v with
| bs.Length -> "matched"
| _ -> "not matched";;
val s : string = "matched"
The second problem point is treated by compiler as variable pattern, and bsLength is assigned a value of i instead of values being compared, as you expected; second matching rule does not have chances to kick in.
The match statement doesn't work like you think it does - the correct syntax is
match i with
| t when t = bs.Length
In the second case, you actually create a new variable called bsLength which hides the definition of the earlier bsLength and matches all integers, so you get the rule never matched warning.
Following up my previous question, I'm slowly getting the hang of FParsec (though I do find it particularly hard to grok).
My next newbie F# question is, how do I extract data from the list the parser creates?
For example, I loaded the sample code from the previous question into a module called Parser.fs, and added a very simple unit test in a separate module (with the appropriate references). I'm using XUnit:
open Xunit
[<Fact>]
let Parse_1_ShouldReturnListContaining1 () =
let interim = Parser.parse("1")
Assert.False(List.isEmpty(interim))
let head = interim.Head // I realise that I have only one item in the list this time
Assert.Equal("1", ???)
Interactively, when I execute parse "1" the response is:
val it : Element list = [Number "1"]
and by tweaking the list of valid operators, I can run parse "1+1" to get:
val it : Element list = [Number "1"; Operator "+"; Number "1"]
What do I need to put in place of my ??? in the snippet above? And how do I check that it is a Number, rather than an Operator, etc.?
F# types (including lists) implement structural equality. This means that if you compare two lists that contain some F# types using =, it will return true when the types have the same length and contain elements with the same properties.
Assuming that the Element type is a discriminated union defined in F# (and is not an object type), you should be able to write just:
Assert.Equal(interim, [Number "1"; Operator "+"; Number "1"])
If you wanted to implement the equality yourself, then you could use pattern matching;
let expected = [Number "1"]
match interim, expected with
| Number a, Number b when a = b -> true
| _ -> false
I have a function that will create a select where clause, but right now everything has to be a string.
I would like to look at the variable passed in and determine what type it is and then treat it properly.
For example, numeric values don't have single quotes around them, option type will either be null or have some value and boolean will actually be zero or one.
member self.BuildSelectWhereQuery (oldUser:'a) = //'
let properties = List.zip oldUser.ToSqlValuesList sqlColumnList
let init = false, new StringBuilder()
let anyChange, (formatted:StringBuilder) =
properties |> Seq.fold (fun (anyChange, sb) (oldVal, name) ->
match(anyChange) with
| true -> true, sb.AppendFormat(" AND {0} = '{1}'", name, oldVal)
| _ -> true, sb.AppendFormat("{0} = '{1}'", name, oldVal)
) init
formatted.ToString()
Here is one entity:
type CityType() =
inherit BaseType()
let mutable name = ""
let mutable stateId = 0
member this.Name with get() = name and set restnameval=name <- restnameval
member this.StateId with get() = stateId and set stateidval=stateId <- stateidval
override this.ToSqlValuesList = [this.Name; this.StateId.ToString()]
So, if name was some other value besides a string, or stateId can be optional, then I have two changes to make:
How do I modify ToSqlValuesList to
have the variable so I can tell the
variable type?
How do I change my select function
to handle this?
I am thinking that I need a new function does the processing, but what is the best FP way to do this, rather than using something like typeof?
You can use a type test pattern in a match. Would this meet your needs?
let f (x : obj) =
match x with
| :? int -> "int"
| :? string -> "string"
| :? bool -> "bool"
| _ -> "who knows?"
I think that one clear functional approach would be to define a data type that represents the various (more complicated situations) that you need to handle. You mentioned that a value may be optional and that you need to distinguish numeric and textual values (for the encoding to SQL).
You could define a discriminated union (if there are other cases that you'd like to handle, the definition may be a bit more complicated):
type SqlValue =
| Missing
| Numeric of string
| Textual of string
Note that the Textual case also carries string, because I assume that the client who produces the value takes care of converting it to string - this is only information for your SQL query generator (so that it knows whether it needs to add quotes).
Your ToSqlValuesList member would return a list of values string & SqlValue, so for example, a sample product could be represented using the following list:
columns = [ "Name"; "Price"; "Description" ]
values = [ Textual("Tea"); Numeric(10); Missing ]
In the code that generates the SQL query, you'd use pattern matching to handle all the different cases (most importantly, encode string to avoid SQL injection in case the value is Textual :-)).
EDIT You'd need to implement the conversion from the specific data types to the SqlValue representation in every client. However, this can be simplified by writing a utility type (using the fact that members can be overloaded):
type SqlValue with
static member From(a:int) = Numeric(a.ToString())
static member From(a:int option) =
match a with None -> Missing | Some(n) -> SqlValue.From(n)
// ... similarly for other types
In the implementation of ToSqlValuesList, you would write SqlValue.From(description) and it would deal with the details autoamtically.
A more sophisticated approach would be to annotate public members of the types representing your data entities with .NET attributes and use Reflection to extract the values (and their types) at runtime. This is more advanced, but quite elegant (there is a nice exmaple of this technique in Don Syme's Expert F# book)
The Google yields plenty of example of adding and deleting entries in an F# dictionary (or other collection). But I don't see examples to the equivalent of
myDict["Key"] = MyValue;
I've tried
myDict.["Key"] <- MyValue
I have also attempted to declare the Dictionary as
Dictionary<string, mutable string>
as well several variants on this. However, I haven't hit on the correct combination yet... if it is actually possible in F#.
Edit: The offending code is:
type Config(?fileName : string) =
let fileName = defaultArg fileName #"C:\path\myConfigs.ini"
static let settings =
dict[ "Setting1", "1";
"Setting2", "2";
"Debug", "0";
"State", "Disarray";]
let settingRegex = new Regex(#"\s*(?<key>([^;#=]*[^;#= ]))\s*=\s*(?<value>([^;#]*[^;# ]))")
do File.ReadAllLines(fileName)
|> Seq.map(fun line -> settingRegex.Match(line))
|> Seq.filter(fun mtch -> mtch.Success)
|> Seq.iter(fun mtch -> settings.[mtch.Groups.Item("key").Value] <- mtch.Groups.Item("value").Value)
The error I'm getting is:
System.NotSupportedException: This value may not be mutated
at Microsoft.FSharp.Core.ExtraTopLevelOperators.dict#37-2.set_Item(K key, V value)
at <StartupCode$FSI_0036>.$FSI_0036_Config.$ctor#25-6.Invoke(Match mtch)
at Microsoft.FSharp.Collections.SeqModule.iter[T](FastFunc`2 action, IEnumerable`1 sequence)
at FSI_0036.Utilities.Config..ctor(Option`1 fileName)
at <StartupCode$FSI_0041>.$FSI_0041.main#()
stopped due to error
f# has two common associative data structures:
The one you are most used to, the mutable Dictionary which it inherits that's to it's presence in the BCL and uses a hashtable under the hood.
let dict = new System.Collections.Generic.Dictionary<string,int>()
dict.["everything"] <- 42
The other is known as Map and is, in common functional style, immutable and implemented with binary trees.
Instead of operations that would change a Dictionary, maps provide operations which return a new map which is the result of whatever change was requested. In many cases, under the hood there is no need to make an entirely new copy of the entire map, so those parts that can be shared normally are. For example:
let withDouglasAdams = Map.add "everything" 42 Map.empty
The value withDouglasAdams will remain forever as an association of "everything" to 42. so if you later do:
let soLong = Map.remove "everything" withDouglasAdams
Then the effect of this 'removal' is only visible via the soLong value.
F#'s Map is, as mentioned, implemented as a binary tree. Lookup is therefore O(log n) whereas a (well behaved) dictionary should be O(1). In practice a hash based dictionary will tend to outperform the tree based one in almost all simple (low number of elements, low probability of collision) as such is commonly used. That said the immutable aspect of the Map may allow you to use it in situations where the dictionary would instead require more complex locking or to write more 'elegant' code with fewer side effects and thus it remains a useful alternative.
This is not however the source of your problem. The dict 'operator' returns an explicity immutable IDictionary<K,T> implementation (despite not indicating this in it's documentation).
From fslib-extra-pervasives.fs (note also the use of options on the keys):
let dict l =
// Use a dictionary (this requires hashing and equality on the key type)
// Wrap keys in an Some(_) option in case they are null
// (when System.Collections.Generic.Dictionary fails). Sad but true.
let t = new Dictionary<Option<_>,_>(HashIdentity.Structural)
for (k,v) in l do
t.[Some(k)] <- v
let d = (t :> IDictionary<_,_>)
let c = (t :> ICollection<_>)
let ieg = (t :> IEnumerable<_>)
let ie = (t :> System.Collections.IEnumerable)
// Give a read-only view of the dictionary
{ new IDictionary<'key, 'a> with
member s.Item
with get x = d.[Some(x)]
and set (x,v) = raise (NotSupportedException(
"This value may not be mutated"))
...
What error do you get? I tried the following and it compiles just fine
let map = new System.Collections.Generic.Dictionary<string,int>()
map.["foo"] <- 42
EDIT Verify that this code ran just fine as well .