I have around 100k discriminated union cases I have to convert to strings, but it seems to be extremely slow.
As a comparison, the following executes (in F# interactive) in 3seconds on average :
open System
let buf = Text.StringBuilder()
let s = DateTime.Now
for i in 1 .. 100000 do
Printf.bprintf buf "%A" "OtherFinancingInterest" //string
buf.Length <- 0
printfn "elapsed : %0.2f" (DateTime.Now - s).TotalMilliseconds
While the following executes (also in F# interactive) in over a minute...
open System
let buf = Text.StringBuilder()
let s = DateTime.Now
for i in 1 .. 100000 do
Printf.bprintf buf "%A" OtherFinancingInterest //DU
buf.Length <- 0
printfn "elapsed : %0.2f" (DateTime.Now - s).TotalMilliseconds
The discriminated union has 25 values (the result is still extremely slow, around 16 seconds with two cases, but less so than with 25). Any idea if that is "normal" or if I may be doing something wrong ?
Many thanks
The %A format specifier pretty prints any F# value. It uses reflection to do so. It should only really be used for debugging purposes, and not in normal application code.
Note that using %s in your first example using a string makes it a lot faster because there is no type checking needed at runtime.
For the DU, there is a hack you could use to make the reflection only happen once on application load:
type FinancingInterest =
| OtherFinancingInterest
open FSharp.Reflection
let private OtherFinancingInterestStringMap =
FSharpType.GetUnionCases typeof<FinancingInterest>
|> Array.map (fun c -> FSharpValue.MakeUnion(c, [||]) :?> FinancingInterest)
|> Array.map (fun x -> x, sprintf "%A" x)
|> Map.ofArray
type FinancingInterest with
member this.AsString = OtherFinancingInterestStringMap |> Map.find this
You would also use this with the %s format specifier:
Printf.bprintf buf "%s" OtherFinancingInterest.AsString
I had similar timings to yours in your example, and now this one comes down to 40ms.
This only works as long as all of the DU cases don't have an arguments. You will get an exception on application load as soon as you try anything like this:
type FinancingInterest =
| Foo of string
| OtherFinancingInterest
Having said all this, I think you're better off writing a simple function that explicitly converts your type into a string value, writing out the names in full with repetition if necessary. The names of discriminated union cases should not generally be thought of as data that affects your program. You would usually expect to be able to safely rename case names without affecting runtime behaviour at all.
Related
I have a couple of wrappers of Printf-family API (e.g. for logging)
type Logger(writer: TextWriter) =
...
member x.Log (fmt: PrintfFormat<'Printer, _, _, _>): 'Printer =
Printf.fprintfn writer fmt
Let's say I want to introduce the notion of "log level" to it, add Logger.Info(), Debug(), Trace() etc. which are basically another layer on top of Logger.Log(), and if logger.Level <- LogLevel.Info then Debug() and Trace() should turn into no-ops. Pretty common requirement, I'd say.
But I can't seem to return a 'Printer-typed value without actually calling any of Printf-family API.
type LogLevel = TRACE | DEBUG | INFO
type LevelLogger(writer: TextWriter, level: LogLevel) =
inherit Logger(writer)
member x.Info (fmt: PrintfFormat<'Printer, _, _, _>): 'Printer =
if level <= LogLevel.INFO then
x.Log fmt
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
warning FS0064: This construct causes code to be less generic than indicated by the type annotations. The type variable 'Printer has been constrained to be type 'unit'.
let l0 = Logger(Console.Error)
l0.Log "foo %d %s %f" 1 "foo" 1.23
let l1 = LevelLogger(Console.Error, LogLevel.INFO)
l1.Info "foo %d %s %f" 1 "foo" 1.23
^^^^^^^^^^^^^^^^^^^^^^
error FS0001: This expression was expected to have type
'unit'
but here has type
''a -> string -> 'b -> unit'
error FS0003: This value is not a function and cannot be applied.
It would be ideal if I could write
member x.Info (fmt: PrintfFormat<'Printer, _, _, _>): 'Printer =
if level <= LogLevel.INFO then x.Log fmt
else Printf.noop fmt
But does there exist such a utility API? Or can we write one?
Workarounds just for "getting things done" that I'm aware of:
fprintfn TextWriter.Null, I believe, still formats all args and can be unnecessarily expensive
kbprintfn with a continuation that conditionally calls writer.WriteLine(sb) has the same issue
In C++ codebase, this kind of dynamic config is often complemented with preprocessor directives (#if ...), like, enabling TRACE level only under -DDEBUG; but I don't know if we can write an ergonomic API with them in F#.
Also, PrintfFormat is always allocated out of a format string anyways, then the "no-op" API can never be a real no-op? But I'm hoping it might still be much light-weight than TextWriter.Null, if that's ever possible.
Ahhh why is PrintfImpl internal??
https://github.com/fsharp/fsharp/blob/master/src/fsharp/FSharp.Core/printf.fs
Update: My attempts so far
https://sharplab.io/#gist:70dd6f915ba14f36c122f601732af1c1
LevelLogger0 doesn't type check, and my understanding is that both LevelLogger1 and 2 would waste CPU & RAM in logger.Trace "%A" (Array.zeroCreate<double> 1_000_000_000) even when the current threshold is INFO.
I tried the same a time ago and did not find a solution.
Meanwhile string interpolation was added and I find it better at all ends over F#s safe but (as we see here a bit tricky) printf. String interpolation is type safe, improves readability and leads to less brackets. Another disadvantage of printf is its behavior during debugging.
I do logging with an extension method that has the Conditional attribute
type ILogger with // this is Serilog.ILogger which I use
[<Conditional("LOG")>]
member o.InformationConditional(s: string) =
o.Information(s)
to be used like
let a = 72
let b = "hello"
logger.InformationConditional($"foo {a} bar {b}")
Keeping with printf you can create the string with sprintf.
The easiest way to get something like your Printf.noop is to use Printf.kprintf, which takes a continuation that is called with the composed string. If you pass ignore as the continuation, this does nothing. However, this unfortunately does not quite work:
member x.Info (fmt: PrintfFormat<'Printer, _, _, _>): 'Printer =
if level <= LogLevel.INFO then x.Log fmt
else Printf.kprintf ignore fmt
This would logically do the trick, but the issue is that the type of fmt expected by kprintf and fprintf differs (in that they have a different state). So, if you want to do this, you will always need to use kprintf and then decide what to do based on the log level inside the continuation:
member x.Info (fmt: PrintfFormat<'Printer, _, _, _>): 'Printer =
fmt |> Printf.kprintf (fun s ->
if level <= LogLevel.INFO then x.Log "%s" s)
I'm trying to implement a custom Arbitrary that generates glob syntax patterns like a*c?. I think my implementation is correct, it's just that, when running the test with Xunit, FsCheck doesn't seem to be using the custom arbitrary Pattern to generate the test data. When I use LINQPad however everything works as expected. Here's the code:
open Xunit
open FsCheck
type Pattern = Pattern of string with
static member op_Explicit(Pattern s) = s
type MyArbitraries =
static member Pattern() =
(['a'..'c']#['?'; '*'])
|> Gen.elements
|> Gen.nonEmptyListOf
|> Gen.map (List.map string >> List.fold (+) "")
|> Arb.fromGen
|> Arb.convert Pattern string
Arb.register<MyArbitraries>() |> ignore
[<Fact>]
let test () =
let prop (Pattern p) = p.Length = 0
Check.QuickThrowOnFailure prop
This is the output:
Falsifiable, after 2 tests (0 shrinks) (StdGen (1884571966,296370531)): Original: Pattern null with exception: System.NullReferenceException ...
And here is the code I'm running in LINQPad along with the output:
open FsCheck
type Pattern = Pattern of string with
static member op_Explicit(Pattern s) = s
type MyArbitraries =
static member Pattern() =
(['a'..'c']#['?'; '*'])
|> Gen.elements
|> Gen.nonEmptyListOf
|> Gen.map (List.map string >> List.fold (+) "")
|> Arb.fromGen
|> Arb.convert Pattern string
Arb.register<MyArbitraries>() |> ignore
let prop (Pattern p) = p.Length = 0
Check.Quick prop
Falsifiable, after 1 test (0 shrinks) (StdGen (1148389153,296370531)): Original: Pattern "a*"
As you can see FsCheck generates a null value for the Pattern in the Xunit test although I'm using Gen.elements and Gen.nonEmptyListOf to control the test data. Also, when I run it a couple times, I'm seeing test patterns that are out of the specified character range. In LINQPad those patterns are generated correctly. I also tested the same with a regular F# console application in Visual Studio 2017 and there the custom Arbitrary works as expected as well.
What is going wrong? Is FsCheck falling back to the default string Arbitrary when running in Xunit?
You can clone this repo to see for yourself: https://github.com/bert2/GlobMatcher
(I don't want to use Prop.forAll, because each test will have multiple custom Arbitrarys and Prop.forAll doesn't go well with that. As far as I know I can only tuple them up, because the F# version of Prop.forAll only accepts a single Arbitrary.)
Don't use Arb.register. This method mutates global state, and due to the built-in parallelism support in xUnit.net 2, it's undetermined when it runs.
If you don't want to use the FsCheck.Xunit Glue Library, you can use Prop.forAll, which works like this:
[<Fact>]
let test () =
let prop (Pattern p) = p.Length = 0
Check.QuickThrowOnFailure (Prop.forAll (MyArbitraries.Pattern()) prop)
(I'm writing this partially from memory, so I may have made some small syntax mistakes, but hopefully, this should give you an idea on how to proceed.)
If, on the other hand, you choose to use FsCheck.Xunit, you can register your custom Arbitraries in a Property annotation, like this:
[<Property(Arbitrary = [|typeof<MyArbitraries>|])>]
let test (Pattern p) = p.Length = 0
As you can see, this takes care of much of the boilerplate; you don't even have to call Check.QuickThrowOnFailure.
The Arbitrary property takes an array of types, so when you have more than one, this still works.
If you need to write many properties with the same array of Arbitraries, you can create your own custom attributes that derives from the [<Property>] attribute. Here's an example:
type Letters =
static member Char() =
Arb.Default.Char()
|> Arb.filter (fun c -> 'A' <= c && c <= 'Z')
type DiamondPropertyAttribute() =
inherit PropertyAttribute(
Arbitrary = [| typeof<Letters> |],
QuietOnSuccess = true)
[<DiamondProperty>]
let ``Diamond is non-empty`` (letter : char) =
let actual = Diamond.make letter
not (String.IsNullOrWhiteSpace actual)
All that said, I'm not too fond of 'registering' Arbitraries like this. I much prefer using the combinator library, because it's type-safe, which this whole type-based mechanism isn't.
Coming from an OO background, I am having trouble wrapping my head around how to solve simple issues with FP when trying to avoid mutation.
let mutable run = true
let player1List = ["he"; "ho"; "ha"]
let addValue lst value =
value :: lst
while run do
let input = Console.ReadLine()
addValue player1List input |> printfn "%A"
if player1List.Length > 5 then
run <- false
printfn "all done" // daz never gunna happen
I know it is ok to use mutation in certain cases, but I am trying to train myself to avoid mutation as the default. With that said, can someone please show me an example of the above w/o using mutation in F#?
The final result should be that player1List continues to grow until the length of items are 6, then exit and print 'all done'
The easiest way is to use recursion
open System
let rec makelist l =
match l |> List.length with
|6 -> printfn "all done"; l
| _ -> makelist ((Console.ReadLine())::l)
makelist []
I also removed some the addValue function as it is far more idiomatic to just use :: in typical F# code.
Your original code also has a common problem for new F# coders that you use run = false when you wanted run <- false. In F#, = is always for comparison. The compiler does actually warn about this.
As others already explained, you can rewrite imperative loops using recursion. This is useful because it is an approach that always works and is quite fundamental to functional programming.
Alternatively, F# provides a rich set of library functions for working with collections, which can actually nicely express the logic that you need. So, you could write something like:
let player1List = ["he"; "ho"; "ha"]
let player2List = Seq.initInfinite (fun _ -> Console.ReadLine())
let listOf6 = Seq.append player1List list2 |> Seq.take 6 |> List.ofSeq
The idea here is that you create an infinite lazy sequence that reads inputs from the console, append it at the end of your initial player1List and then take first 6 elements.
Depending on what your actual logic is, you might do this a bit differently, but the nice thing is that this is probably closer to the logic that you want to implement...
In F#, we use recursion to do loop. However, if you know how many times you need to iterate, you could use F# List.fold like this to hide the recursion implementation.
[1..6] |> List.fold (fun acc _ -> Console.ReadLine()::acc) []
I would remove the pipe from match for readability but use it in the last expression to avoid extra brackets:
open System
let rec makelist l =
match List.length l with
| 6 -> printfn "all done"; l
| _ -> Console.ReadLine()::l |> makelist
makelist []
A real F# noob question, but what is |> called and what does it do?
It's called the forward pipe operator. It pipes the result of one function to another.
The Forward pipe operator is simply defined as:
let (|>) x f = f x
And has a type signature:
'a -> ('a -> 'b) -> 'b
Which resolves to: given a generic type 'a, and a function which takes an 'a and returns a 'b, then return the application of the function on the input.
You can read more detail about how it works in an article here.
I usually refer to |> as the pipelining operator, but I'm not sure whether the official name is pipe operator or pipelining operator (though it probably doesn't really matter as the names are similar enough to avoid confusion :-)).
#LBushkin already gave a great answer, so I'll just add a couple of observations that may be also interesting. Obviously, the pipelining operator got it's name because it can be used for creating a pipeline that processes some data in several steps. The typical use is when working with lists:
[0 .. 10]
|> List.filter (fun n -> n % 3 = 0) // Get numbers divisible by three
|> List.map (fun n -> n * n) // Calculate squared of such numbers
This gives the result [0; 9; 36; 81]. Also, the operator is left-associative which means that the expression input |> f |> g is interpreted as (input |> f) |> g, which makes it possible to sequence multiple operations using |>.
Finally, I find it quite interesting that pipelining operaor in many cases corresponds to method chaining from object-oriented langauges. For example, the previous list processing example would look like this in C#:
Enumerable.Range(0, 10)
.Where(n => n % 3 == 0) // Get numbers divisible by three
.Select(n => n * n) // Calculate squared of such numbers
This may give you some idea about when the operator can be used if you're comming fromt the object-oriented background (although it is used in many other situations in F#).
As far as F# itself is concerned, the name is op_PipeRight (although no human would call it that). I pronounce it "pipe", like the unix shell pipe.
The spec is useful for figuring out these kinds of things. Section 4.1 has the operator names.
http://research.microsoft.com/en-us/um/cambridge/projects/fsharp/manual/spec.html
Don't forget to check out the library reference docs:
http://msdn.microsoft.com/en-us/library/ee353754(v=VS.100).aspx
which list the operators.
I'm trying to learn F# by going through some of the Euler problems and I found an issue I haven't been able to figure out. This is my naive solution.
let compute =
let mutable f = false
let mutable nr = 0
while f = false do
nr <- nr + 20
f = checkMod nr
nr
When i do this I get the error message warning FS0020: This expression should have type 'unit', but has type 'bool' on the expression "nr <- nr +20". I've tried rewriting and moving the expressions around and I always get that error on the line below the while statement.
I'm writing this using VS2010 Beta.
Since I can imagine this weg page becoming the 'canonical' place to look up information about warning FS0020, here's my quick summary of the three commonest cases in which you get the warning, and how to fix them.
Intentionally discarding the result of a function that is called only for its side-effects:
// you are calling a function for its side-effects, intend to ignore result
let Example1Orig() =
let sb = new System.Text.StringBuilder()
sb.Append("hi") // warning FS0020
sb.Append(" there") // warning FS0020
sb.ToString()
let Example1Fixed() =
let sb = new System.Text.StringBuilder()
sb.Append("hi") |> ignore
sb.Append(" there") |> ignore
sb.ToString()
Warning is useful, pointing out an error (function has no effects):
// the warning is telling you useful info
// (e.g. function does not have an effect, rather returns a value)
let Example2Orig() =
let l = [1;2;3]
List.map (fun x -> x * 2) l // warning FS0020
printfn "doubled list is %A" l
let Example2Fixed() =
let l = [1;2;3]
let result = List.map (fun x -> x * 2) l
printfn "doubled list is %A" result
Confusing assignment operator and equality comparison operator:
// '=' versus '<-'
let Example3Orig() =
let mutable x = 3
x = x + 1 // warning FS0020
printfn "%d" x
let Example3Fixed() =
let mutable x = 3
x <- x + 1
printfn "%d" x
The following line:
f = checkMod nr
is an equality check, not an assignment as I believe you are intending. Change it to:
f <- checkMod nr
and all should work fine. I'm not sure why you've used the correct syntax on the previous line and not that line...
Also, the line while f = false do should really be simplified to while not f do; equality checks on booleans are rather convoluted.
As I side note, I feel a need to point out that you are effectively trying to use F# as an imperative language. Use of mutable variables and while loops are strongly discouraged in functional languages (including F#), especially when a purely functional (and simpler) solution exists, as in this situation. I recommend you read up a bit on programming in the functional style. Of course, just getting to grips with the syntax is a useful thing in itself.
If you're trying to adopt the functional style, try to avoid mutable values.
For example like this:
let nr =
let rec compute nr =
if checkMod nr then nr else compute (nr + 20)
compute 0
while expressions in F# take a little getting used to if you're coming from an imperative language. Each line in a while expression must evaluate to unit (think void from C++/C#). The overall expression then also evaluates to unit.
In the example:
nr <- nr + 20
evaluates to unit whereas
f = checkMod nr
evaluates to a bool as Noldorin noted. This results in a warning message being reported. You can actually turn the warning off if you so desire. Just put the following at the top of your file:
#nowarn "0020"
I've been programming in an imperative style for a long time, so getting used to the functional programming mindset took a while.
In your example, you're trying to find the first multiple of 20 that passes your checkMod test. That's the what part. For the functional how part, I recommend browsing through the methods available to sequences. What you need is the first element of a sequence (multiples of 20) passing your test, like this:
let multi20 = Seq.initInfinite (fun i -> i*20)
let compute = multi20 |> Seq.find checkMod
The first let generates an infinite list of twentyples (I made that one up). The second let finds the first number in said list that passes your test. Your task is to make sure that there actually is a number that will pass the test, but that's of course also true for the imperative code.
If you want to condense the two above lines into one, you can also write
let computeCryptic = Seq.initInfinite ((*) 20) |> Seq.find checkMod
but I find that pulling stunts like that in code can lead to headaches when trying to read it a few weeks later.
In the same spirit as Brian's post, here is another way to get warning FS0020: In a nutshell, I accidentally tupled the function arguments.
Being an F# newbie, I had a difficult time debugging the code below, which for the second line (let gdp...) gave the warning FS0020: This expression should have type 'unit', but has type '(string -> ^a -> unit) * string * float'. It turns out that line was not the problem at all; instead, it was the printfn line that was messed up. Removing the comma separators from the argument list fixed it.
for country in wb.Regions.``Arab World``.Countries do
let gdp = country.Indicators.``GDP per capita (current US$)``.[2010]
let gdpThous = gdp / 1.0e3
printfn "%s, %s (%.2f)" country.Name, country.CapitalCity, gdpThous