FsCheck with Setup and Teardown - f#

Question

Summary
Are there any events that can be run before every property case so that I can run setup and teardown for each run of a property?
Full version
I want to be able to test paired behaviors like "I can always fetch written records" or "output of readAllLines equals input to writeAllLines" with properties. I also want the property to not care how the operation sets are implemented (i.e. if any resources need to be cleaned up).
Each run of the property should
be independent from other runs
maintain state between calls of the operations within this single run
not know about how how the operations maintain state
not be a resource leak
I'm using FsCheck and Expecto. Examples will be in Expecto, but the problem isn't specific to the framework.
It is super easy to write this sort of setup and teardown with example-based tests. They take a predictable argument set, so I can run them in a wrapper that adds before and after events.
let testWithEnv setup cleanup name test =
let testWrap () =
let (api, env) = setup ()
test api
cleanup env
testCase name testWrap
The same can't be done with property tests. They have an unknown number of arguments that will largely be filled with random data.
I can apply the set of paired behaviors pretty easily, but any created resources like streams are left undisposed.
let testPropertyWithEnv setup cleanup name test =
let testWrap () =
let (api, env) = setup () // this is actually run once, but immutable so the individual runs don't leak state
test api // have to return this to pass along unapplied parameters
testProperty name testWrap
I've looked into
Runner events
Looking at how to run FsCheck tests the closest hooks appear to be
OnStartFixture which is only run once per test class
OnArguments is run after every pass and would potentially work to run cleanup
Model-based features
There is also the experimental Model-based testing features that could work. However, it seems really heavy considering I'm only concerned with the external consistency of operations. I do not want access to the backing state.
Giving up and inlining
I can always write
testProperty "name" (fun arg1 arg2 ->
let (api,env) = setup ()
//test code here
cleanup env
)
but I'd like to avoid the boilerplate in every property and the exposure of the backing state.
IDisposables
Disposable objects also don't address the lack of setup hook.
More hands-on run loop
I looked into ways of running the property tests in my wrapper, but the smallest runner Check.one is for a single property, with no hooks between runs of the property.
Lazy wrapper
Making the wrapper lazy also doesn't work testProperty name lazy(testWithSetup)

Answer 1

There isn't really anything in FsCheck to help you, and even if there was I don't think it'd be straightforwardly exposed in Expecto. I also don't think it's that straightforward to add on the FsCheck side - that said happy to discuss that further if you open an issue in the FsCheck repo.
In any case, with some clever use of partial application and at the cost of some slight boilerplate, it is in fact possible to wrap "variadic" functions, which I think is essentially what you're asking.
Behold, the code:
// these types are here to make the signatures look nicer
type Api = Api
type Env = Env
let testProperty k =
// call the property with "random" arguments
for i in 0..2 do
k i (char i) (string i)
let setup() =
printfn "setup ran"
(Api, Env)
let teardown Env =
printfn "teardown ran"
let test0 Api arg1 =
printfn "test0 %A" arg1
let test1 Api (arg1:int) (arg2:char) =
printfn "test1 %A %A" arg1 arg2
let test2 Api arg1 arg2 arg3 =
printfn "testFun %A %A %A" arg1 arg2 arg3
let testWithEnv (setup:unit -> Api*Env) (teardown: Env -> unit) (test: Api -> 'a) (k: 'a -> unit) :unit =
let (api, env) = setup()
k (test api)
teardown env
let (<*>) (f,k) arg =
f, (fun c -> k c arg)
let (<!>) f arg =
f, (fun k -> k arg)
let run (f, k) = f k
testProperty (fun arg1 arg2 arg3 ->
testWithEnv setup teardown test2 <!> arg1 <*> arg2 <*> arg3 |> run
)
The idea here is that you use the operators <!> and <*> to string together any number of arguments of any type, pass that to the testWithEnv function and then call run on the result. The operators and the run are basically necessary to build up and then apply the variadic list of arguments.
It's all type safe i.e. if you forget to pass an argument or it's of the wrong type, you'll get a type error, though admittedly it might not be as clear as normal function application.
I recommend pasting this in an IDE and checking out the types, that will help greatly with understanding what is going on.
There are a few other styles you can write this in. E.g. with a slightly different definition and a the function you can write something like
let (<+>) k arg =
fun c -> k c arg
let the arg =
fun k -> k arg
testWithEnv setup teardown test2 (the arg1 <+> arg2 <+> arg3)
This replaces the run function with the and needs only one operator <+>. There's probably other ways to cut it too, pick your poison.