I'm having trouble understanding how to manage multiple observables that depend on each other. I want to define a function with the following signature:
clock:IObservable<unit> -> obs:IObservable<'a> -> IObservable<'a>
So that events from obs can only be emitted once per clock tick, and excess events from obs are discarded.
I have tried mapping the two observables and then merging them into one stream, but it is not the solution.
The built-in F# library for Observables comes with only a few basic functions, so this is not something you can easily do using the built-in primitives. You can probably use a function from the full Rx library, which has a nice F# wrapper and comes with huge number of operations (but that makes it a bit hard to figure out which one is the one that you need).
An alternative purely F# approach would be to use agent-based programming. This lets you nicely handle complex concurrency patterns. The following implements an agent that has Tick and Event as two kinds of messages. It remembers the last Event and when Tick happens, it trigges the returned observable with the last Event value:
open System
type WhenTickMessage<'T> =
| Tick
| Event of 'T
let whenTick (clock:IObservable<_>) (event:IObservable<_>) =
let result = new Event<_>()
let agent = MailboxProcessor.Start(fun inbox ->
let rec loop event = async {
let! msg = inbox.Receive()
match msg with
| Tick ->
event |> Option.iter (fun e -> result.Trigger(e))
return! loop None
| Event e ->
return! loop (Some e) }
loop None)
clock.Add(fun _ -> agent.Post Tick)
event.Add(fun e -> agent.Post (Event e))
result.Publish
Related
I have a list/sequence as follows Result<DataEntry, exn> []. This list is populated by calling multiple API endpoints in parallel based on some user inputs.
I don't care if some of the calls fail as long as at least 1 succeeds. I then need to perform multiple operations on the success list.
My question is how to partition the Result list into exn [] and DataEntry [] lists. I tried the following:
// allData is Result<DataEntry, exn> []
let filterOutErrors (input: Result<DataEntry, exn>) =
match input with
| Ok v -> true
| _ -> false
let values, err = allData |> Array.partition filterOutErrors
This in principle meets the requirement since values contains all the success cases but understandably the compiler can't infer the types so both values and err contains Result<DataEntry, exn>.
Is there any way to split a list of result Result<Success, Err> such that you end up with separate lists of the inner type?
Is there any way to split a list of result Result<Success, Err> such that you end up with separate lists of the inner type?
Remember that Seq / List / Array are foldable, so you can use fold to convert a Seq / List / Array of 'Ts into any other type 'S. Here you want to go from []Result<DataEntry, exn> to, e.g., the tuple list<DataEntry> * list<exn>. We can define the following folder function, that takes an initial state s of type list<'a> * list<'b> and a Result Result<'a, 'b> and returns your tuple of lists list<'a> * list<'b>:
let listFolder s r =
match r with
| Ok data -> (data :: (fst s), snd s)
| Error err -> (fst s, err :: (snd s))
then you can fold over your array as follows:
let (values, err) = Seq.fold listFolder ([], []) allData
You can extract the good and the bad like this.
let values =
allData
|> Array.choose (fun r ->
match r with
| Result.Ok ok -> Some ok
| Result.Error _ -> None)
let err =
allData
|> Array.choose (fun r ->
match r with
| Result.Ok _ -> None
| Result.Error error -> Some error)
You seem confused about whether you have arrays or lists. The F# code you use, in the snippet and in your question text, all points to use of arrays, in spite of you several times mentioning lists.
It has recently been recommended that we use array instead of the [] symbol in types, since there are inconsistencies in the way F# uses the symbol [] to mean list in some places, and array in other places. There is also the symbol [||] for arrays, which may add more confusion.
So that would be recommending Result<DataEntry,exn> array in this case.
The answer from Víctor G. Adán is functional, but it's a downside that the API requires you to pass in two empty lists, exposing the internal implementation.
You could wrap this into a "starter" function, but then the code grows, requires nested functions or using modules and the intention is obscured.
The answer from Bent Tranberg, while more readable requires two passes of the data, and it seems inefficient to map into Option type just to be able to filter on it using .Choose.
I propose KISS'ing it with some good old mutation.
open System.Collections.Generic
let splitByOkAndErrors xs =
let oks = List<'T>()
let errors = List<'V>()
for x in xs do
match x with
| Ok v -> oks.Add v
| Error e -> errors.Add e
(oks |> seq, errors |> seq)
I know I know, mutation, yuck right? I believe you should not shy away from that even in F#, use the right tool for every situation: the mutation is kept local to the function, so it's still pure. The API is clean just taking in the list of Result to split, there is no concepts like folding, recursive calls, list cons pattern matching etc. to understand, and the function won't reverse the input list, you also have the option to return array or seq, that is, you are not confined to a linked list that can only be appended to in O(1) in the head - which in my experience seldom fits well into business case, win win win in my book.
I general, I hope to see F# grow into a more multi-paradigm programming language in the community's mind. It's nice to see these functional solutions, but I fear they scare some people away unnecessarily, as F# is already multi-paradigm!
Currently, I have a function that receives raw data from the outside, process it, and sends it to a callback:
let Process callback rawData =
let data = rawData //transforming into actual data....
callback data
callback is a function 'T -> unit. In my case specifically, it's the Post function of a MailboxProcessor (being called like Process mailbox.Post rawData)
The process function is called multiple times, and each time I push the processed data into the mailbox queue. So far so good.
Now I want to change this code in a way I can publish this processed data to various consumers, using the rx extensions for FSharp (FSharp.Control.Reactive). This means that callback will be either an Observable, or a function that publishes to subscribers. How do I do this?
I found two options:
Create a class that implements IObservable, and pass that object to the Process function. I'd like to avoid creating classes if possible.
Use the Subject.behavior. This does exactly what I want, except it requires a initial state, which doesnt make sense semantically in this case, and apparently Subjects are frowned upon (from a link in the ReactiveX site http://davesexton.com/blog/post/To-Use-Subject-Or-Not-To-Use-Subject.aspx).
What would be the better way, from a functional programming perspective? Is there a better way?
Here's one idea: You can use an object expression to implement IObservable<_> without the overhead of an explicit class:
let createObservable subscribe =
{
new IObservable<_> with
member __.Subscribe(observer) =
subscribe observer
}
To use this, specify a subscribe function of type IObserver<_> -> IDisposable. No classes needed.
Using the observe { .. } computation builder works, but there is a function in the FSharp.Control.Reactive library that does the same thing:
open FSharp.Control.Reactive
let obs = Observable.ofSeq [1;2;3;4;5]
If I was using the observe { .. } computation builder, I'd also use the fact that it supports for loop, which makes your code a bit simpler:
let Process initialData = observe {
for x in initialData do yield x }
Got it. Fsharp reactive provides the keyword observe from the module FSharp.Control.Reactive.Builders. This allows you to create ad-hoc observables:
open FSharp.Control.Reactive.Builders
//In my real case, "initialData" is a byte stream and
//at each step I read a few bytes off of it
let Process initialData =
let rec loop data =
observe {
match data with
| x :: xs ->
yield x
yield! loop xs
| [] -> ()
}
loop initialData
let obs = Process ([1;2;3;4;5])
obs.Subscribe(fun d -> printfn "Consumer A: %A" d) |> ignore
obs.Subscribe(fun d -> printfn "Consumer B: %A" d) |> ignore
Threading.Thread.Sleep 1000
obs.Subscribe(fun d -> printfn "Late consumer: %A" d) |> ignore
Important to note that this creates a cold observable, so the Late consumer receives all events.
I was trying to write a generic mapFoldWhile function, which is just mapFold but requires the state to be an option and stops as soon as it encounters a None state.
I don't want to use mapFold because it will transform the entire list, but I want it to stop as soon as an invalid state (i.e. None) is found.
This was myfirst attempt:
let mapFoldWhile (f : 'State option -> 'T -> 'Result * 'State option) (state : 'State option) (list : 'T list) =
let rec mapRec f state list results =
match list with
| [] -> (List.rev results, state)
| item :: tail ->
let (result, newState) = f state item
match newState with
| Some x -> mapRec f newState tail (result :: results)
| None -> ([], None)
mapRec f state list []
The List.rev irked me, since the point of the exercise was to exit early and constructing a new list ought to be even slower.
So I looked up what F#'s very own map does, which was:
let map f list = Microsoft.FSharp.Primitives.Basics.List.map f list
The ominous Microsoft.FSharp.Primitives.Basics.List.map can be found here and looks like this:
let map f x =
match x with
| [] -> []
| [h] -> [f h]
| (h::t) ->
let cons = freshConsNoTail (f h)
mapToFreshConsTail cons f t
cons
The consNoTail stuff is also in this file:
// optimized mutation-based implementation. This code is only valid in fslib, where mutation of private
// tail cons cells is permitted in carefully written library code.
let inline setFreshConsTail cons t = cons.(::).1 <- t
let inline freshConsNoTail h = h :: (# "ldnull" : 'T list #)
So I guess it turns out that F#'s immutable lists are actually mutable because performance? I'm a bit worried about this, having used the prepend-then-reverse list approach as I thought it was the "way to go" in F#.
I'm not very experienced with F# or functional programming in general, so maybe (probably) the whole idea of creating a new mapFoldWhile function is the wrong thing to do, but then what am I to do instead?
I often find myself in situations where I need to "exit early" because a collection item is "invalid" and I know that I don't have to look at the rest. I'm using List.pick or Seq.takeWhile in some cases, but in other instances I need to do more (mapFold).
Is there an efficient solution to this kind of problem (mapFoldWhile in particular and "exit early" in general) with functional programming concepts, or do I have to switch to an imperative solution / use a Collections.Generics.List?
In most cases, using List.rev is a perfectly sufficient solution.
You are right that the F# core library uses mutation and other dirty hacks to squeeze some more performance out of the F# list operations, but I think the micro-optimizations done there are not particularly good example. F# list functions are used almost everywhere so it might be a good trade-off, but I would not follow it in most situations.
Running your function with the following:
let l = [ 1 .. 1000000 ]
#time
mapFoldWhile (fun s v -> 0, s) (Some 1) l
I get ~240ms on the second line when I run the function without changes. When I just drop List.rev (so that it returns the data in the other order), I get around ~190ms. If you are really calling the function frequently enough that this matters, then you'd have to use mutation (actually, your own mutable list type), but I think that is rarely worth it.
For general "exit early" problems, you can often write the code as a composition of Seq.scan and Seq.takeWhile. For example, say you want to sum numbers from a sequence until you reach 1000. You can write:
input
|> Seq.scan (fun sum v -> v + sum) 0
|> Seq.takeWhile (fun sum -> sum < 1000)
Using Seq.scan generates a sequence of sums that is over the whole input, but since this is lazily generated, using Seq.takeWhile stops the computation as soon as the exit condition happens.
I want to process to types of a message
Add x makes program remember number x
Print makes it print all remembered numbers
Why would I write this:
open System
type Message =
| Add of int
| Print
let mailbox = new MailboxProcessor<Message>(fun inbox ->
let rec loop history = async{
let! msg=inbox.Receive()
match msg with
| Add x -> return! loop(history + x.ToString()+" ")
| Print ->
printfn "%s" history
return! loop(history)
}
loop ""
)
[<EntryPoint>]
let main argv =
mailbox.Start()
mailbox.Post(Add 12)
mailbox.Post(Add 56)
mailbox.Post(Print)
mailbox.Post(Add 34)
mailbox.Post(Print)
ignore <| Console.ReadLine()
0
instead of this:
open System
open System.Reactive.Subjects
type Message =
| Add of int
| Print
let subject = new Subject<Message>()
[<EntryPoint>]
let main argv =
subject
|> Observable.scan(fun history msg ->
match msg with
| Add x -> history + x.ToString()+" "
| Print ->
printfn "%s" history
history
) ""
|> Observable.subscribe(fun _->())
|> ignore
subject.OnNext(Add 12)
subject.OnNext(Add 56)
subject.OnNext(Print)
subject.OnNext(Add 34)
subject.OnNext(Print)
ignore <| Console.ReadLine()
0
The MailboxProcessor adds additional level of complexity. I need a state machine which takes a state and returns a state. But it forces me to take inbox, which is used to receive state.
Does it has any advantages to IObservable?
No, they're not duplicates of one another. MailboxProcessor and IObservable are low-level building blocks of two different models of computation - actor model and functional reactive programming respectively.
Both deal with asynchronicity, but emphasize different qualities. It's might be possible to build your solution in terms of one or the other - as you noticed in your simple example - but you will find one or the other more natural to use in a particular context.
MailboxProcessors are particularly useful for thread-safe, lock-free access to a resource, such as a file. You can have multiple threads manipulating the resource through an asynchronous interface, and the MailboxProcessor guarantees that only one of those requests is processed at a time.
I need my state to be passed along while being able to chain functions with the maybe workflow. Is there a way for 2 workflows to share the same context? If no, what is the way of doing it?
UPDATE:
Well, I have a state that represents a segment of available ID's for the entities that I am going to create in the database. So once an ID is acquired the state has to be transformed to a newer state with the next available ID and thrown away so that nobody can use it again. I don't want to mutate the state for the sake of being idiomatic. The State monad looks like a way to go as it hides the transformation and passes the state along. Once the state workflow is in place I cannot use the Maybe workflow which is something I use everywhere.
As stated in the previous answer, one way to combine workflows in F# (Monads in Haskell) is by using a technique called Monad Transformers.
In F# this is really tricky, here is a project that deals with that technique.
It's possible to write the example of the previous answer by automatically combining State and Maybe (option), using that library:
#r #"c:\packages\FSharpPlus-1.0.0\lib\net45\FSharpPlus.dll"
open FSharpPlus
open FSharpPlus.Data
// Stateful computation
let computation =
monad {
let! x = get
let! y = OptionT (result (Some 10))
do! put (x + y)
let! x = get
return x
}
printfn "Result: %A" (State.eval (OptionT.run computation) 1)
So this is the other alternative, instead of creating your custom workflow, use a generic workflow that will be automatically inferred (a-la Haskell).
In F# you cannot easily mix different types of computation expressions as you would do in Haskell by using Monad Transformers or similar techniques. You could however build your own Monad, embedding state threading and optional values, as in:
type StateMaybe<'T> =
MyState -> option<'T> * MyState
// Runs the computation given an initial value and ignores the state result.
let evalState (sm: StateMaybe<'T>) = sm >> fst
// Computation expression for SateMaybe monad.
type StateMaybeBuilder() =
member this.Return<'T> (x: 'T) : StateMaybe<'T> = fun s -> (Some x, s)
member this.Bind(sm: StateMaybe<'T>, f: 'T -> StateMaybe<'S>) = fun s ->
let mx,s' = sm s
match mx with
| Some x -> f x s'
| None -> (None, s)
// Computation expression builder.
let maybeState = new StateMaybeBuilder()
// Lifts an optional value to a StateMaybe.
let liftOption<'T> (x: Option<'T>) : StateMaybe<'T> = fun s -> (x,s)
// Gets the current state.
let get : StateMaybe<MyState> = fun s -> (Some s,s)
// Puts a new state.
let put (x: MyState) : StateMaybe<unit> = fun _ -> (Some (), x)
Here's an example computation:
// Stateful computation
let computation =
maybeState {
let! x = get
let! y = liftOption (Some 10)
do! put (x + y)
let! x = get
return x
}
printfn "Result: %A" (evalState computation 1)
StateMaybe may be generalized further by making the type of the state component generic.
Others already gave you a direct answer to your question. However, I think that the way the question is stated leads to a solution that is not very idiomatic from the F# perspective - this might work for you as long as you are the only person working on the code, but I would recommend against doing that.
Even with the added details, the question is still fairly general, but here are two suggestions:
There is nothing wrong with reasonably used mutable state in F#. For example, it is perfectly fine to create a function that generates IDs and pass it along:
let createGenerator() =
let currentID = ref 0
(fun () -> incr currentID; !currentID)
Do you really need to generate the IDs while you are building the entities? It sounds like you could just generate a list of entities without ID and then use Seq.zip to zip the final list of entities with list of IDs.
As for the maybe computation, are you using it to handle regular, valid states, or to handle exceptional states? (It sounds like the first, which is the right way of doing things - but if you need to handle truly exceptional states, then you might want to use ordinary .NET exceptions).