I've a requirement where I need to cache the results of an event into an asyncSeq and iterate though this asyncSeq once a long running function has returned. The idea here is to iterate through profiles and apply runFunction to both the cached results as well as the future results (hence the use of ofObservableBuffered).
I would like to know what's the best way to do this. I'm using AsyncSeq.Cache as shown below. AsyncSeq.Cache is using a combination of ResizeArray and MailboxProcessor to accomplish the caching. However, I'm not sure if this will lead to a memory leak.
let profiles =
client.ProfileReceived
|> AsyncSeq.ofObservableBuffered
|> AsyncSeq.cache
do! longRunningFunction()
do! function2 (fun _ ->
async {
do! profiles
|> AsyncSeq.iterAsync(fun profile -> async {
do! runFunction profile
return()
})
}
Your question does not give all details about your scenario, but I think the answer is that you do not need AsyncSeq.cache and using just AsyncSeq.ofObservableBuffered should be enough.
Asynchronous sequences generate values on demand ("pull based") which means that elements are only generated when needed. Observables are "push based" which means that they generate data whenever the data source decides.
To map from "push based" to "pull based", you either need to drop data (when the listener is not ready to accept the next item) or cache data. If you cache, then you may potentially run out of memory if the producer is faster than the consumer - but this is inevitable by design problem. The AsyncSeq.ofObservableBuffered function does the latter.
AsyncSeq.cache is useful if you have one data source, but want to consume it from multiple different places. Without this, the data source will generate data repeatedly for each consumer, so cache enables generating the data just once. However, if you are using ofObservableBuffered is already doing the same thing - i.e. caching all the generated values.
I believe the only reason why you might want to keep cache is if you subscribe to the observable at a later time - in which case, it would keep elements generated before you started consuming values (and use two caches, one of which would keep not-yet-consumed elements (possibly safe if the consumer is fast enough) and one which would keep all the already generated elements (certainly potential to run out of memory over a longer period of time)).
The setup is similar to this.
One agent, (dataSource) is generating data, and a single agent (dataProcessor) is processing the data. There is a lot more data being generated than dataProcessor can process, and I am not interested in processing all messages, just processing the latest piece of data.
One possible solution, proposed there by Jon Harrop there "is to greedily eat all messages in the inbox when one arrives and discard all but the most recent".
Another approach is not to listen for all messages, but rather for dataProcessor to PostAndReply dataSource to get the latest piece of data.
What are the pros and cons of these approaches?
This is an intriguing question and there are quite likely several possible perspectives. I think the most notable aspect is that the choice will affect how you design the API at the interface between the two components:
In "Consume all" approach, the producer has a very simple API where it triggers some event whenever a value is produced and your consumer will subscribe to it. This means that you could have other subscribers listening to updates from the producer and doing something else than your consumer from this question.
In "Call to get latest" approach, the producer will presumably need to be written so that it keeps the current state and discards old values. It will then provide blocking async API to get the latest value. It could still expose an event for other consumers though. The consumer will need to actively poll for changes (in a busy loop of some sorts).
You could also have a producer with an event as in "Consume all", but then create another component that listens to any given event, keeps the latest value and makes it available via a blocking async call to any other client.
Here some advantages/disadvantages I can think of:
In (1) the producer is very simple; the consumer is harder to write
In (2) the producer needs to do a bit more work, but the consumer is simple
In (3), you are adding another layer, but in a fairly reusable way.
I would probably go with either (2) (if I only need this for one data source) or with (3) after checking that it does not affect the performance.
As for (3), the sketch of what I was thinking would look something like this:
type KeepLastMessage<'T> =
| Update of 'T
| Get of AsyncReplyChannel<'T>
type KeepLast<'T>(initial:'T, event:IObservable<'T>) =
let agent = MailboxProcessor.Start(fun inbox ->
let rec loop last = async {
let! msg = inbox.Receive()
match msg with
| Update last -> return! loop last
| Get ch -> ch.Reply(last); return! loop last }
loop initial)
member x.AsyncGet() = agent.PostAndAsyncReply(Get)
HOW I WISH I HAD PHRASED MY QUESTION TO BEGIN WITH
Take a table with 26 keys, a-z and let them have integer values.
Create a process, Ouch, that does two things over and over again
In one transaction, write random values for a, b, and c such that those values always sum to 10
In another transaction, read the values for a, b and c and complain if their values do not sum to 10
If you spin-up even a few of these processes you will see that very quickly a, b and c are in a state where their values do not sum to 10. I believe there is no way to ask mnesia to "lock these 3 records before starting the writes (or reads)", one can only have mnesia lock the records as it gets to them (so to speak) which allows for the set of records' values to violate my "must sum to 10" constraint.
If I am right, solutions to this problem include
lock the entire table before writing (or reading) the set of 3 records -- I hate to lock whole table for 3 recs,
Create a process that keeps track of who is reading or writing which keys and protects bulk operations from anyone else writing or reading until the operation is completed. Of course I would have to make sure that all processes made use of this... crap, I guess this means writing my own AccessMod as the fourth parameter to activity/4 which seems like a non-trivial exercise
Some other thing that I am not smart enough to figure out.
thoughts?
Ok, I'm an ambitious Erlang newbee, so sorry if this is a dumb question, but
I am building an application-specific, in-memory distributed cache and I need to be able to write sets of Key, Value pairs in one transaction and also retrieve sets of values in one transaction. In other words I need to
1) Write 40 key,value pairs into the cache and ensure that no one else can read or write any of these 40 keys during this multi-key write operation; and,
2) Read 40 keys in one operation and get back 40 values knowing that all 40 values have been unchanged from the moment that this read operation started until it ended.
The only way I can think of doing this is to lock the entire table at the beginning of the fetch_keylist([ListOfKeys]) or at the beginning of the write_keylist([KeyValuePairs], but I don't want to do this because I have many processes simultaneously doing their own multi_key reads and writes and I don't want to lock the entire table any time any process needs to read/write a relatively small subset of records.
Help?
Trying to be more clear: I do not think this is just about using vanilla transactions
I think I am asking a more subtle question than this. Imagine that I have a process that, within a transaction, iterates through 10 records, locking them as it goes. Now imagine this process starts but before it iterates to the 3rd record ANOTHER process updates the 3rd record. This will be just fine as far as transactions go because the first process hadn't locked the 3rd record (yet) and the OTHER process modified it and released it before the first process got to it. What I want is to be guaranteed that once my first process starts that no other process can touch the 10 records until the first process is done with them.
PROBLEM SOLVED - I'M AN IDIOT... I guess...
Thank you all for your patients, especially Hynek -Pichi- Vychodil!
I prepared my test code to show the problem, and I could in fact reproduce the problem. I then simplified the code for readability and the problem went away. I was not able to again reproduce the problem. This is both embarrassing and mysterious to me since I had this problem for days. Also mnesia never complained that I was executing operations outside of a transaction and I have no dirty transactions anywhere in my code, I have no idea how I was able to introduce this bug into my code!
I have pounded the notion of Isolation into my head and will not doubt that it exists again.
Thanks for the education.
Actually, turns out the problem was using try/catch around mnesia operations within a transaction. See here for more.
Mnesia transaction will do exactly this thing for you. It is what is transaction for unless you do dirty operations. So just place your write and read operations to one transaction a mnesia will do rest. All operations in one transaction is done as one atomic operation. Mnesia transaction isolation level is what is sometimes known as "serializable" i.e. strongest isolation level.
Edit:
It seems you missed one important point about concurrent processes in Erlang. (To be fair it is not only true in Erlang but in any truly concurrent environment and when someone arguing else it is not really concurrent environment.) You can't distinguish which action happen first and which happen second unless you do some synchronization. Only way you can do this synchronization is using message passing. You have guaranteed only one thing about messages in Erlang, ordering of messages sent from one process to other process. It means when you send two messages M1 and M2 from process A to process B they arrives in same order. But if you send message M1 from A to B and message M2 from C to B they can arrive in any order. Simply because how you can even tell which message you sent first? It is even worse if you send message M1 from A to B and then M2 from A to C and when M2 arrives to C send M3 from C to B you don't have guarantied that M1 arrives to B before M3. Even it will happen in one VM in current implementation. But you can't rely on it because it is not guaranteed and can change even in next version of VM just due message passing implementation between different schedulers.
It illustrates problems of event ordering in concurrent processes. Now back to the mnesia transaction. Mnesia transaction have to be side effect free fun. It means there may not be any message sending outside from transaction. So you can't tell which transaction starts first and when starts. Only thing you can tell if transaction succeed and they order you can only determine by its effect. When you consider this your subtle clarification makes no sense. One transaction will read all keys in atomic operation even it is implemented as reading one key by one in transaction implementation and your write operation will be also performed as atomic operation. You can't tell if write to 4th key in second transaction was happen after you read 1st key in first transaction because there it is not observable from outside. Both transaction will be performed in particular order as separate atomic operation. From outside point of view all keys will be read in same point of time and it is work of mnesia to force it. If you send message from inside of transaction you violate mnesia transaction property and you can't be surprised it will behave strange. To be concrete, this message can be send many times.
Edit2:
If you spin-up even a few of these processes you will see that very
quickly a, b and c are in a state where their values do not sum to 10.
I'm curious why you think it would happen or you tested it? Show me your test case and I will show mine:
-module(transactions).
-export([start/2, sum/0, write/0]).
start(W, R) ->
mnesia:start(),
{atomic, ok} = mnesia:create_table(test, [{ram_copies,[node()]}]),
F = fun() ->
ok = mnesia:write({test, a, 10}),
[ ok = mnesia:write({test, X, 0}) || X <-
[b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z]],
ok
end,
{atomic, ok} = mnesia:transaction(F),
F2 = fun() ->
S = self(),
erlang:send_after(1000, S, show),
[ spawn_link(fun() -> writer(S) end) || _ <- lists:seq(1,W) ],
[ spawn_link(fun() -> reader(S) end) || _ <- lists:seq(1,R) ],
collect(0,0)
end,
spawn(F2).
collect(R, W) ->
receive
read -> collect(R+1, W);
write -> collect(R, W+1);
show ->
erlang:send_after(1000, self(), show),
io:format("R: ~p, W: ~p~n", [R,W]),
collect(R, W)
end.
keys() ->
element(random:uniform(6),
{[a,b,c],[a,c,b],[b,a,c],[b,c,a],[c,a,b],[c,b,a]}).
sum() ->
F = fun() ->
lists:sum([X || K<-keys(), {test, _, X} <- mnesia:read(test, K)])
end,
{atomic, S} = mnesia:transaction(F),
S.
write() ->
F = fun() ->
[A, B ] = L = [ random:uniform(10) || _ <- [1,2] ],
[ok = mnesia:write({test, K, V}) || {K, V} <- lists:zip(keys(),
[10-A-B|L])],
ok
end,
{atomic, ok} = mnesia:transaction(F),
ok.
reader(P) ->
case sum() of
10 ->
P ! read,
reader(P);
_ ->
io:format("ERROR!!!~n",[]),
exit(error)
end.
writer(P) ->
ok = write(),
P ! write,
writer(P).
If it would not work it would be really serious problem. There are serious applications including payment systems which rely on it. If you have test case which shows it is broken, please report bug at erlang-bugs#erlang.org
Have you tried mnesia Events ? You can have the reader subscribe to mnesia's Table Events especially write events so as not to interrupt the process doing the writing. In this way, mnesia just keeps sending a copy of what has been written in real-time to the other process which checks what the values are at any one time. take a look at this:
subscriber()->
mnesia:subscribe({table,YOUR_TABLE_NAME,simple}),
%% OR mnesia:subscribe({table,YOUR_TABLE_NAME,detailed}),
wait_events().
wait_events()->
receive
%% For simple events
{mnesia_table_event,{write, NewRecord, ActivityId}} ->
%% Analyse the written record as you wish
wait_events();
%% For detailed events
{mnesia_table_event,{write, YOUR_TABLE, NewRecord, [OldRecords], ActivityId}} ->
%% Analyse the written record as you wish
wait_events();
_Any -> wait_events()
end.
Now you spawn your analyser as a process like this:
spawn(?MODULE,subscriber,[]).
This makes the whole process to run without any process being blocked, mnesia needs not lock any tabel or record because now what you have is a writer process and an analyser process. The whole thing will run in real-time. Remember that there are many other events that you can make use of if you wish by pattern matching them in the subscriber wait_events() receive body.
Its possible to build a heavy duty gen_server or complete application intended for reception and analysis of all your mnesia events. Its usually better to have one capable subscriber than many failing event subscribers. If i have understood you question well, this unblocking solution fits your requirements.
mnesia:read/3 with write locks seems to be suffient.
Mnesia's transaction is implemented by read-write lock and locks are well-formed (holding lock untill the end of transaction). So the isolation level is serializable.
The granularity of locks are per record as long as you access by primary key.
How come that Solution 2 is more efficient than Solution 1?
(The time is the average of 100 runs, and the total folders they go through is 13217)
// Solution 1 (2608,9ms)
let rec folderCollector path =
async { let! dirs = Directory.AsyncGetDirectories path
do! [for z in dirs -> folderCollector z]
|> Async.Parallel |> Async.Ignore }
// Solution 2 (2510,9ms)
let rec folderCollector path =
let dirs = Directory.GetDirectories path
for z in dirs do folderCollector z
I would have thought that Solution 1 would be faster because it's async, and that I run it in Parallel. What am I'm missing?
As Daniel and Brian already clearly explained, your solution is probably creating too many short-lived asynchronous computations (so the overhead is more than the gains from parallelism). The AsyncGetDirectories operation also probably isn't really non-blocking as it is not doing much work. I don't see a truly async version of this operation anywhere - how is it defined?
Anyway, using the ordinary GetDirectories, I tried the following version (which creates only a small number of parallel asyncs):
// Synchronous version
let rec folderCollectorSync path =
let dirs = Directory.GetDirectories path
for z in dirs do folderCollectorSync z
// Asynchronous version that uses synchronous when 'nesting <= 0'
let rec folderCollector path nesting =
async { if nesting <= 0 then return folderCollectorSync path
else let dirs = Directory.GetDirectories path
do! [for z in dirs -> folderCollector z (nesting - 1) ]
|> Async.Parallel |> Async.Ignore }
Calling a simple synchronous version after certain number of recursive calls is a common trick - it is used when parallelizing any tree-like structure that is very deep. Using folderCollector path 2, this will start only tens of parallel tasks (as opposed to thousands), so it will be more efficient.
On a sample directory I used (with 4800 sub-dirs and 27000 files), I get:
folderCollectorSync path takes 1 second
folderCollector path 2 takes takes 600ms (result is similar for any nesting between 1 and 4)
From the comments:
Your function incurs the cost of async without any of the benefits because
you're creating too many asyncs for the short amount of work to be done
your function is not CPU, but rather IO, bound
I expect for a problem like this, you may have the best results if at the top-level you do async/parallel work, but then have the sub-calls be sync. (Or if the trees are very deep, maybe have the first two levels be async, and then sync after that.)
The keys are load-balancing and granularity. Too tiny a piece of work, and the overhead of async outweighs the benefits of parallelism. So you want big enough chunks of work to leverage parallel and overcome the overheads. But if the work pieces are too large and unbalanced (e.g. one top-level dir has 10000 files, and 3 other top-level dirs have 1000 each), then you also suffer because one guy is busy while the rest finish quickly, and you don't maximize parallelism.
If you can estimate the work for each sub-tree beforehand, you can do even better scheduling.
Apparently, your code is IO-bound. Keep in mind how HDDs work. When u use Async to do multiple read, the reading heads of the HDD have to jump back and forth to serve different read commands at the same time, which introduces latency. This will likely become much worse if the data on disk is heavily fragmented.
I have some stuff written in c# that executes concurrent code, making heavy use of the Task Parallel Library (Task and Future continuation chains).
I'm now porting this to F# and am trying to figure out the pros and cons of using F# Async workflows vs. the constructs in the TPL. I'm leaning towards TPL, but I think it could be done either way.
Does anyone have tips and wisdom about writing concurrent programs in F# to share?
The name pretty much sums up the difference: asynchronous programming vs. parallel programming. But in F# you can mix and match.
F# Asynchronous Workflows
F# async workflows are helpful when you want to have code execute asynchronously, that is starting a task and not waiting around for the final result. The most common usage of this is IO operations. Having your thread sit there in an idle loop waiting for your hard disk to finish writing wastes resources.
If you began the write operation asynchronously you can suspend the thread and have it woken up later by a hardware interrupt.
Task Parallel Library
The Task Parallel Library in .NET 4.0 abstracts the notion of a task - such as decoding an MP3, or reading some results from a database. In these situations you actually want the result of the computation and at some point in time later are waiting for the operation's result. (By accessing the .Result property.)
You can easily mix and match these concepts. Such as doing all of your IO operations in a TPL Task object. To the programmer you have abstracted the need to 'deal with' that extra thread, but under the covers you're wasting resources.
Like wise you can create a series of F# async workflows and run them in parallel (Async.Parallel) but then you need to wait for the final result (Async.RunSynchronously). This frees you from needing to explicitly start all the tasks, but really you are just performing the computation in parallel.
In my experience I find that the TPL is more useful because usually I want to execute N operations in parallel. However, F# async workflows are ideal when there is something that is going on 'behind the scenes' such as a Reactive Agent or Mailbox type thing. (You send something a message, it processes it and sends it back.)
Hope that helps.
In 4.0 I would say:
If your function is sequential, use Async workflows. They simply read better.
Use the TPL for everything else.
It's also possible to mix and match. They've added support for running a workflow as a task and creating tasks that follow the async Begin/End pattern using TaskFactory.FromAsync, the TPL equivalent of Async.FromBeginEnd or Async.BuildPrimitive.
let func() =
let file = File.OpenRead("foo")
let buffer = Array.zeroCreate 1024
let task1 = Task.Factory.FromAsync(file.BeginRead(buffer, 0, buffer.Length, null, null), file.EndRead)
task1.Start()
let task2 = Async.StartAsTask(file.AsyncRead(1024))
printfn "%d" task2.Result.Length
It's also worth noting that both the Async Workflows runtime and the TPL are going to create an extra kernel primitive (an Event) and use WaitForMultipleObjects to track I/O completion, rather than using completion ports and callbacks. This is undesirable in some applications.