I am using Akka.net (version="1.0.8") .I have two Actors one is reading Input from database and other is accepting that input and contains some processing logic. Actor “A” is Input actor and “B” is actor for processing input.
When we send around 50 000 messages (size around 30kb each) to “actor B”
for ex.
[1..10000000]
|> List.iter(fun msg ->
actRef.Tell(JsonConvert.SerializeObject(ProcessData (MessageData))))
I get below errors with some dead letters:
“[WARNING][4/29/2016 10:08:18 AM][Thread 0009]
[[akka://Pankaj/system/endpointManager/reliableEndpointWriterakka.tcp%3a%2f%2fRemoteFSharp%40***.**.**.**%3a8788-1]]
Association with remote system akka.tcp://RemoteFSharp#***.**.**.**:8788 has failed;
address is now gated for 5000 ms.
Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated …
[INFO]
[4/29/2016 10:08:18 AM][Thread 0021][akka://Pankaj/deadLetters]
Message String from akka://Pankaj/deadLetters to akka://Pankaj/deadLetters was not delivered.
1 dead letters encountered… “”
But when I use Ask method it works fine.
[1..1000000]
|> List.iter(fun msg ->
actRef.Ask(JsonConvert.SerializeObject(ProcessData (MessageData))).Wait())
If I put a delay of 10ms before every Tell call then I get some error after processing 200 000 records.
Wwhat exactly I am missing here?
What is the dead letter exception and how to fix it?
Related
I have a tool that is using a Telegram chatbot to interact with its users.
Telegram limits the call rate, so I use a queue system that gets flushed at regular intervals.
So, the current code is very basic:
// flush the message queue
let flushMessageQueue() =
if not messageQueue.IsEmpty then <- messageQueue is a ConcurrentQueue
// get all the messages
let messages =
messageQueue
|> Seq.unfold(fun q ->
match q.TryDequeue () with
| true, m -> Some (m, q)
| _ -> None)
// put all the messages in a single string
let messagesString = String.Join("\n", messages)
// send the data
client.SendTextMessageAsync(chatId, messagesString, ParseMode.Markdown)
|> Async.AwaitTask
|> Async.RunSynchronously
|> ignore
this is called at regular interval, while the write is:
// broadcast message
let broadcastMessage message =
messageQueue.Enqueue(message)
printfn "%s" (message.Replace ("```", String.Empty))
But as messages became more complex, two problems came at once:
Part of the output is formatted text with simple markdown:
Some blocks of lines are wrapped between ``` sections
There are some ``` sections as well inside some lines
The text is UTF-8 and uses a bunch of symbols
Some example of text may be:
```
this is a group of lines
with one, or many many lines
```
and sometimes there are things ```like this``` as well
And... I found out that Telegram limits message size to 4kb as well
So, I thought of two things:
I can maintain a state with the open / close ``` and pull from a queue, wrap each line in triple back ticks based on the state and push into another queue that will be used to make the 4kb block.
I can keep taking messages from the re-formatted queue and aggregate them until I reach 4kb, or the end of the queue and loop around.
Is there an elegant way to do this in F#?
I remember seeing a snippet where a collection function was used to aggregate data until a certain size but it looked very inefficient as it was making a collection of line1, line1+line2, line1+line2+line3... and then picking the one with the right size.
There's variable in my module, and there's receive method to renew variable value. And multiple process are calling this method simultaneously. I need lock this variable when one process is modifying it. Sample as below
mytest.erl
%%%-------------------------------------------------------------------
-module(mytest).
%% API
-export([start_link/0,display/1,callDisplay/2]).
start_link()->
Pid=spawn(mytest,display,["Hello"]),
Pid.
display(Val) ->
io:format("It started: ~p",[Val]),
NextVal=
receive
{call,Msg}->
NewVal=Val++" "++Msg++" ",
NewVal;
stop->
true
end,
display(NextVal).
callDisplay(Pid,Val)->
Pid!{call,Val}.
Start it
Pid=mytest:start_link().
Two process are calling it in the same time
P1=spawn(mytest,callDisplay,[Pid,"Walter"]),
P2=spawn(mytest,callDisplay,[Pid,"Dave"]).
I hope it can add "Walter", "Dave" one by one like "Hello Walter Dave", however, when there're too many of them running together, some Names(Walter, Dave, etc) will be override.
Because when P1, P2 started the same time, Val both are "Hello". P1 add "Walter" to become "Hello Walter", P2 add "Dave" to become "Hello Dave". P1 saved it firstly to NextVal as "Hello Walter", then P2 saved it to NextVal as "Hello Dave", so result will be "Hello Dave". "Hello Walter" is replaced by "Hello Dave", and "Walter" lost forever.
Is there any way I can lock "Val", so when we add "Walter", "Dave" will waiting till Value setting is done?
Even though it's an old question but it's worth explaining.
From what you said and if I'm correct,
you expect to see
"Hello Walter", and "Hello Dave". However, you're seeing successive names been appended to the former as, "Hello Walter Dave.."
This behavior is normal and to see that let look briefly at Erlang memory model. Erlang process memory is divided into three main parts:
Process Control Block(PCB):
This hold the process pid, registered name,table,states and pointers to messages in the it's queue.
Stack:
This hold function parameters, local variables and function return address.
Private Heap: This hold incoming message compound data like tuple, list and binary(not larger than 64 bytes).
All data in these memory belong to and are private to the owning process.
Stage1:
When Pid=spawn(mytest,display,["Hello"]) is called, the server process is created, then the display function with "Hello" passed as argument is called. Since display/1 is executed in the serve process, the "Hello" argument lives in the server's process stack. Execution of display/1 continues until it reaches the receive clause then block and await message matching your format.
Stage 2:
Now P1 starts, it executes ServerPid ! {call, "Walter"}, then P2 executes ServerPid ! {call, "Dave"}. In both cases, erlang makes a copy of the message and send it to the server's process mailbox (Private Heap). This copied message in the mailbox belongs to the server process not the client's.
Now, when {call, "Walter"} is matched, Msg get bound to "Walter".
From stage1, we know Val is bounded to "Hello", Newval then get bounded to "Val ++ " " ++ Msg" = "Hello Walter".
At this point, P2's message, {call, "Dave"}, is still in the server's mailbox awaiting the next receive clause which will happen in the next recursive call to display/1. NextVal get bound to NewVal and the recursive call to dispaly/1 with "Hello Walter" passed as argument is made. This gives the first print "Hello Walter " which now also lives in the server's process stack.
Now when the receive clause is reach again, P2's message {call, "Dave"} is matched.
Now NewVal and NextVal get bound to "Hello Walter" ++ " " ++ "Dave" = "Hello Walter Dave". This get passed as argument to display/1 as the new Val to print Hello Walter Dave. In a nutshell, this variable is updated on every server loop. It serves the same purpose as the State term in gen_server behavior. In your case, successive client calls just appends the message to this serve state variable. Now to your question,
Is there any way I can lock Val, so when we add "Walter", "Dave" will waiting till Value setting is done?
No. Not by locking. Erlang does not work this way.
There are no process locking constructs as it does not need one.
Data(Variables) are always immutable and private(except large binaries which stays in the Shared Heap) to the process that created it.
Also, it's not the actual message you used in the Pid ! Msg construct that is process by the receiving process. It's it copy. The Val parameter in yourdisplay/1 function is private and belongs to the server process because it lives in it stack memory as every call to display/1 is made by the server process itself. So there is no way any other process can lock not even see that variable.
Yes. By sequential message processing
This is exactly what the server process is doing. Polling one message a time from it queue. When {call, "Walter"} was taken, {call, "Dave"} was waiting in the queue. The reason why you see unexpected greeting is because the you change the server state, the display/1 parameter for the next display/1 call which process {call, "Dave"}
I am trying to build a very simple barrier-synchronization server, where the server is initially fed a number of processes that will be communicating with it. When a process is done, it receives a message with that process' Pid, and it keeps a list of every process to do so. When the barrier reaches zero (all processes have sent messages), my server needs to send a message to each of these (I am using [Pid | ProcList] as my list of pids).
I have tried using a helper function to no avail, list comprehensions keep me in an infinite loop, and as such I am looking into how to use lists:foreach to take care of this.
I am fairly new to functional programming, but from what I understand, this foreach needs to take in the list as well as a lambda-calculus function to send a message to each node in the list. Due to the infix nature of "!", I have yet to find a way to do this without causing syntax errors.
How you've made infinite loop in list comprehension? I must say, that's quite challenging. Try this:
Message = % broadcast message goes here
ListOfPids = % list of recipients
[Pid ! Message || Pid <- ListOfPids].
If you want to use foreach, than it takes one argument function as first argument, so need to wrap send first, as it is two argument function.
Message = % broadcast message goes here
ListOfPids = % list of recipients
Fun = fun (Pid) -> Pid ! Message end,
lists:foreach(Fun, ListOfPids).
new to Erlang and just having a bit of trouble getting my head around the new paradigm!
OK, so I have this internal function within an OTP gen_server:
my_func() ->
Result = ibrowse:send_req(?ROOTPAGE,[{"User-Agent",?USERAGENT}],get),
case Result of
{ok, "200", _, Xml} -> %<<do some stuff that won't interest you>>
,ok;
{error,{conn_failed,{error,nxdomain}}} -> <<what the heck do I do here?>>
end.
If I leave out the case for handling the connection failed then I get an exit signal propagated to the supervisor and it gets shut down along with the server.
What I want to happen (at least I think this is what I want to happen) is that on a connection failure I'd like to pause and then retry send_req say 10 times and at that point the supervisor can fail.
If I do something ugly like this...
{error,{conn_failed,{error,nxdomain}}} -> stop()
it shuts down the server process and yes, I get to use my (try 10 times within 10 seconds) restart strategy until it fails, which is also the desired result however the return value from the server to the supervisor is 'ok' when I would really like to return {error,error_but_please_dont_fall_over_mr_supervisor}.
I strongly suspect in this scenario that I'm supposed to handle all the business stuff like retrying failed connections within 'my_func' rather than trying to get the process to stop and then having the supervisor restart it in order to try it again.
Question: what is the 'Erlang way' in this scenario ?
I'm new to erlang too.. but how about something like this?
The code is long just because of the comments. My solution (I hope I've understood correctly your question) will receive the maximum number of attempts and then do a tail-recursive call, that will stop by pattern-matching the max number of attempts with the next one. Uses timer:sleep() to pause to simplify things.
%% #doc Instead of having my_func/0, you have
%% my_func/1, so we can "inject" the max number of
%% attempts. This one will call your tail-recursive
%% one
my_func(MaxAttempts) ->
my_func(MaxAttempts, 0).
%% #doc This one will match when the maximum number
%% of attempts have been reached, terminates the
%% tail recursion.
my_func(MaxAttempts, MaxAttempts) ->
{error, too_many_retries};
%% #doc Here's where we do the work, by having
%% an accumulator that is incremented with each
%% failed attempt.
my_func(MaxAttempts, Counter) ->
io:format("Attempt #~B~n", [Counter]),
% Simulating the error here.
Result = {error,{conn_failed,{error,nxdomain}}},
case Result of
{ok, "200", _, Xml} -> ok;
{error,{conn_failed,{error,nxdomain}}} ->
% Wait, then tail-recursive call.
timer:sleep(1000),
my_func(MaxAttempts, Counter + 1)
end.
EDIT: If this code is in a process which is supervised, I think it's better to have a simple_one_for_one, where you can add dinamically whatever workers you need, this is to avoid delaying initialization due to timeouts (in a one_for_one the workers are started in order, and having sleep's at that point will stop the other processes from initializing).
EDIT2: Added an example shell execution:
1> c(my_func).
my_func.erl:26: Warning: variable 'Xml' is unused
{ok,my_func}
2> my_func:my_func(5).
Attempt #0
Attempt #1
Attempt #2
Attempt #3
Attempt #4
{error,too_many_retries}
With 1s delays between each printed message.
Can I Nest receive {tcp, Socket, Bin} -> calls? For example I have a top level loop called Loop, which upon receipt of tcp data calls a function, parse_header, to parse header data (an integer which indicates the kind of data to follow and thus its size), after that I need to receive the entire payload before moving on. I might only receive 4 bytes when I need a full 20 bytes and would like to call receive in a separate function called parse_payload. So the call chain would look like loop->parse_header->parse_payload and I would like parse_payload to call receive {tcp, Socket, Bin} ->. I don't know if this ok or if I'm completely going to mess things up and can only do it in the Loop function. Can someone enlighten me? If I am allowed to do this is am I violating some sort of best practice?
Maybe you can check the sample code for "erlang programming".
The download page is Erlang Programming Source Code
In file socket_examples.erl, please check "receive_data" function.
For perse message, I think you should determine how to seperate messages one by one (fixed length or with termination byte), then parse message's header, and payload.
receive_data(Socket, SoFar) ->
receive
{tcp,Socket,Bin} -> %% (3)
receive_data(Socket, [Bin|SoFar]);
{tcp_closed,Socket} -> %% (4)
list_to_binary(reverse(SoFar)) %% (5)
end.
You can also set a gen_tcp socket in passive mode. This way, the owning process won't receive the input by messages but has to fetch it using gen_tcp:recv(Socket, ByteCount) which returns either {ok, Input} or {error, Reason}. As this methods waits infinitely for the bytes you might want to add a timeout using gen_tcp:recv/3. (Erlang documentation of gen_tcp:recv)
While at first glance it might seem the process is now completely unable to react to messages sent to it, there is the following workaround improving the situation a bit:
f1(X) ->
receive
message1 ->
... do something ...,
f1(X);
message2 ->
... do something ...,
f1(X)
after 0 %timeout in ms
{ok, Input} = gen_tcp:recv(Socket, ByteCount, Timeout),
... do something ... % maybe call some times gen_tcp:recv again
f1(X)
end.
If you don't add a timeout to gen_tcp:recv here, other processes could wait ages for f1 to handle their messages.