KafkaReceiver.create(options).receive().groupBy(i-> Integer.parseInt(i.value())%2 ==0? "Even":"Odd").doOnNext(x ->
System.out.println("Inside map the thread is " + Thread.currentThread().getName()))
.flatMap(partitionFlux ->
partitionFlux.publishOn(scheduler)
.concatMap(m -> writeBigQuery(m)
.doOnSuccess(r -> m.receiverOffset().commit())
.doOnError(e -> System.out.println("ErrorMono"+e.getStackTrace()))
.retry(3))).subscribe
(System.out::println);
I want to write a method which gives me a tuple of FLUX(one with odd stream and another with even stream
Signature of the method looks like this: private Tuple2<Flux<ReceiverRecord<Integer,String>>, Flux<ReceiverRecord<Integer,String>>> getTuple
(Flux<GroupedFlux<String,ReceiverRecord<Integer,String>>> flux)
Can someone help with the implementation of this?
Related
I am struggling with how to set different cache response headers based on whether the result is an Ok or an Error. My code is something like the following (but with other types in the result):
let resultToJson (result:Result<'a,string>) : HttpHandler =
match result with
| Ok o -> Successful.ok (json o)
| Error s -> ServerErrors.internalError (text s)
I can add the headers by doing something like the following:
let resultToJson (result:Result<'a,string>) : HttpHandler =
fun (next : HttpFunc) (ctx : HttpContext) ->
let response =
let headers = ctx.Response.Headers
match result with
| Ok o ->
headers.Add("Cache-Control", new StringValues("public, max-age=10, stale-while-revalidate=2"))
headers.Add("Vary", new StringValues("Origin"))
Successful.ok (json o)
| Error s ->
headers.Add("Cache-Control", new StringValues("no-cache"))
ServerErrors.internalError (text s)
response next ctx
But this does not feel right. I would like to use the standard HttpHandlers from the ResponseCaching module to set the right cache headers:
publicResponseCaching 10 (Some "Origin") // For Ok: Add 10 sec public cache, Vary by Origin
noResponseCaching // For Error: no caching
How do I achieve this?
The response cache handler is supposed to be piped into an normal pipeline. Your choice between Ok and Error is a choose function, so you can use a choose that takes a list of handlers that can be attempted. To reject a path, just return a task { return None }, to move forward, it's next ctx.
If you want to keep all the logic in one controller, like you have now, just keep your match and pipe your json/text response into one of the caching handlers.
let fn = json o >=> publicResponseCaching 30 None) in fn next ctx
if it's nested inside a hander, instead of in a pipeline, you have to apply the next & ctx
I found the solution to my problem.
Yes, I can chain the HttpHandlers like Gerard and Honza Brestan mentioned, using the fish operator (>=>). The reason I could not make that work in the first place was that I also had created a fish operator for the Result type in an opened module. Basically I had created proper fish soup
As soon as I refactored my code so that the module containing the Result fish operator was not open in this scope, everything worked fine as expected.
Another point to remember is that response caching needs to be called before the finalizing HttpHandler, otherwise it will not be called:
// Simplified code
let resultToJson =
function
| Ok o -> publicResponseCaching 10 (Some "Origin") >=> Successful.ok(json o)
| Error e -> noResponseCaching >=> ServerErrors.internalError(text e)
E.g. suppose I have a list that looks something roughly like this:
Handlers = [{foo, FooHandler}, {bar, BarHandler} | Etc()]
The best that I can come up with is this:
receive
Message ->
Handler = find_matching_handler(Message, Handlers),
Handler(Message)
end
The problem with this is that if Message does not match anything in Handlers, it's too late: I've taken it out of the mailbox.
I guess if there's a way to put a message back into the mailbox (into the save queue) without reordering, then that would take care of it. Simply resending to self() would reorder. It would also not restart the receive, and even if it did, you might get stuck in a spin loop until a message of interest arrives. Is there a way to put a message into the mailbox's save queue?
Another near solution that I thought of was to use match guard, but IIUC, you can only use BIFs in guards, which seems to preclude using find_matching_handler (unless there is a BIF for that).
Another near solution: map matching:
receive
M when Handlers#{M := Handler} -> Handler(M) % booyah?
end
Alas, I have not found an incantation that satisfies Erlang...
Match on the message:
loop() ->
receive
{foo, Data} ->
handle_foo(Data),
loop();
{bar, Data} ->
handle_bar(Data),
loop()
end.
This is the basic way of distinguishing between message forms.
You can also be less direct and match in a function head you pass all messages to:
loop() ->
receive
Message ->
handle_message(Message),
loop()
end.
handle_message({foo, Data}) ->
foo(Data),
ok;
handle_message({bar, Data}) ->
bar(Data),
ok.
A combination of the first and second forms is sort of the way gen_server type callback modules are structured in OTP. The message handlers receive a slightly more complex set of arguments and exist in their own module (the part you write), and the actual receive occurs in the generic gen_server module.
You can use a selective receive pattern to periodcally scan the mailbox for handler messages. Something like this:
check_msg_handlers(Handlers) ->
[check_handler(X) || X <- Handlers],
timer:sleep(500),
check_msg_handlers(Handlers).
check_handler(Handler) ->
receive
{_Handler={M,F}, Msg} ->
M:F(Msg)
after
0 ->
no_msg
end.
Note the receive X -> Y after -> N no_msg end, this is the selective receive. When using a timeout of N=0 it effectively becomes a scan of the mailbox to see if the X message is present or not, i.e. it becomes a non-blocking receive. The order of the messages is preserved after the scan as required in your case.
The LYSE chapter More On Multiprocessing has a section on selective receives that is very good.
Looking at:
member this.PostAndReply : (AsyncReplyChannel<'Reply> -> 'Msg) * ?int -> 'Reply
I can't figure out why the signature looks so counter-intuitive to me. What we want to do is posting a message to an agent, and wait for a reply.
Why do we have to give him a weird function as a 'message'?
See again this MSDN snippet:
let rec loop() =
printf "> "
let input = Console.ReadLine()
printThreadId("Console loop")
let reply = agent.PostAndReply(fun replyChannel -> input, replyChannel)
if (reply <> "Stopping.") then
printfn "Reply: %s" reply
loop()
else
()
loop()
I'd rather prefer something like this:
member this.PostAndReply : 'Msg * ?int -> 'Reply
Thanks
This type signature looks pretty confusing when you see it for the first time, but it does make sense.
The F# library design
The idea behind the is that when you call PostAndReply you need to give it a function that:
constructs a message of type 'Msg (to be sent to the agent)
after the F# runtime builds a channel for sending messages back to the caller (channels are represented as values of type AsyncReplyChannel<'Reply>).
The message that you construct needs to contain the reply channel, but the F# library does not know how you want to represent your messages (and so it does not know how you want to store the reply channel in the message). As a result, the library asks you to write a function that will construct the message for the agent after the system constructs the channel.
Your alternative suggestion
The problem with your suggestion is that if PostAndReply had a type 'Msg -> 'Reply, the message that the agent receives after it calls Receive would be of the following type:
'Msg * AsyncReplyChannel<'Reply>
... so every message received to the agent would also have to carry a channel for sending replies back. However, you probably don't want to send a reply back for every received message, so this wouldn't really work. Maybe you could use something like:
'Msg * option<AsyncReplyChannel<'Reply>>
... but that's just getting more complicated (and it still isn't quite right, because you can only reply to some messages from 'Msg, but not to all of them).
I am trying to use Erlang/ets to store/update various informations by pattern matching received data. Here is the code
start() ->
S = ets:new(test,[]),
register(proc,spawn(fun() -> receive_data(S) end)).
receive_data(S) ->
receive
{see,A} -> ets:insert(S,{cycle,A}) ;
[[f,c],Fcd,Fca,_,_] -> ets:insert(S,{flag_c,Fcd,Fca});
[[b],Bd,Ba,_,_] -> ets:insert(S,{ball,Bd,Ba})
end,
receive_data(S).
Here A is cycle number, [f,c] is center flag , [b] is ball and Fcd,Fca, Bd, Ba are directions and angle of flag and ball from player.
Sender process is sending these informations. Here, pattern matching is working correctly which I checked by printing values of A, Fcd,Fca..etc. I believe there is something wrong with the use of Erlang/ets.
When I run this code I get error like this
Error in process <0.48.0> with exit value: {badarg,[{ets,insert,[16400,{cycle,7}]},{single,receive_data,1}]
Can anybody tell me what's wrong with this code and how to correct this problem?
The problem is that the owner of the ets-table is the process running the start/1 function and the default behavior for ets is to only allow the owner to write and other processes to read, aka protected. Two solutions:
Create the ets table as public
S = ets:new(test,[public]).
Set the owner to your newly created process
Pid = spawn(fun() -> receive_data(S) end,
ets:give_away(test, Pid, gift)
register(proc,Pid)
Documentation for give_away/3
I need to debug some module in foreign system. The module has public function foo() - how can I know place (module and function name)
from which foo() given module was called? I mean stack of calls.
I cannot stop system, all work I can do by reload this module (but with some debug info).
-module(given).
-export(foo/0).
foo() ->
%% here is my debug - and
%% i need here(!) known about unknown_module:unknown_foo!
ok.
---
-module(unknown_module).
..
unknown_foo() ->
given:foo(). %% see above
Here's a simple trick:
Trace = try throw(42) catch 42 -> erlang:get_stacktrace() end,
erlang:display(Trace)
This might work:
where_am_i() ->
try throw(a)
catch throw:a:Stacktrace ->
Stacktrace
end.
Except that it doesn't work for tail calls. For example, given these two functions:
foo() ->
where_am_i().
bar() ->
X = where_am_i(),
{ok, X}.
I get these results:
4> foo:foo().
[{foo,where_am_i,0},
{erl_eval,do_apply,5},
{shell,exprs,6},
{shell,eval_exprs,6},
{shell,eval_loop,3}]
5> foo:bar().
{ok,[{foo,where_am_i,0},
{foo,bar,0},
{erl_eval,do_apply,5},
{shell,exprs,6},
{shell,eval_exprs,6},
{shell,eval_loop,3}]}
That is, I can only see bar, since foo's call frame has been left already when where_am_i is called.
io:format("~s~n", [element(2, process_info(self(), backtrace))]).
self() can be replaced by any other pid (rpc:pinfo should even work with remote procs). This helps if you cannot even modify the source or beam.
Here is my code for doing this:
format_stack_entry(S) ->
{Module,Fun,Arity,[{file,File},{line,Line}]}=S,
io_lib:format("{~p,~p,~p,[{file,~p},{line,~p]}",[Module,Fun,Arity,File,Line]).
stacktop([Top|_]) ->
Top.
ancestor(N) ->
{_,Stacktrace}=erlang:process_info(self(),current_stacktrace),
ancestor(N+1,Stacktrace).
ancestor(1,S) ->
format_stack_entry(stacktop(S));
ancestor(N,[_|T]) ->
ancestor(N-1,T).
info(Format) -> io:format(lists:concat([ancestor(2),Format,"\r"])).
info(Format,Args) -> io:format(lists:concat([ancestor(2),Format,"\r"]),Args).
Lists is a custom module in the system. Use your foo module instead.