My sole method of debugging (io:format/2) is not working in YAWS. I'm at a loss. My supervisor starts three processes: ETS Manager, YAWS Init, and Ratelimiter. This is successful. I can play around with the rate limiter in the shell... calling the same functions YAWS should be. The difference being the shell behaves as I would expect and I have no idea what is happening in YAWS.
I do know if I spam the command in shell: ratelimiter:limit(IP) it will return true eventually. I can execute the following and it will also return true: ratelimiter:lockout(IP), ratelimiter:blacklist(IP). The limiter is a gen_server.
The functions do the following:
limit/1: Check ETS table if counter > threshold; update counter. If counter > blacklist threshold make entry in mnesia table
blacklist/1: Check mnesia table if entry exists; Yes: reset timer
lockout/1: Immediately enters ID into mnesia table
In my arg_rewrite_mod module I'm doing some checks to ensure I'm getting the HTTP requests I expect, namely GET, POST, and HEAD. I thought this would be a nice place to also do the rate limiting. Do it as soon as possible in the web server's chain of events.
All the changes I've made to the arg_rewrite module seem to work except using "printf"s and the limiter. I'm new to the language so I'm not sure my mistake is obvious or not.
Skeleton of my arg_rewrite_mod:
-module(arg_preproc).
-export([arg_rewrite/1]).
-include("limiter_def.hrl").
-include_lib("/usr/lib/yaws/include/yaws_api.hrl").
is_blacklisted(ID) ->
case ratelimiter:blacklist(ID) of
false -> continue;
true -> throw(blacklist)
end.
is_limited(ID) ->
case ratelimiter:limit(ID) of
false -> continue;
true -> throw(limit)
end.
arg_rewrite(A) ->
Allow = ['GET','POST', 'HEAD'],
try
{IP, _} = A#arg.client_ip_port,
ID = IP,
is_blacklisted(ID),
io:format("~p ~p ~n",[ID, is_blacklisted(ID)]),
%% === Allow expected HTTP requests
HttpReq = (A#arg.req)#http_request.method,
case lists:member(HttpReq, Allow) of
true ->
{_,ReqTgt} = (A#arg.req)#http_request.path,
PassThru = [".css",".jpg",".jpeg",".png",".js"],
%% ... much more ...
false ->
is_limited(ID),
throw(http_method_denied)
end
catch
throw:blacklist -> %% Send back a 429;
throw:limit -> %% Same but no Retry-After;
throw:http_method_denied ->
%%Only thrown experienced
AllowedReq = string:join([atom_to_list(M) || M <- Allow], ","),
A#arg{state=#rewrite_response{status=405,
headers=[{header, {"Allow", AllowedReq}},{header, {connection, "close"}}]
}};
Type:Reason -> {error, {unhandled,{Type, Reason}}}
end.
I can spam curl -I -X HEAD <<any page>> as fast as I can in a bash shell and all I get is HTTP 200. The ETS table has zero entries as well. Using PUT I get a HTTP 405 as intended. I can ratelimiter:lockout({MY_IP}) and get the web page to load in my browser and a HTTP 200 with curl.
I'm confused. Is it the way I started YAWS?
start() ->
os:putenv("YAWSHOME", ?HOMEPATH_YAWS),
code:add_patha(?MODPATH_YAWS),
ok = case (R = application:start(yaws)) of
{error, {already_started, _}} -> ok;
_ -> R
end,
{ok,self()}. %% Tell supervisor everything okay in a manner it expects.
I did this because I thought it would be "easier."
When starting Yaws as part of another application, it's important to use its embedding support. One important thing the Yaws embedding startup code does is set the application environment variable embedded to true:
application:set_env(yaws, embedded, true),
Yaws checks this variable in several of its code paths, especially during initialization, in order to avoid assuming that it's running as a stand-alone daemon process.
Regarding rate limiting, rather than using an arg rewriter, you might consider using a shaper. The yaws_shaper module provides a behavior that expects its callback module to implement two functions:
check/1: yaws_shaper calls this to allow the callback module to decide whether to allow the request from the client. It passes client host information as the callback argument. Your shaper callback module returns either the atom allow to allow the request to proceed, or the tuple {deny, Status, Message} where Status is an HTTP status code to return to the client, such as 429 to indicate the client is making too many requests, and Message is any extra HTML to be returned to the client. (It might be nice if Message could include a reply header such as Retry-After as well; this is something I'll consider adding to Yaws.)
update/3: yaws_shaper calls this when the response for a client is ready to be returned. The first argument is the client host information, the second argument is the number of "hits" (the value 1 for each request), and the third argument is the number of bytes being delivered in response to the client's request. Your shaper callback module can return ok from update/3 (Yaws does not use the return value).
A shaper can use this framework to track how many requests each client is making and how much data Yaws is delivering to each client, and use that information to limit or deny particular clients.
And finally, while "printf debugging" works, it's less than ideal especially in Erlang, which has built-in tracing. You should consider learning the dbg module so you can trace any function you want, see who called it, see what arguments are being passed to it, see what it returns, etc.
I am trying the hot code feature of erlang following the guide from LYAE but i do not understand how to make the update message to get triggered.
I have a module which runs a method that is upgradeable:
Module
-module(upgrade).
-export([main/1,upgrade/1,init/1,init_link/1]).
-record(state,{ version=0,comments=""}).
init(State)->
spawn(?MODULE,main,[State]).
main(State)->
receive
update->
NewState=?MODULE:upgrade(State),
if NewState#state.version>3 -> exit("Max Version Reached") end,
?MODULE:main(NewState);
SomeMessage->
main(State)
end.
upgrade(State=#state{version=Version,comments=Comments})->
Comm=case Version rem 2 of
0 -> "Even version";
_ -> "Uneven version"
end,
#state{version=Version+1,comments=Comm}.
Shell
>c(upgrade).
>rr(upgrade,state).
>U=upgrade:init(#state{version=0,comments="initial"}).
>Monitor=monitor(process,U).
> ......to something to trigger the update message
> flush(). % see the exit message reason
I do not understand how can i perform a hot code reload in order to trigger the update message.
I want when i use flush to get the exit reason from my main method.
The process expects to get the atom update as a message. Since you have the pid of the process in the variable U, you can send the message like this:
U ! update.
Note that the strings Even version and Uneven version are only kept in the state, never printed, so you won't see those. The only thing you'll see is the exit message, after sending update four times and calling flush().
I am trying to make a simple UDP packet decoder.
packet_decoder(Packet)->
<<Opts:8,MobIdLength:8,MobId:64,MobIdType:8,MgeType:8,SeqNum:16,Rest/binary>> = Packet,
io:format("Options:~p~n",Opts),
io:format("MobIdLength:~p~n",MobIdLength),
io:format("MobId:~p~n",MobId),
io:format("MobIdType:~p~n",MobIdType),
io:format("MgeType:~p~n",MgeType),
io:format("SeqNum:~p~n",SeqNum).
Packet is passed by a receive loop:
rcv_loop(Socket) ->
inet:setopts(Socket, [{active, once}, binary]),
io:format("rcvr started:~n"),
receive
{udp, Socket, Host, Port, Bin} ->
packet_decoder(Bin),
rcv_loop(Socket)
end.
I keep getting(following error edited 9/7/12 9:30 EST):
** exception error: no match of right hand side value
<<131,8,53,134,150,4,149,0,80,15,1,2,1,2,0,16,80,71,115,
52,80,71,115,53,24,63,227,197,211,...>>
in function udp_server:packet_decoder/1
called as udp_server:packet_decoder(<<131,8,53,134,150,4,149,0,80,15,
1,2,1,2,0,16,80,71,115,52,80,71,
115,53,24,63,227,197,...>>)
in call from udp_server:rcv_loop/1
in call from udp_server:init/0
If I create the same variable in the Erlang shell as a binary, i.e.
Packet = <<131,8,53,134,150,4,149,0,80,15,1,2,1,2,0,16,80,71,115,52,80,71,115,53,24,63,227,197,211,228,89,72,0,0,0,0,0,0,0,16,0,5,5,32,1,4,255,159,15,18,28,0,34,62,2,0,0,0,0,0,0,0,47,67>>.
<<Opts:8,MobIdLength:8,MobId:64,MobIdType:8,MgeType:8,SeqNum:16,Rest/binary>> = Packet.
It works just fine. Is there some subtlety in passing this to a function that I am missing? I have tried what I think is everything(except the right way). I tried setting the type and size. I also just tried
<<Rest/binary>> = Packet.
To no avail. Any help much appreciated.
The error you are getting when you run your code does not match your code. The error you are getting:
** exception error: no match of right hand side value ...
is a badmatch error and comes from an explicit = match where the pattern does not match the value from the RHS. There is no = in the code for rcv_loop/1. This implies that the loop you are running is not this code. So there are some questions to be asked:
When you have recompiled the module containing rcv_loop/1 have you restarted the loop so you run the new code? This is not done automagically.
Are you sure you are loading/running the code you think you are? I know this question sounds stupid but it is very easy, and not uncommon, to work on one version of the code and load another. You need to get the paths right.
The other things about mentioned about your code would not give this error. The calls to io:format/2 are wrong but would result in errors when you make the actual calls to io:format/2. Using the variable Socket as you do is not an error, it just means that you only want to receive UDP packets from just that socket.
EDIT : the first part of my answer was completely wrong so in order to not mislead, I deleted it.
Like spotted Alexey Kachayev io:format takes as second parameter a list, so :
packet_decoder(Packet)->
<<Opts:8,MobIdLength:8,MobId:64,MobIdType:8,MgeType:8,SeqNum:16,Rest/binary>> = Packet,
io:format("Options:~p~n",[Opts]),
io:format("MobIdLength:~p~n",[MobIdLength]),
io:format("MobId:~p~n",[MobId]),
io:format("MobIdType:~p~n",[MobIdType]),
io:format("MgeType:~p~n",[MgeType]),
io:format("SeqNum:~p~n",[SeqNum]).
I figured it out(kinda). I had been working on this in erlide in eclipse which had worked fine for all of the other parts of the. I tried compiling it from the erl shell and it worked fine. There must be some minor difference in the way eclipse is representing the source or the way it invokes the erlang compiler and shell. I will take it up with erlide.org. Thanks for the help!
I've recently been trying to get to grips with Erlang, but am slightly confused by this...
Problem
I am trying to write a function object in the shell, which receives any message and echoes it back to the shell followed by a new line.
Attempt
My attempt at the function is:
Echo = fun() -> receive Any -> io:fwrite(Any), io:fwrite("~n") end end.
However, if I spawn a new process for this...
someNode#someHost 2> Pid = spawn(Echo).
<0,76,0>
...and then send it a message...
someNode#someHost 3> Pid ! "Hello".
Hello"Hello"
...I don't seem to get a new line character after the write, and before the message return.
Question
Is there something wrong with my approach, that stops this (very simple) example working as expected?
Thanks in advance
Your problem is atomicity.
After printing the first line, the "main thread" gets scheduled and prints the result of Pid ! Msg, i.e. Msg.
io:fwrite can take arguments just like printf in C:
Echo = fun() ->
receive
Any ->
io:fwrite("~p~n", [Any])
end
end
Mainly I want to know if I can send a function in a message in a distributed Erlang setup.
On Machine 1:
F1 = Fun()-> hey end,
gen_server:call(on_other_machine,F1)
On Machine 2:
handler_call(Function,From,State) ->
{reply,Function(),State)
Does it make sense?
Here's an interesting article about "passing fun's to other Erlang nodes". To resume it briefly:
[...] As you might know, Erlang distribution
works by sending the binary encoding
of terms; and so sending a fun is also
essentially done by encoding it using
erlang:term_to_binary/1; passing the
resulting binary to another node, and
then decoding it again using
erlang:binary_to_term/1.[...]
This is pretty obvious
for most data types; but how does it
work for function objects?
When you encode a fun, what is encoded
is just a reference to the function,
not the function implementation.
[...]
[...]the definition of the function is not passed along; just exactly enough information to recreate the fun at an other node if the module is there.
[...] If the module containing the fun has not yet been loaded, and the target node is running in interactive mode; then the module is attempted loaded using the regular module loading mechanism (contained in the module error_handler); and then it tries to see if a fun with the given id is available in said module. However, this only happens lazily when you try to apply the function.
[...] If you never attempt to apply the function, then nothing bad happens. The fun can be passed to another node (which has the module/fun in question) and then everybody is happy.
Maybe the target node has a module loaded of said name, but perhaps in a different version; which would then be very likely to have a different MD5 checksum, then you get the error badfun if you try to apply it.
I would suggest you to read the whole article, cause it's extremely interesting.
You can send any valid Erlang term. Although you have to be careful when sending funs. Any fun referencing a function inside a module needs that module to exist on the target node to work:
(first#host)9> rpc:call(second#host, erlang, apply,
[fun io:format/1, ["Hey!~n"]]).
Hey!
ok
(first#host)10> mymodule:func("Hey!~n").
5
(first#host)11> rpc:call(second#host, erlang, apply,
[fun mymodule:func/1, ["Hey!~n"]]).
{badrpc,{'EXIT',{undef,[{mymodule,func,["Hey!~n"]},
{rpc,'-handle_call_call/6-fun-0-',5}]}}}
In this example, io exists on both nodes and it works to send a function from io as a fun. However, mymodule exists only on the first node and the fun generates an undef exception when called on the other node.
As for anonymous functions, it seems they can be sent and work as expected.
t1#localhost:
(t1#localhost)7> register(shell, self()).
true
(t1#localhost)10> A = me, receive Fun when is_function(Fun) -> Fun(A) end.
hello me you
ok
t2#localhost:
(t2#localhost)11> B = you.
you
(t2#localhost)12> Fn2 = fun (A) -> io:format("hello ~p ~p~n", [A, B]) end.
#Fun<erl_eval.6.54118792>
(t2#localhost)13> {shell, 't1#localhost'} ! Fn2.
I am adding coverage logic to an app built on riak-core, and the merge of results gathered can be tricky if anonymous functions cannot be used in messages.
Also check out riak_kv/src/riak_kv_coverage_filter.erl
riak_kv might be using it to filter result, I guess.