Hello fellow Erlangers :)
Just another Erlang enthusiast playing with the language here. I have a very simple YAWS app module which works fine when accessed by single clients. However, if I try to span multiple clients to simulate light traffic, those clients start receiving errors. Any idea what can be causing it?
Code spawning the clients:
-module(concurrent).
-export([measure/0, get/0]).
get() ->
Result = httpc:request("http://localhost:8080/blah"),
io:format("Result: ~p.\n", [Result]).
get_spawner(0) -> io:format("Done.\n");
get_spawner(Times) ->
spawn(concurrent, get, []),
get_spawner(Times - 1).
measure() ->
io:fwrite("Starting inets...\n"),
inets:start(),
io:fwrite("Done\n"),
io:fwrite("Creating blah...\n"),
httpc:request(put, { "http://localhost:8080/blah", [], "", "" }, [], []),
io:fwrite("Done\n"),
get_spawner(9),
io:fwrite("Stopping inets...\n"),
inets:stop(),
io:fwrite("Done\n"),
init:stop().
I observed that it all works just fine for 8 concurrent clients, but when I span 9 or more clients, they start receiving the following message back from the server:
=ERROR REPORT==== 24-Jan-2017::12:30:04 ===
** Generic server httpc_manager terminating
** Last message in was {request,
{request,undefined,<0.74.0>,0,http,
{"localhost",8080},
"/blah",[],get,
{http_request_h,undefined,"keep-alive",
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,"localhost:8080",
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,[],undefined,undefined,undefined,
undefined,"0",undefined,undefined,
undefined,undefined,undefined,undefined,[]},
{[],[]},
{http_options,"HTTP/1.1",infinity,true,
{essl,[]},
undefined,false,infinity,false},
"http://localhost:8080/blah",[],none,[],
1485261004189,undefined,undefined,false}}
** When Server state == {state,[],httpc_manager__handler_db,
{cookie_db,undefined,8209},
httpc_manager__session_db,httpc_manager,
{options,
{undefined,[]},
{undefined,[]},
0,2,5,120000,2,disabled,false,inet,default,
default,[]}}
** Reason for termination ==
** {'EXIT',
{shutdown,
{gen_server,call,
[httpc_handler_sup,
{start_child,
[<0.61.0>,
{request,#Ref<0.0.6.64>,<0.74.0>,0,http,
{"localhost",8080},
"/blah",[],get,
{http_request_h,undefined,"keep-alive",undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,"localhost:8080",undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,[],undefined,
undefined,undefined,undefined,"0",undefined,
undefined,undefined,undefined,undefined,
undefined,[]},
{[],[]},
{http_options,"HTTP/1.1",infinity,true,
{essl,[]},
undefined,false,infinity,false},
"http://localhost:8080/blah",[],none,[],
1485261004189,undefined,undefined,false},
{options,
{undefined,[]},
{undefined,[]},
0,2,5,120000,2,disabled,false,inet,default,default,
[]},
httpc_manager]},
infinity]}}}
=ERROR REPORT==== 24-Jan-2017::12:30:04 ===
Error in process <0.74.0> with exit value:
{{case_clause,
{undefined,
{error,
{'EXIT',
{shutdown,
{gen_server,call,
[httpc_handler_sup,
{start_child,
[<0.61.0>,
{request,#Ref<0.0.6.64>,<0.74.0>,0,http,
{"localhost",8080},
"/blah",[],get,
{http_request_h,undefined,"keep-alive",
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,"localhost:8080",
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,[],
undefined,undefined,undefined,
undefined,"0",undefined,undefined,
undefined,undefined,undefined,
undefined,[]},
{[],[]},
{http_options,"HTTP/1.1",infinity,true,
{essl,[]},
undefined,false,infinity,false},
"http://localhost:8080/blah",[],none,[],
1485261004189,undefined,undefined,false},
{options,
{undefined,[]},
{undefined,[]},
0,2,5,120000,2,disabled,false,inet,default,
default,[]},
httpc_manager]},
infinity]}}}}}},
[{httpc,handle_request,9,[{file,"httpc.erl"},{line,574}]},
{concurrent,get,0,
[{file,
"c:/Users/piotr_justyna/Documents/rets/performance_tests/concurrent.erl"},
{line,5}]}]}
On the server side I am getting this:
=ERROR REPORT==== 24-Jan-2017::12:28:55 ===
Unhandled reply fr. do_recv() {error,econnaborted}
I'm sure it's something straightforward, just can't figure out what it is. Is it the number of concurrent connections that causes it? Can you help shed more light on this?
Regards,
Piotr
Your code spawns a number of httpc:request/1 calls, but then immediately afterwards it stops inets:
...
get_spawner(9),
io:fwrite("Stopping inets...\n"),
inets:stop(),
...
Because inets is shut down, requests that are still in progress are getting shut down prematurely, as you can see in the error report:
** Reason for termination ==
** {'EXIT',
{shutdown,
{gen_server,call,
[httpc_handler_sup,
You need to make the parent of all the spawned requests wait for those requests to finish before calling inets:stop/0. One way to do that would be to pass the parent pid to each spawned child and once httpc:request/1 returns, have the child send a message to the parent. The parent would spawn all N children, then sit in a loop waiting to receive N completion messages from the children, and only after receiving all completion messages would it call inets:stop/0.
The reason it works for 8 requests but not 9 is due to a race condition around the lack of waiting prior to calling inets:stop/0. The default size of the Yaws pool of request handling acceptor processes happens to be 8. When your code sends 8 requests, the pool handles it quickly enough to beat your code's call to inets:stop/0, but with 9, there's a slight delay as one acceptor handles two requests, and that slight timing change is enough to allow your inets:stop/0 call to clobber one or more of your client processes. You can modify the Yaws acceptor pool size via the acceptor_pool_size configuration variable.
If you're serious about load testing you might consider using an industrial-strength solution like tsung instead of rolling your own.
I'm trying to connect Kafka and produce some messages using Erlang/ekaf.
The code is a simple example explained in ekaf's README, but will exit when application:start is called.
Please note that I've used gen_icmp:ping to make sure the server running Kafka is accessible to this machine.
I have also run python script to produce some random message to this Kafka and it was successful, so most probably there is something I have missed in my Erlang code. :)
Source:
-module(kafka).
-compile(export_all).
run_test() ->
io:format("run_test: start.~n"),
pingKafka(),
try init_ekaf() of
_ -> io:format("run_test: ok~n")
catch
error:Msg -> io:format("run_test: error: ~p~n", [Msg]);
throw:Msg -> io:format("run_test: throw: ~p~n", [Msg]);
exit:Msg -> io:format("run_test: exit: ~p~n", [Msg])
end.
init_ekaf() ->
io:format("init_ekaf: start.~n"),
application:load(ekaf),
application:set_env(ekaf, ekaf_bootstrap_broker, {"kafka.dev", 9092}),
ok = application:start(ekaf),
io:format("init_ekaf: started.~n"),
Topic = <<"foobar">>,
ekaf:produce_sync(Topic, <<"some data">>),
io:format("init_ekaf: message sent.~n"),
ok.
pingKafka() ->
Res = gen_icmp:ping("kafka.dev"),
io:format("pingKafka: ~p.~n", [Res]),
ok.
Output:
run_test: start.
pingKafka: [{ok,"kafka.dev",
{192,168,0,51},
{192,168,0,51},
{12343,0,64,130},
<<" !\"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJK">>}].
init_ekaf: start.
run_test: error: {badmatch,{error,{not_started,gproc}}}
run_test: end.
After reading existing tests in the repository again, I find out that gproc application also need to be started before starting ekaf.
So by adding:
application:start(gproc)
right before application:start(ekaf), the problem solved.
P.S: Found another way to solve the issue which is calling application:ensure_all_started(ekaf) instead of application:start(ekaf).
As mentioned in document, ensure_all_started is an equivalent to calling start/1,2 repeatedly on all dependencies that are not yet started for an application
I am trying out a cowboy example provided by this github repository:
https://github.com/ninenines/cowboy/tree/master/examples/web_server
I build the release successfully using erlang.mk and run the following command, which opens the Erlang shell in my linux terminal:
$ ./_rel/web_server_example/bin/web_server_example console
But when I then open http://localhost:8080 in my web-browser, I get the following error report:
=ERROR REPORT==== 26-Nov-2014::14:33:48 === Error in process <0.166.0> on node 'web_server_example#127.0.0.1' with exit value:
{function_clause,[{cowboy_req,ensure_response,[{ok,{http_req,#Port<0.454>,ranch_tcp,keepalive,<0.166.0>,<<3
bytes>>,'HTTP/1.1',{{127,0,0,1},57150},<<9 bytes>>,undefined,8080,<<1
byte>>,undefined,<<0 bytes>>,undefined,undefined,[{<<4 bytes>>,<<14
bytes>>},{<<10 bytes>>,<<10 bytes>>},{<<13 bytes>>,<<9 bytes>>},{<<6
bytes>>,<<74 bytes>>},{<<10 bytes>>,<<104 bytes>>},{<<15 bytes>>,<<19
bytes>>},{<<15 bytes>>,<<35 bytes>>}],[{<<10 bytes>>,[<<10
bytes>>]}],undefined,[],waiting,<<0 bytes>>,undefined...
=ERROR REPORT==== 26-Nov-2014::14:33:48 === Ranch listener http had connection process started with cowboy_protocol:start_link/4 at
<0.166.0> exit with reason:
{function_clause,[{cowboy_req,ensure_response,[{ok,{http_req,#Port<0.454>,ranch_tcp,keepalive,<0.166.0>,<<"GET">>,'HTTP/1.1',{{127,0,0,1},57150},<<"localhost">>,undefined,8080,<<"/">>,undefined,<<>>,undefined,undefined,[{<<"host">>,<<"localhost:8080">>},{<<"connection">>,<<"keep-alive">>},{<<"cache-control">>,<<"max-age=0">>},{<<"accept">>,<<"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8">>},{<<"user-agent">>,<<"Mozilla/5.0
(X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/39.0.2171.65 Safari/537.36">>},{<<"accept-encoding">>,<<"gzip,
deflate,
sdch">>},{<<"accept-language">>,<<"sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4">>}],[{<<"connection">>,[<<"keep-alive">>]}],undefined,[],waiting,<<>>,undefined,false,done,[],<<>>,undefined}},204],[{file,"src/cowboy_req.erl"},{line,1009}]},{cowboy_protocol,next_request,3,[{file,"src/cowboy_protocol.erl"},{line,454}]}]}
This is "src/cowboy_protocol.erl" around line 454:
-spec next_request(cowboy_req:req(), #state{}, any()) -> ok.
next_request(Req, State=#state{req_keepalive=Keepalive, timeout=Timeout},
HandlerRes) ->
cowboy_req:ensure_response(Req, 204),
%% If we are going to close the connection,
%% we do not want to attempt to skip the body.
case cowboy_req:get(connection, Req) of
close ->
terminate(State);
_ ->
%% Skip the body if it is reasonably sized. Close otherwise.
Buffer = case cowboy_req:body(Req) of
{ok, _, Req2} -> cowboy_req:get(buffer, Req2);
_ -> close
end,
%% Flush the resp_sent message before moving on.
if HandlerRes =:= ok, Buffer =/= close ->
receive {cowboy_req, resp_sent} -> ok after 0 -> ok end,
?MODULE:parse_request(Buffer,
State#state{req_keepalive=Keepalive + 1,
until=until(Timeout)}, 0);
true ->
terminate(State)
end
end.
And the webb_server_app.erl file:
%% Feel free to use, reuse and abuse the code in this file.
%% #private
-module(web_server_app).
-behaviour(application).
%% API.
-export([start/2]).
-export([stop/1]).
%% API.
start(_Type, _Args) ->
Dispatch = cowboy_router:compile([
{'_', [
{"/[...]", cowboy_static, {priv_dir, web_server, "", [
{mimetypes, cow_mimetypes, all},
{dir_handler, directory_handler}
]}}
]}
]),
{ok, _} = cowboy:start_http(http, 100, [{port, 8080}], [
{env, [{dispatch, Dispatch}]},
{middlewares, [cowboy_router, directory_lister, cowboy_handler]}
]),
web_server_sup:start_link().
stop(_State) ->
ok.
Does anyone have any suggestion as to what is exactly causing this problem, and how to solve it? Thanks.
EDIT:
I can confirm that the fault was in the Erlang OTP version R16B02 in my case. Changing to the latest Erlang release (17.3), as well as resolving missing file dependencies that arose during the configuration stage (with the solutions in the following link):
https://sites.google.com/site/comptekkia/erlang/how-to-install-erlang-on-ubuntu-10-10
Solved the problem(s). The web_server example runs without error now.
The error says function clause, so the arguments to cowboy_req:ensure_response/2 must be wrong. And indeed they are, because first argument is {ok, Request} instead of Request. You have to trace back, which function called next_request/3 with bad argument, because it clearly should be called without ok.
Probably somewhere at the end, you will find something like:
Req = some_function(...)
And you will need to change it to:
{ok, Req} = some_function(...)
Good luck and happy bug hunting :D
UPDATE: I just cloned the repo and it works fine for me. I got the directory listing, so it is not bug in cowboy, but somewhere in user code.