Error in producing data to Kafka using Erlang/ekaf - erlang

I'm trying to connect Kafka and produce some messages using Erlang/ekaf.
The code is a simple example explained in ekaf's README, but will exit when application:start is called.
Please note that I've used gen_icmp:ping to make sure the server running Kafka is accessible to this machine.
I have also run python script to produce some random message to this Kafka and it was successful, so most probably there is something I have missed in my Erlang code. :)
Source:
-module(kafka).
-compile(export_all).
run_test() ->
io:format("run_test: start.~n"),
pingKafka(),
try init_ekaf() of
_ -> io:format("run_test: ok~n")
catch
error:Msg -> io:format("run_test: error: ~p~n", [Msg]);
throw:Msg -> io:format("run_test: throw: ~p~n", [Msg]);
exit:Msg -> io:format("run_test: exit: ~p~n", [Msg])
end.
init_ekaf() ->
io:format("init_ekaf: start.~n"),
application:load(ekaf),
application:set_env(ekaf, ekaf_bootstrap_broker, {"kafka.dev", 9092}),
ok = application:start(ekaf),
io:format("init_ekaf: started.~n"),
Topic = <<"foobar">>,
ekaf:produce_sync(Topic, <<"some data">>),
io:format("init_ekaf: message sent.~n"),
ok.
pingKafka() ->
Res = gen_icmp:ping("kafka.dev"),
io:format("pingKafka: ~p.~n", [Res]),
ok.
Output:
run_test: start.
pingKafka: [{ok,"kafka.dev",
{192,168,0,51},
{192,168,0,51},
{12343,0,64,130},
<<" !\"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJK">>}].
init_ekaf: start.
run_test: error: {badmatch,{error,{not_started,gproc}}}
run_test: end.

After reading existing tests in the repository again, I find out that gproc application also need to be started before starting ekaf.
So by adding:
application:start(gproc)
right before application:start(ekaf), the problem solved.
P.S: Found another way to solve the issue which is calling application:ensure_all_started(ekaf) instead of application:start(ekaf).
As mentioned in document, ensure_all_started is an equivalent to calling start/1,2 repeatedly on all dependencies that are not yet started for an application

Related

yaws appmods Accept-Language A record is not correct

I work with yaws from a docker container for years, basically taken from https://github.com/segeda/docker-yaws/blob/master/Dockerfile:
FROM erlang:20-alpine
...
&& git clone https://github.com/klacke/yaws.git /yaws-src \
...
Things just used to run fine, but all of a sudden my code fails, and I cannot find the error(s). I could not even find a git version which worked when it had before, so I suspect it might not be a code fault, but then -- what could it be?
It cannot be my code, can it? Because I boiled it down to the example given in http://yaws.hyber.org/appmods.yaws and produced the same error:
%% this is my appmod called from yaws
-module(myurl).
-author('kklepper').
-include("../../include/yaws_api.hrl").
%-include("/usr/local/lib/yaws/include/yaws_api.hrl").
% relative or absolute -- either way same result
-export([out/1]).
-define(debug, true).
-ifdef(debug).
-define(trace(Str, X), io:format("Mod:~w line:~w ~p ~p~n",
[?MODULE,?LINE, Str, X])).
-else.
-define(trace(X, Y), true).
-endif.
box(Str) ->
{'div',[{class,"box"}],
{pre,[],Str}}.
out(A) ->
?trace('A', A),
{ehtml,
[{p,[],
box(io_lib:format("A#arg.appmoddata = ~p~n"
"A#arg.appmod_prepath = ~p~n"
"A#arg.querydata = ~p~n",
[A#arg.appmoddata,
A#arg.appmod_prepath,
A#arg.querydata]))}]}.
The file yaws_api.hrl was and still is present:
/yaws # ls -la /usr/local/lib/yaws/include/yaws_api.hrl
-rw-r--r-- 1 1000 50 5563 May 13 2018 /usr/local/lib/yaws/include/yaws_api.hrl
With trace you can see that the A record is not correct -- no A#arg.querydata etc. -- hence the error. Why?
Obviously, querydata, for example, as such is given: lg=en.
Mod:myurl line:22 'A' {arg,#Port<0.2934>,
{{10,255,0,2},52801},
{headers,"keep-alive",
"text/html, application/xhtml+xml, application/xml;q=0.9, image/webp, image/apng, */*;q=0.8, application/signed-exchange;v=b3",
"voxx.b2d",undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36",
undefined,[],undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,
[{http_header,11,'Accept-Language',undefined,
"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"},
{http_header,10,'Accept-Encoding',undefined,
"gzip, deflate"},
{http_header,0,"Upgrade-Insecure-Requests",
undefined,"1"}]},
{http_request,'GET',
{abs_path,"/industries?lg=en"},
{1,1}},
{http_request,'GET',
{abs_path,"/industries?lg=en"},
{1,1}},
undefined,"/industries","lg=en","industries","/ci",
"/","/ci/industries",undefined,undefined,<0.163.0>,
[],[],[],"/industries",myurl}
=ERROR REPORT==== 4-Oct-2019::00:46:06 ===
ERROR erlang code threw an uncaught exception:
File: appmod:0
Class: error
Exception: {badrecord,arg}
Req: {http_request,'GET',{abs_path,"/industries?lg=en"},{1,1}}
Stack: [{myurl,out,1,
[{file,"/usr/local/lib/yaws/voxx/ebin/myurl.erl"},{line,29}]},
{yaws_server,deliver_dyn_part,8,
[{file,"yaws_server.erl"},{line,2921}]},
{yaws_server,aloop,4,[{file,"yaws_server.erl"},{line,1274}]},
{yaws_server,acceptor0,2,[{file,"yaws_server.erl"},{line,1073}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]
By now I ran out of ideas. Any hint?
The data in question seems to be there. I am interested among others in language data and used to get the accepted language from A like so:
get_lang(A) ->
find_lang(find_http_header('Accept-Language', (A#arg.headers)#headers.other)).
find_http_header(Key,Headers) when is_list(Headers) ->
case lists:keysearch(Key,3,Headers) of
{value,{_,_,_,_,Value}} -> Value;
false -> undefined
end.
find_lang(AcceptLanguage) ->
case AcceptLanguage of
undefined ->
"en";
_ ->
L = lists:nth(1,string:tokens(
lists:nth(1,string:tokens(
lists:nth(1,string:tokens(AcceptLanguage, ";"))
,","))
,"-")),
L
end.
Without the correct data structure, this cannot work.
A detail you included in your question shows that your appmod is getting a badrecord exception when trying to access the #arg record that Yaws passes to it. There are a couple ways to get such an exception:
Pass an instance of a data type other than the expected record type to the function. This isn't happening in this case, since Yaws is calling the function and is correctly passing an #arg record instance.
Compile the caller and the called function with two different definitions of the same record. We can simulate this problem in an Erlang shell as follows:
First, define a record #arg containing two fields
Define a function that takes an instance of that record as an argument and returns the value of one of the fields
Redefine the record in the shell to have just one field
Pass an instance of the redefined record to the function, which will cause a badrecord exception
1> rd(arg, {f,g}).
arg
2> F = fun(A) -> A#arg.f end.
#Fun<erl_eval.7.126501267>
3> rd(arg, {f}).
arg
4> F(#arg{f=1}).
** exception error: {badrecord,arg}
Since you're using Yaws 2.0.7 and you're also cloning Yaws from github, it's likely you're compiling some part of your code, probably your appmod, against the Yaws master branch and then running it against 2.0.7. Sometime after I tagged Yaws 2.0.7 on github, I accepted a change that added a new field to the #arg record definition. What this means is that any #arg record Yaws 2.0.7 creates and passes to your appmod will result in a badrecord exception since the appmod expects a record containing the additional field.
I can think of a few ways to avoid this problem:
Get rid of your Yaws github clone and build everything against your installation of Yaws 2.0.7.
In your Yaws clone, git checkout yaws-2.0.7 so that anything you access or use from the clone is the same as what's in your Yaws 2.0.7 installation.
Uninstall Yaws 2.0.7, then compile and install from your Yaws clone. Perhaps apply a local tag first, then build and install from that, so that your deployments can be more easily reproduced, and so that you can grab Yaws updates from github without breaking anything locally.

[Cowboy-Erlang]: Error when pin-pointing to localhost:8080 using provided cowboy example web_server

I am trying out a cowboy example provided by this github repository:
https://github.com/ninenines/cowboy/tree/master/examples/web_server
I build the release successfully using erlang.mk and run the following command, which opens the Erlang shell in my linux terminal:
$ ./_rel/web_server_example/bin/web_server_example console
But when I then open http://localhost:8080 in my web-browser, I get the following error report:
=ERROR REPORT==== 26-Nov-2014::14:33:48 === Error in process <0.166.0> on node 'web_server_example#127.0.0.1' with exit value:
{function_clause,[{cowboy_req,ensure_response,[{ok,{http_req,#Port<0.454>,ranch_tcp,keepalive,<0.166.0>,<<3
bytes>>,'HTTP/1.1',{{127,0,0,1},57150},<<9 bytes>>,undefined,8080,<<1
byte>>,undefined,<<0 bytes>>,undefined,undefined,[{<<4 bytes>>,<<14
bytes>>},{<<10 bytes>>,<<10 bytes>>},{<<13 bytes>>,<<9 bytes>>},{<<6
bytes>>,<<74 bytes>>},{<<10 bytes>>,<<104 bytes>>},{<<15 bytes>>,<<19
bytes>>},{<<15 bytes>>,<<35 bytes>>}],[{<<10 bytes>>,[<<10
bytes>>]}],undefined,[],waiting,<<0 bytes>>,undefined...
=ERROR REPORT==== 26-Nov-2014::14:33:48 === Ranch listener http had connection process started with cowboy_protocol:start_link/4 at
<0.166.0> exit with reason:
{function_clause,[{cowboy_req,ensure_response,[{ok,{http_req,#Port<0.454>,ranch_tcp,keepalive,<0.166.0>,<<"GET">>,'HTTP/1.1',{{127,0,0,1},57150},<<"localhost">>,undefined,8080,<<"/">>,undefined,<<>>,undefined,undefined,[{<<"host">>,<<"localhost:8080">>},{<<"connection">>,<<"keep-alive">>},{<<"cache-control">>,<<"max-age=0">>},{<<"accept">>,<<"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8">>},{<<"user-agent">>,<<"Mozilla/5.0
(X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/39.0.2171.65 Safari/537.36">>},{<<"accept-encoding">>,<<"gzip,
deflate,
sdch">>},{<<"accept-language">>,<<"sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4">>}],[{<<"connection">>,[<<"keep-alive">>]}],undefined,[],waiting,<<>>,undefined,false,done,[],<<>>,undefined}},204],[{file,"src/cowboy_req.erl"},{line,1009}]},{cowboy_protocol,next_request,3,[{file,"src/cowboy_protocol.erl"},{line,454}]}]}
This is "src/cowboy_protocol.erl" around line 454:
-spec next_request(cowboy_req:req(), #state{}, any()) -> ok.
next_request(Req, State=#state{req_keepalive=Keepalive, timeout=Timeout},
HandlerRes) ->
cowboy_req:ensure_response(Req, 204),
%% If we are going to close the connection,
%% we do not want to attempt to skip the body.
case cowboy_req:get(connection, Req) of
close ->
terminate(State);
_ ->
%% Skip the body if it is reasonably sized. Close otherwise.
Buffer = case cowboy_req:body(Req) of
{ok, _, Req2} -> cowboy_req:get(buffer, Req2);
_ -> close
end,
%% Flush the resp_sent message before moving on.
if HandlerRes =:= ok, Buffer =/= close ->
receive {cowboy_req, resp_sent} -> ok after 0 -> ok end,
?MODULE:parse_request(Buffer,
State#state{req_keepalive=Keepalive + 1,
until=until(Timeout)}, 0);
true ->
terminate(State)
end
end.
And the webb_server_app.erl file:
%% Feel free to use, reuse and abuse the code in this file.
%% #private
-module(web_server_app).
-behaviour(application).
%% API.
-export([start/2]).
-export([stop/1]).
%% API.
start(_Type, _Args) ->
Dispatch = cowboy_router:compile([
{'_', [
{"/[...]", cowboy_static, {priv_dir, web_server, "", [
{mimetypes, cow_mimetypes, all},
{dir_handler, directory_handler}
]}}
]}
]),
{ok, _} = cowboy:start_http(http, 100, [{port, 8080}], [
{env, [{dispatch, Dispatch}]},
{middlewares, [cowboy_router, directory_lister, cowboy_handler]}
]),
web_server_sup:start_link().
stop(_State) ->
ok.
Does anyone have any suggestion as to what is exactly causing this problem, and how to solve it? Thanks.
EDIT:
I can confirm that the fault was in the Erlang OTP version R16B02 in my case. Changing to the latest Erlang release (17.3), as well as resolving missing file dependencies that arose during the configuration stage (with the solutions in the following link):
https://sites.google.com/site/comptekkia/erlang/how-to-install-erlang-on-ubuntu-10-10
Solved the problem(s). The web_server example runs without error now.
The error says function clause, so the arguments to cowboy_req:ensure_response/2 must be wrong. And indeed they are, because first argument is {ok, Request} instead of Request. You have to trace back, which function called next_request/3 with bad argument, because it clearly should be called without ok.
Probably somewhere at the end, you will find something like:
Req = some_function(...)
And you will need to change it to:
{ok, Req} = some_function(...)
Good luck and happy bug hunting :D
UPDATE: I just cloned the repo and it works fine for me. I got the directory listing, so it is not bug in cowboy, but somewhere in user code.

What causes a "not mocked" error when using meck (Erlang)?

I'm a meck (and Erlang) newbie and I'm struggling a bit with meck. I'm getting the following error:
=ERROR REPORT==== 27-Jan-2014::16:20:05 ===
Error in process <0.1825.0> with exit value: {{not_mocked,substatsDb},
[{meck_proc,gen_server,3,[{file,"src/meck_proc.erl"},{line,443}]},{meck_code_gen,exec,4,
[{file,"src/meck_code_gen.erl"},{line,147}]},{substats,loop,1,
[{file,"/Users/uyounri/gitsandbox/subpub/src/su...
At the beginning of my test I declare the module to be mocked:
meck:new(substats)
At the very end of my test the last thing I do is to unload the module that was mocked mocked:
meck:unload(substats)
The mocking seems to be working as expected until towards the end of the test when the above error is produced.
Thanks,
Rich
EDIT
Adding 2 ?debugFmt() lines seems to have fixed the problem; at least I no longer get the error. Here's the complete function that was modified:
stop(_) ->
meck:expect(substatsDb, stop, 1, fun(dbconn) -> ok end),
substats:stop(),
%% Note: this and the next ?debugFmt() calls prevent a meck
%% exit error. They both have to be present to prevent the error
?debugFmt("stop:~n", []),
meck:unload(substatsDb),
?debugFmt("stop: exit~n", []).
Have you tried adding the option passthrough when mocking the module?
meck:new(substatsDb, [passthrough])
Also, you are using the module "substatsDb" in the meck:expect call, but doing the meck:new for another module (substats), you should be doing everything for the same modules (new, expect, and unload)
hope it helps!

Meck behaving strangely for multiple mocked modules

I have following module
-module(bhavcopy_downloader).
-export([download/2]).
download(From, SaveTo) ->
{ok, {{Status, _}, _, Body}} = lhttpc:request(From, "GET", [], infinity),
case Status of
200 -> file:write(SaveTo, Body),
true;
_ -> false
end.
And following tests for the above code
file_download_test_() ->
{foreach,
fun() ->
meck:new(lhttpc)
meck:new(file, [unstick])
end,
fun(_) ->
meck:unload(file),
meck:unload(lhttpc)
end,
{"saves the file at specified location",
fun() ->
meck:expect(lhttpc, request, 4, {ok, {{200, "OK"}, [], <<"response">>}}),
meck:expect(file, write_file, fun(Path, Data) ->
?assertEqual(Path, "~/Downloads/data-downloader/test.html"),
?assertEqual(Data, <<"response">>)
end),
?assertEqual(true, bhavcopy_downloader:download("http://google.com", "~/Downloads/data-downloader/test.html")),
?assert(meck:validate(file))
end}]
}.
When I run the tests I get following error (only part of the error pasted below for brevity). Looking at the error below, I am kind of feeling that file module is not being mocked (or the mock of file module being overridden when I set the other mock using meck:new(lhttpc). What could be going wrong here?
=ERROR REPORT==== 16-Feb-2013::20:17:24 ===
** Generic server file_meck terminating
** Last message in was {'EXIT',<0.110.0>,
{compile_forms,
{error,
[{[],
[{none,compile,
{crash,beam_asm,
{undef,
[{file,get_cwd,[],[]},
{filename,absname,1,
[{file,"filename.erl"},{line,67}]},
{compile,beam_asm,1,
[{file,"compile.erl"},{line,1245}]},
{compile,'-internal_comp/4-anonymous-1-',2,
[{file,"compile.erl"},{line,273}]},
{compile,fold_comp,3,
[{file,"compile.erl"},{line,291}]},
{compile,internal_comp,4,
[{file,"compile.erl"},{line,275}]},
{compile,'-do_compile/2-anonymous-0-',2,
[{file,"compile.erl"},{line,152}]}]}}}]}],
[{"src/lhttpc_types.hrl",
[{31,erl_lint,{new_builtin_type,{boolean,0}}},
{31,erl_lint,{renamed_type,bool,boolean}}]}]}}}
This is a catch 22 in Meck, caused by the fact that Meck uses the Erlang compiler, which in turns uses the file module. When Meck tries recompile the file module it needs the file module (through the compiler) and thus crashes.
As of this moment, Meck doesn't handle mocking the file module. Your best alternative is to wrap the file module calls in another module and mock this module instead.
(It is theoretically possible to fix this in Meck by using the internals of the compiler and the code server instead, for example erlang:load_module/2, however this is quite tricky and needs to be designed and tested well)

Avoid NZEC error in Erlang in SPOJ

I've written code in Erlang, and I get the correct answer on my machine. But when I submit it on SPOJ it gives an NZEC (non zero exit code) error. I have used built-in functions like halt() and init:stop(), and their specification clearly says that they are used to avoid non-zero exit code error. But still I get the same error. How can I solve this problem?
EDIT The code as required by a comment:
-module(factorial).
-export([main/0]).
main() ->
{ok, [No_of_cases]} = io:fread("", "~d"),
loop(No_of_cases).
loop(0) ->
%init:stop();
halt(1);
loop(No_of_cases) ->
{ok, [Number]} = io:fread("", "~d"),
ResultFactorial = find_factorial(Number,1),
io:format("~p~n",[ResultFactorial]),
loop(No_of_cases-1).
find_factorial(0,Product) ->
Product;
find_factorial(Number,Product) ->
find_factorial(Number-1,Product*Number).
I got the answer. The trick is that your module name always has to be tested and the entry point should be function main . For example, after compilation it should be run as tested:main().

Resources