YAWs embedded with appmod not working for me - erlang

Alright, what am I doing wrong here. I'm trying the simple example of embedded YAWs from http://yaws.hyber.org/embed.yaws but with an appmod. I've added the my_app.erl file and compiled it. It works if not in embedded YAWs so I think it is specific to embedded.
-module(ybed).
-compile(export_all).
start() ->
{ok, spawn(?MODULE, run, [])}.
run() ->
Id = "embedded",
GconfList = [{ebin_dir, ["/Users/someuser/yawsembedded/ebin"]}],
Docroot = "/Users/someuser/yawstest",
SconfList = [{port, 8888},
{listen, {0,0,0,0}},
{docroot, Docroot},
{appmods, [{"/", my_app}]}
],
{ok, SCList, GC, ChildSpecs} =
yaws_api:embedded_start_conf(Docroot, SconfList, GconfList),
[supervisor:start_child(ybed_sup, Ch) || Ch <- ChildSpecs],
yaws_api:setconf(GC, SCList),
{ok, self()}.
Getting this Error:
ERROR erlang code threw an uncaught exception:
File: appmod:0
Class: error
Exception: undef
Req: {http_request,'GET',{abs_path,"/demo"},{1,1}}
Stack: [{my_app,out,
[{arg,#Port<0.2721>,
{{127,0,0,1},63720},
{headers,"keep-alive",
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"0.0.0.0:8888",undefined,undefined,undefined,undefined,
undefined,undefined,undefined,
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0",
undefined,[],undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
[{http_header,0,"Dnt",undefined,"1"},
{http_header,10,'Accept-Encoding',undefined,
"gzip, deflate"},
{http_header,11,'Accept-Language',undefined,"null"}]},
{http_request,'GET',{abs_path,"/demo"},{1,1}},
{http_request,'GET',{abs_path,"/demo"},{1,1}},
undefined,"/demo",undefined,undefined,
"/Users/someuser/yawstest","/",
"/Users/someuser/yawstest/demo",undefined,undefined,
<0.63.0>,[],"/","/",undefined}],
[]},
{yaws_server,deliver_dyn_part,8,
[{file,"yaws_server.erl"},{line,2818}]},
{yaws_server,aloop,4,[{file,"yaws_server.erl"},{line,1232}]},
{yaws_server,acceptor0,2,[{file,"yaws_server.erl"},{line,1068}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]

The stack trace shows that your my_app:out/1 function is getting called, but you're getting an undef exception. This is occurring because the runtime can't find the my_app:out/1 function, which means either it can't find the module or the module exists but does not export an out/1 function. For example, I was able to duplicate the error using the example code by not providing a my_app module.
First, make sure your my_app.erl file exports an out/1 function. Here's a trivial one that just returns a 405 error for all requests:
-module(my_app).
-export([out/1]).
out(_Arg) ->
{status, 405}.
Compile your my_app.erl file and put the compiled my_app.beam file either in a load path already known to the Erlang runtime, or in a directory you add to the load path. In your code it appears you're trying the latter approach, since you're specifically adding an ebin_dir with this Yaws global configuration directive:
GconfList = [{ebin_dir, ["/Users/someuser/yawsembedded/ebin"]}],
You need to verify that the /Users/someuser/yawsembedded/ebin directory exists, and that the compiled my_app.beam file is located there.

Related

yaws appmods Accept-Language A record is not correct

I work with yaws from a docker container for years, basically taken from https://github.com/segeda/docker-yaws/blob/master/Dockerfile:
FROM erlang:20-alpine
...
&& git clone https://github.com/klacke/yaws.git /yaws-src \
...
Things just used to run fine, but all of a sudden my code fails, and I cannot find the error(s). I could not even find a git version which worked when it had before, so I suspect it might not be a code fault, but then -- what could it be?
It cannot be my code, can it? Because I boiled it down to the example given in http://yaws.hyber.org/appmods.yaws and produced the same error:
%% this is my appmod called from yaws
-module(myurl).
-author('kklepper').
-include("../../include/yaws_api.hrl").
%-include("/usr/local/lib/yaws/include/yaws_api.hrl").
% relative or absolute -- either way same result
-export([out/1]).
-define(debug, true).
-ifdef(debug).
-define(trace(Str, X), io:format("Mod:~w line:~w ~p ~p~n",
[?MODULE,?LINE, Str, X])).
-else.
-define(trace(X, Y), true).
-endif.
box(Str) ->
{'div',[{class,"box"}],
{pre,[],Str}}.
out(A) ->
?trace('A', A),
{ehtml,
[{p,[],
box(io_lib:format("A#arg.appmoddata = ~p~n"
"A#arg.appmod_prepath = ~p~n"
"A#arg.querydata = ~p~n",
[A#arg.appmoddata,
A#arg.appmod_prepath,
A#arg.querydata]))}]}.
The file yaws_api.hrl was and still is present:
/yaws # ls -la /usr/local/lib/yaws/include/yaws_api.hrl
-rw-r--r-- 1 1000 50 5563 May 13 2018 /usr/local/lib/yaws/include/yaws_api.hrl
With trace you can see that the A record is not correct -- no A#arg.querydata etc. -- hence the error. Why?
Obviously, querydata, for example, as such is given: lg=en.
Mod:myurl line:22 'A' {arg,#Port<0.2934>,
{{10,255,0,2},52801},
{headers,"keep-alive",
"text/html, application/xhtml+xml, application/xml;q=0.9, image/webp, image/apng, */*;q=0.8, application/signed-exchange;v=b3",
"voxx.b2d",undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36",
undefined,[],undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,
[{http_header,11,'Accept-Language',undefined,
"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"},
{http_header,10,'Accept-Encoding',undefined,
"gzip, deflate"},
{http_header,0,"Upgrade-Insecure-Requests",
undefined,"1"}]},
{http_request,'GET',
{abs_path,"/industries?lg=en"},
{1,1}},
{http_request,'GET',
{abs_path,"/industries?lg=en"},
{1,1}},
undefined,"/industries","lg=en","industries","/ci",
"/","/ci/industries",undefined,undefined,<0.163.0>,
[],[],[],"/industries",myurl}
=ERROR REPORT==== 4-Oct-2019::00:46:06 ===
ERROR erlang code threw an uncaught exception:
File: appmod:0
Class: error
Exception: {badrecord,arg}
Req: {http_request,'GET',{abs_path,"/industries?lg=en"},{1,1}}
Stack: [{myurl,out,1,
[{file,"/usr/local/lib/yaws/voxx/ebin/myurl.erl"},{line,29}]},
{yaws_server,deliver_dyn_part,8,
[{file,"yaws_server.erl"},{line,2921}]},
{yaws_server,aloop,4,[{file,"yaws_server.erl"},{line,1274}]},
{yaws_server,acceptor0,2,[{file,"yaws_server.erl"},{line,1073}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]
By now I ran out of ideas. Any hint?
The data in question seems to be there. I am interested among others in language data and used to get the accepted language from A like so:
get_lang(A) ->
find_lang(find_http_header('Accept-Language', (A#arg.headers)#headers.other)).
find_http_header(Key,Headers) when is_list(Headers) ->
case lists:keysearch(Key,3,Headers) of
{value,{_,_,_,_,Value}} -> Value;
false -> undefined
end.
find_lang(AcceptLanguage) ->
case AcceptLanguage of
undefined ->
"en";
_ ->
L = lists:nth(1,string:tokens(
lists:nth(1,string:tokens(
lists:nth(1,string:tokens(AcceptLanguage, ";"))
,","))
,"-")),
L
end.
Without the correct data structure, this cannot work.
A detail you included in your question shows that your appmod is getting a badrecord exception when trying to access the #arg record that Yaws passes to it. There are a couple ways to get such an exception:
Pass an instance of a data type other than the expected record type to the function. This isn't happening in this case, since Yaws is calling the function and is correctly passing an #arg record instance.
Compile the caller and the called function with two different definitions of the same record. We can simulate this problem in an Erlang shell as follows:
First, define a record #arg containing two fields
Define a function that takes an instance of that record as an argument and returns the value of one of the fields
Redefine the record in the shell to have just one field
Pass an instance of the redefined record to the function, which will cause a badrecord exception
1> rd(arg, {f,g}).
arg
2> F = fun(A) -> A#arg.f end.
#Fun<erl_eval.7.126501267>
3> rd(arg, {f}).
arg
4> F(#arg{f=1}).
** exception error: {badrecord,arg}
Since you're using Yaws 2.0.7 and you're also cloning Yaws from github, it's likely you're compiling some part of your code, probably your appmod, against the Yaws master branch and then running it against 2.0.7. Sometime after I tagged Yaws 2.0.7 on github, I accepted a change that added a new field to the #arg record definition. What this means is that any #arg record Yaws 2.0.7 creates and passes to your appmod will result in a badrecord exception since the appmod expects a record containing the additional field.
I can think of a few ways to avoid this problem:
Get rid of your Yaws github clone and build everything against your installation of Yaws 2.0.7.
In your Yaws clone, git checkout yaws-2.0.7 so that anything you access or use from the clone is the same as what's in your Yaws 2.0.7 installation.
Uninstall Yaws 2.0.7, then compile and install from your Yaws clone. Perhaps apply a local tag first, then build and install from that, so that your deployments can be more easily reproduced, and so that you can grab Yaws updates from github without breaking anything locally.

Error in producing data to Kafka using Erlang/ekaf

I'm trying to connect Kafka and produce some messages using Erlang/ekaf.
The code is a simple example explained in ekaf's README, but will exit when application:start is called.
Please note that I've used gen_icmp:ping to make sure the server running Kafka is accessible to this machine.
I have also run python script to produce some random message to this Kafka and it was successful, so most probably there is something I have missed in my Erlang code. :)
Source:
-module(kafka).
-compile(export_all).
run_test() ->
io:format("run_test: start.~n"),
pingKafka(),
try init_ekaf() of
_ -> io:format("run_test: ok~n")
catch
error:Msg -> io:format("run_test: error: ~p~n", [Msg]);
throw:Msg -> io:format("run_test: throw: ~p~n", [Msg]);
exit:Msg -> io:format("run_test: exit: ~p~n", [Msg])
end.
init_ekaf() ->
io:format("init_ekaf: start.~n"),
application:load(ekaf),
application:set_env(ekaf, ekaf_bootstrap_broker, {"kafka.dev", 9092}),
ok = application:start(ekaf),
io:format("init_ekaf: started.~n"),
Topic = <<"foobar">>,
ekaf:produce_sync(Topic, <<"some data">>),
io:format("init_ekaf: message sent.~n"),
ok.
pingKafka() ->
Res = gen_icmp:ping("kafka.dev"),
io:format("pingKafka: ~p.~n", [Res]),
ok.
Output:
run_test: start.
pingKafka: [{ok,"kafka.dev",
{192,168,0,51},
{192,168,0,51},
{12343,0,64,130},
<<" !\"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJK">>}].
init_ekaf: start.
run_test: error: {badmatch,{error,{not_started,gproc}}}
run_test: end.
After reading existing tests in the repository again, I find out that gproc application also need to be started before starting ekaf.
So by adding:
application:start(gproc)
right before application:start(ekaf), the problem solved.
P.S: Found another way to solve the issue which is calling application:ensure_all_started(ekaf) instead of application:start(ekaf).
As mentioned in document, ensure_all_started is an equivalent to calling start/1,2 repeatedly on all dependencies that are not yet started for an application

[Cowboy-Erlang]: Error when pin-pointing to localhost:8080 using provided cowboy example web_server

I am trying out a cowboy example provided by this github repository:
https://github.com/ninenines/cowboy/tree/master/examples/web_server
I build the release successfully using erlang.mk and run the following command, which opens the Erlang shell in my linux terminal:
$ ./_rel/web_server_example/bin/web_server_example console
But when I then open http://localhost:8080 in my web-browser, I get the following error report:
=ERROR REPORT==== 26-Nov-2014::14:33:48 === Error in process <0.166.0> on node 'web_server_example#127.0.0.1' with exit value:
{function_clause,[{cowboy_req,ensure_response,[{ok,{http_req,#Port<0.454>,ranch_tcp,keepalive,<0.166.0>,<<3
bytes>>,'HTTP/1.1',{{127,0,0,1},57150},<<9 bytes>>,undefined,8080,<<1
byte>>,undefined,<<0 bytes>>,undefined,undefined,[{<<4 bytes>>,<<14
bytes>>},{<<10 bytes>>,<<10 bytes>>},{<<13 bytes>>,<<9 bytes>>},{<<6
bytes>>,<<74 bytes>>},{<<10 bytes>>,<<104 bytes>>},{<<15 bytes>>,<<19
bytes>>},{<<15 bytes>>,<<35 bytes>>}],[{<<10 bytes>>,[<<10
bytes>>]}],undefined,[],waiting,<<0 bytes>>,undefined...
=ERROR REPORT==== 26-Nov-2014::14:33:48 === Ranch listener http had connection process started with cowboy_protocol:start_link/4 at
<0.166.0> exit with reason:
{function_clause,[{cowboy_req,ensure_response,[{ok,{http_req,#Port<0.454>,ranch_tcp,keepalive,<0.166.0>,<<"GET">>,'HTTP/1.1',{{127,0,0,1},57150},<<"localhost">>,undefined,8080,<<"/">>,undefined,<<>>,undefined,undefined,[{<<"host">>,<<"localhost:8080">>},{<<"connection">>,<<"keep-alive">>},{<<"cache-control">>,<<"max-age=0">>},{<<"accept">>,<<"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8">>},{<<"user-agent">>,<<"Mozilla/5.0
(X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/39.0.2171.65 Safari/537.36">>},{<<"accept-encoding">>,<<"gzip,
deflate,
sdch">>},{<<"accept-language">>,<<"sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4">>}],[{<<"connection">>,[<<"keep-alive">>]}],undefined,[],waiting,<<>>,undefined,false,done,[],<<>>,undefined}},204],[{file,"src/cowboy_req.erl"},{line,1009}]},{cowboy_protocol,next_request,3,[{file,"src/cowboy_protocol.erl"},{line,454}]}]}
This is "src/cowboy_protocol.erl" around line 454:
-spec next_request(cowboy_req:req(), #state{}, any()) -> ok.
next_request(Req, State=#state{req_keepalive=Keepalive, timeout=Timeout},
HandlerRes) ->
cowboy_req:ensure_response(Req, 204),
%% If we are going to close the connection,
%% we do not want to attempt to skip the body.
case cowboy_req:get(connection, Req) of
close ->
terminate(State);
_ ->
%% Skip the body if it is reasonably sized. Close otherwise.
Buffer = case cowboy_req:body(Req) of
{ok, _, Req2} -> cowboy_req:get(buffer, Req2);
_ -> close
end,
%% Flush the resp_sent message before moving on.
if HandlerRes =:= ok, Buffer =/= close ->
receive {cowboy_req, resp_sent} -> ok after 0 -> ok end,
?MODULE:parse_request(Buffer,
State#state{req_keepalive=Keepalive + 1,
until=until(Timeout)}, 0);
true ->
terminate(State)
end
end.
And the webb_server_app.erl file:
%% Feel free to use, reuse and abuse the code in this file.
%% #private
-module(web_server_app).
-behaviour(application).
%% API.
-export([start/2]).
-export([stop/1]).
%% API.
start(_Type, _Args) ->
Dispatch = cowboy_router:compile([
{'_', [
{"/[...]", cowboy_static, {priv_dir, web_server, "", [
{mimetypes, cow_mimetypes, all},
{dir_handler, directory_handler}
]}}
]}
]),
{ok, _} = cowboy:start_http(http, 100, [{port, 8080}], [
{env, [{dispatch, Dispatch}]},
{middlewares, [cowboy_router, directory_lister, cowboy_handler]}
]),
web_server_sup:start_link().
stop(_State) ->
ok.
Does anyone have any suggestion as to what is exactly causing this problem, and how to solve it? Thanks.
EDIT:
I can confirm that the fault was in the Erlang OTP version R16B02 in my case. Changing to the latest Erlang release (17.3), as well as resolving missing file dependencies that arose during the configuration stage (with the solutions in the following link):
https://sites.google.com/site/comptekkia/erlang/how-to-install-erlang-on-ubuntu-10-10
Solved the problem(s). The web_server example runs without error now.
The error says function clause, so the arguments to cowboy_req:ensure_response/2 must be wrong. And indeed they are, because first argument is {ok, Request} instead of Request. You have to trace back, which function called next_request/3 with bad argument, because it clearly should be called without ok.
Probably somewhere at the end, you will find something like:
Req = some_function(...)
And you will need to change it to:
{ok, Req} = some_function(...)
Good luck and happy bug hunting :D
UPDATE: I just cloned the repo and it works fine for me. I got the directory listing, so it is not bug in cowboy, but somewhere in user code.

erlang rfc4627 decode error

I am trying to decode json data in yaws, getting an error which is not clear to identify the issue. The Json data is
{
"airport": [
{"airport": "MAA"},
{"city": "Chennai"},
{"country": "India"},
{"name": "Anna International Airport"}
]
}
The command i am using is
{ok, Json, _} = rfc4627:decode(Arg#arg.clidata).
The error is
Exception: undef Req:
{http_request,'POST',{abs_path,"/sample/rest.yaws"},{1,1}} Stack:
[{rfc4627,decode,
[<<"{\n \"airport\": [\n {\"airport\": \"MAA\"},\n {\"city\": \"Chennai\"},\n {\"country\":
\"India\"},\n {\"name\": \"Anna International Airport\"}\n
]\n}">>],
[]},
You're getting an undef exception, indicating that you're calling an undefined function. The error shows the stack, and at the top of the stack is the rfc4627:decode/1 function; that's the one that's undefined.
My guess is that your load path does not include the directory where you've stored the compiled rfc4627 module. You can add that directory to the Yaws load path by modifying your yaws.conf file and adding something like the following to the global config section (near the top):
ebin_dir = /path/to/where/rfc4627/is/stored
Note that you're allowed to have multiple ebin_dir settings; each is added to the load path.

Meck behaving strangely for multiple mocked modules

I have following module
-module(bhavcopy_downloader).
-export([download/2]).
download(From, SaveTo) ->
{ok, {{Status, _}, _, Body}} = lhttpc:request(From, "GET", [], infinity),
case Status of
200 -> file:write(SaveTo, Body),
true;
_ -> false
end.
And following tests for the above code
file_download_test_() ->
{foreach,
fun() ->
meck:new(lhttpc)
meck:new(file, [unstick])
end,
fun(_) ->
meck:unload(file),
meck:unload(lhttpc)
end,
{"saves the file at specified location",
fun() ->
meck:expect(lhttpc, request, 4, {ok, {{200, "OK"}, [], <<"response">>}}),
meck:expect(file, write_file, fun(Path, Data) ->
?assertEqual(Path, "~/Downloads/data-downloader/test.html"),
?assertEqual(Data, <<"response">>)
end),
?assertEqual(true, bhavcopy_downloader:download("http://google.com", "~/Downloads/data-downloader/test.html")),
?assert(meck:validate(file))
end}]
}.
When I run the tests I get following error (only part of the error pasted below for brevity). Looking at the error below, I am kind of feeling that file module is not being mocked (or the mock of file module being overridden when I set the other mock using meck:new(lhttpc). What could be going wrong here?
=ERROR REPORT==== 16-Feb-2013::20:17:24 ===
** Generic server file_meck terminating
** Last message in was {'EXIT',<0.110.0>,
{compile_forms,
{error,
[{[],
[{none,compile,
{crash,beam_asm,
{undef,
[{file,get_cwd,[],[]},
{filename,absname,1,
[{file,"filename.erl"},{line,67}]},
{compile,beam_asm,1,
[{file,"compile.erl"},{line,1245}]},
{compile,'-internal_comp/4-anonymous-1-',2,
[{file,"compile.erl"},{line,273}]},
{compile,fold_comp,3,
[{file,"compile.erl"},{line,291}]},
{compile,internal_comp,4,
[{file,"compile.erl"},{line,275}]},
{compile,'-do_compile/2-anonymous-0-',2,
[{file,"compile.erl"},{line,152}]}]}}}]}],
[{"src/lhttpc_types.hrl",
[{31,erl_lint,{new_builtin_type,{boolean,0}}},
{31,erl_lint,{renamed_type,bool,boolean}}]}]}}}
This is a catch 22 in Meck, caused by the fact that Meck uses the Erlang compiler, which in turns uses the file module. When Meck tries recompile the file module it needs the file module (through the compiler) and thus crashes.
As of this moment, Meck doesn't handle mocking the file module. Your best alternative is to wrap the file module calls in another module and mock this module instead.
(It is theoretically possible to fix this in Meck by using the internals of the compiler and the code server instead, for example erlang:load_module/2, however this is quite tricky and needs to be designed and tested well)

Resources