Using elcloud_s3 to upload to files to S3 i - erlang

I am trying to use the erlang module erlcloud_s3 to upload a file to S3. For some reason that I have not been able to work out it does not work. I have included my code (with keys removed of course) and the response that I get when I try to run this. Can someone tell me what I am missing (or if there is a better AWS package for Erlang that actually has some docs or examples?)
-module(compose).
-define('ACCESS_KEY', "********************").
-define('SECRET_ACCESS_KEY', "****************************************").
-define('BUCKET', "zacharykessin").
-export([upload/2, upload_file/2]).
upload_file(Key, Path) ->
R = file:read_file(Path),
{ok, Binary} = R,
upload(Key, Binary).
upload(Key, Value) ->
A = erlcloud_ec2:configure(?ACCESS_KEY, ?SECRET_ACCESS_KEY),
error_logger:info_msg("~p:~p Settng up AWS ~p to S3 ~n",
[?MODULE, ?LINE, A]),
R = erlcloud_s3:put_object(?BUCKET, Key, Value, [], [{"Content-type", "image/jpeg"}]),
error_logger:info_msg("~p:~p Uploaded File ~p to S3 ~n",
[?MODULE, ?LINE, R]),
{ok, R}.
{noproc,
{gen_server,call,
[httpc_manager,
{request,
{request,undefined,<0.2.0>,0,https,
{"zacharykessin.s3.amazonaws.com",443},
"/test",[],put,
{http_request_h,undefined,"keep-alive",
"Thu, 15 Mar 2012 14:22:14 GMT",
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,
["AWS ","********************",58,
<<"1O6HYjq8RU8sqtD8oZd1T+bMNCE=">>],
undefined,undefined,
"zacharykessin.s3.amazonaws.com",
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,[],undefined,undefined,...},
{"application/octet_stream",
<<255,216,255,224,0,16,74,70,73,70,0,1,2,1,
1,44,1,44,0,0,255,225,25,59,69,120,105,...>>},
{http_options,"HTTP/1.1",infinity,true,
{essl,[]},
undefined,false,infinity,false},
"https://zacharykessin.s3.amazonaws.com/test",
[],none,[],1331821334636,undefined,undefined,
false}},
infinity]}}

Make sure to run
inets:start()
somewhere before running your code.
The error you are getting means that there is no such process (noproc) registered as httpc_manager, which is a part of inets library application.

Related

Erlang eunit test skipped (no code changed!)

I had a failing test. I made some changes, undid them, run my test again. Instead of failing, I get this.
All of my .beam files are present. My git status shows no changes.
=====================================================
Failed: 0. Skipped: 0. Passed: 0.
One or more tests were cancelled.
error
60> room:test().
=ERROR REPORT==== 14-Feb-2017::19:36:57 ===
** Generic server <0.1375.0> terminating
** Last message in was {join,#{auth => "auth",name => "Blandline"}}
** When Server state == {<0.1376.0>,<0.1379.0>,<0.1380.0>}
** Reason for termination ==
** {function_clause,
[{table,terminate,
[{{case_clause,{ok,#{x => "Blandline"}}},
[{table,handle_call,3,[{file,"table.erl"},{line,15}]},
{gen_server,try_handle_call,4,
[{file,"gen_server.erl"},{line,615}]},
{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]},
{<0.1376.0>,<0.1379.0>,<0.1380.0>}],
[{file,"table.erl"},{line,47}]},
{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,629}]},
{gen_server,terminate,7,[{file,"gen_server.erl"},{line,795}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
room: firstMove_test...*skipped*
undefined
*unexpected termination of test process*
::{function_clause,[{table,terminate,
[{{case_clause,{ok,#{x => "Blandline"}}},
[{table,handle_call,3,[...]},
{gen_server,try_handle_call,4,...},
{gen_server,handle_msg,...},
{proc_lib,...}]},
{<0.1376.0>,<0.1379.0>,<0.1380.0>}],
[{file,"table.erl"},{line,47}]},
{gen_server,try_terminate,3,
[{file,"gen_server.erl"},{line,629}]},
{gen_server,terminate,7,
[{file,"gen_server.erl"},{line,795}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]}
=======================================================
Failed: 0. Skipped: 0. Passed: 0.
One or more tests were cancelled.
error
s:s is just a shortcut for gen_server:call. I'm testing other modules from this module.
-module(room).
-behaviour(gen_server).
-compile(export_all).
-include_lib("eunit/include/eunit.hrl").
testRoom() ->
{ok, R} = room:go(),
{ok, T, _} = s:s(R, {join, #{name => "Blandline", auth => "auth"}}),
{ok, T, _} = s:s(R, {join, #{name => "Kreutzer", auth => "auth"}}),
{R, T}.
firstMove_test() ->
{R, T} = testRoom(),
{ok, #{actions := _, board := _, clock := _}} = s:s(R, {play, T, {take, x, {1,2}}, {"Blandline", "auth"}}).
badMove_test() ->
{R, T} = testRoom(),
{error, invalid_input} = s:s(R, {play, T, {take, x, {1,2,3}}, {"Blandline", "auth"}}).
badAuth_test() ->
{R, T} = testRoom(),
{error, invalid_auth} = s:s(R, {play, T, {take, x, {1,2}}, {"Blandline", "badauth"}}).
badTable_test() ->
{R, _} = testRoom(),
{error, bad_table} = s:s(R, {play, self(), {take, x, {1,2}}, {"Blandline", "badauth"}}).
As the error report is telling you:
[{{case_clause,{ok,#{x => "Blandline"}}},
[{table,handle_call,3,[{file,"table.erl"},{line,15}]},
You have, in file table.erl line 15 a case clause that doesn't match :-)
I think that the reason why the test started failing although you undid your changes is because you are running the tests from the Erlang shell and you have something (like a gen_server) still running that fails for some reasons.
Said in another way, your tests are not repeatable because either they don't teardown properly all the dependencies or because they require external dependencies like this gen_server to be started by hand in the Erlang shell before running the tests.
I strongly suggest to forget the Erlang shell to run tests. Use rebar3 to build and run the tests from a terminal. This will force you to take care of all the dependencies and to have robust tests. The time taken to make this work will be well spent :-)

YAWS responds with errors under light load. "Unhandled reply fr. do_recv() {error,econnaborted}"

Hello fellow Erlangers :)
Just another Erlang enthusiast playing with the language here. I have a very simple YAWS app module which works fine when accessed by single clients. However, if I try to span multiple clients to simulate light traffic, those clients start receiving errors. Any idea what can be causing it?
Code spawning the clients:
-module(concurrent).
-export([measure/0, get/0]).
get() ->
Result = httpc:request("http://localhost:8080/blah"),
io:format("Result: ~p.\n", [Result]).
get_spawner(0) -> io:format("Done.\n");
get_spawner(Times) ->
spawn(concurrent, get, []),
get_spawner(Times - 1).
measure() ->
io:fwrite("Starting inets...\n"),
inets:start(),
io:fwrite("Done\n"),
io:fwrite("Creating blah...\n"),
httpc:request(put, { "http://localhost:8080/blah", [], "", "" }, [], []),
io:fwrite("Done\n"),
get_spawner(9),
io:fwrite("Stopping inets...\n"),
inets:stop(),
io:fwrite("Done\n"),
init:stop().
I observed that it all works just fine for 8 concurrent clients, but when I span 9 or more clients, they start receiving the following message back from the server:
=ERROR REPORT==== 24-Jan-2017::12:30:04 ===
** Generic server httpc_manager terminating
** Last message in was {request,
{request,undefined,<0.74.0>,0,http,
{"localhost",8080},
"/blah",[],get,
{http_request_h,undefined,"keep-alive",
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,"localhost:8080",
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,[],undefined,undefined,undefined,
undefined,"0",undefined,undefined,
undefined,undefined,undefined,undefined,[]},
{[],[]},
{http_options,"HTTP/1.1",infinity,true,
{essl,[]},
undefined,false,infinity,false},
"http://localhost:8080/blah",[],none,[],
1485261004189,undefined,undefined,false}}
** When Server state == {state,[],httpc_manager__handler_db,
{cookie_db,undefined,8209},
httpc_manager__session_db,httpc_manager,
{options,
{undefined,[]},
{undefined,[]},
0,2,5,120000,2,disabled,false,inet,default,
default,[]}}
** Reason for termination ==
** {'EXIT',
{shutdown,
{gen_server,call,
[httpc_handler_sup,
{start_child,
[<0.61.0>,
{request,#Ref<0.0.6.64>,<0.74.0>,0,http,
{"localhost",8080},
"/blah",[],get,
{http_request_h,undefined,"keep-alive",undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,"localhost:8080",undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,[],undefined,
undefined,undefined,undefined,"0",undefined,
undefined,undefined,undefined,undefined,
undefined,[]},
{[],[]},
{http_options,"HTTP/1.1",infinity,true,
{essl,[]},
undefined,false,infinity,false},
"http://localhost:8080/blah",[],none,[],
1485261004189,undefined,undefined,false},
{options,
{undefined,[]},
{undefined,[]},
0,2,5,120000,2,disabled,false,inet,default,default,
[]},
httpc_manager]},
infinity]}}}
=ERROR REPORT==== 24-Jan-2017::12:30:04 ===
Error in process <0.74.0> with exit value:
{{case_clause,
{undefined,
{error,
{'EXIT',
{shutdown,
{gen_server,call,
[httpc_handler_sup,
{start_child,
[<0.61.0>,
{request,#Ref<0.0.6.64>,<0.74.0>,0,http,
{"localhost",8080},
"/blah",[],get,
{http_request_h,undefined,"keep-alive",
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,"localhost:8080",
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,[],
undefined,undefined,undefined,
undefined,"0",undefined,undefined,
undefined,undefined,undefined,
undefined,[]},
{[],[]},
{http_options,"HTTP/1.1",infinity,true,
{essl,[]},
undefined,false,infinity,false},
"http://localhost:8080/blah",[],none,[],
1485261004189,undefined,undefined,false},
{options,
{undefined,[]},
{undefined,[]},
0,2,5,120000,2,disabled,false,inet,default,
default,[]},
httpc_manager]},
infinity]}}}}}},
[{httpc,handle_request,9,[{file,"httpc.erl"},{line,574}]},
{concurrent,get,0,
[{file,
"c:/Users/piotr_justyna/Documents/rets/performance_tests/concurrent.erl"},
{line,5}]}]}
On the server side I am getting this:
=ERROR REPORT==== 24-Jan-2017::12:28:55 ===
Unhandled reply fr. do_recv() {error,econnaborted}
I'm sure it's something straightforward, just can't figure out what it is. Is it the number of concurrent connections that causes it? Can you help shed more light on this?
Regards,
Piotr
Your code spawns a number of httpc:request/1 calls, but then immediately afterwards it stops inets:
...
get_spawner(9),
io:fwrite("Stopping inets...\n"),
inets:stop(),
...
Because inets is shut down, requests that are still in progress are getting shut down prematurely, as you can see in the error report:
** Reason for termination ==
** {'EXIT',
{shutdown,
{gen_server,call,
[httpc_handler_sup,
You need to make the parent of all the spawned requests wait for those requests to finish before calling inets:stop/0. One way to do that would be to pass the parent pid to each spawned child and once httpc:request/1 returns, have the child send a message to the parent. The parent would spawn all N children, then sit in a loop waiting to receive N completion messages from the children, and only after receiving all completion messages would it call inets:stop/0.
The reason it works for 8 requests but not 9 is due to a race condition around the lack of waiting prior to calling inets:stop/0. The default size of the Yaws pool of request handling acceptor processes happens to be 8. When your code sends 8 requests, the pool handles it quickly enough to beat your code's call to inets:stop/0, but with 9, there's a slight delay as one acceptor handles two requests, and that slight timing change is enough to allow your inets:stop/0 call to clobber one or more of your client processes. You can modify the Yaws acceptor pool size via the acceptor_pool_size configuration variable.
If you're serious about load testing you might consider using an industrial-strength solution like tsung instead of rolling your own.

Error in producing data to Kafka using Erlang/ekaf

I'm trying to connect Kafka and produce some messages using Erlang/ekaf.
The code is a simple example explained in ekaf's README, but will exit when application:start is called.
Please note that I've used gen_icmp:ping to make sure the server running Kafka is accessible to this machine.
I have also run python script to produce some random message to this Kafka and it was successful, so most probably there is something I have missed in my Erlang code. :)
Source:
-module(kafka).
-compile(export_all).
run_test() ->
io:format("run_test: start.~n"),
pingKafka(),
try init_ekaf() of
_ -> io:format("run_test: ok~n")
catch
error:Msg -> io:format("run_test: error: ~p~n", [Msg]);
throw:Msg -> io:format("run_test: throw: ~p~n", [Msg]);
exit:Msg -> io:format("run_test: exit: ~p~n", [Msg])
end.
init_ekaf() ->
io:format("init_ekaf: start.~n"),
application:load(ekaf),
application:set_env(ekaf, ekaf_bootstrap_broker, {"kafka.dev", 9092}),
ok = application:start(ekaf),
io:format("init_ekaf: started.~n"),
Topic = <<"foobar">>,
ekaf:produce_sync(Topic, <<"some data">>),
io:format("init_ekaf: message sent.~n"),
ok.
pingKafka() ->
Res = gen_icmp:ping("kafka.dev"),
io:format("pingKafka: ~p.~n", [Res]),
ok.
Output:
run_test: start.
pingKafka: [{ok,"kafka.dev",
{192,168,0,51},
{192,168,0,51},
{12343,0,64,130},
<<" !\"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJK">>}].
init_ekaf: start.
run_test: error: {badmatch,{error,{not_started,gproc}}}
run_test: end.
After reading existing tests in the repository again, I find out that gproc application also need to be started before starting ekaf.
So by adding:
application:start(gproc)
right before application:start(ekaf), the problem solved.
P.S: Found another way to solve the issue which is calling application:ensure_all_started(ekaf) instead of application:start(ekaf).
As mentioned in document, ensure_all_started is an equivalent to calling start/1,2 repeatedly on all dependencies that are not yet started for an application

YAWs embedded with appmod not working for me

Alright, what am I doing wrong here. I'm trying the simple example of embedded YAWs from http://yaws.hyber.org/embed.yaws but with an appmod. I've added the my_app.erl file and compiled it. It works if not in embedded YAWs so I think it is specific to embedded.
-module(ybed).
-compile(export_all).
start() ->
{ok, spawn(?MODULE, run, [])}.
run() ->
Id = "embedded",
GconfList = [{ebin_dir, ["/Users/someuser/yawsembedded/ebin"]}],
Docroot = "/Users/someuser/yawstest",
SconfList = [{port, 8888},
{listen, {0,0,0,0}},
{docroot, Docroot},
{appmods, [{"/", my_app}]}
],
{ok, SCList, GC, ChildSpecs} =
yaws_api:embedded_start_conf(Docroot, SconfList, GconfList),
[supervisor:start_child(ybed_sup, Ch) || Ch <- ChildSpecs],
yaws_api:setconf(GC, SCList),
{ok, self()}.
Getting this Error:
ERROR erlang code threw an uncaught exception:
File: appmod:0
Class: error
Exception: undef
Req: {http_request,'GET',{abs_path,"/demo"},{1,1}}
Stack: [{my_app,out,
[{arg,#Port<0.2721>,
{{127,0,0,1},63720},
{headers,"keep-alive",
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"0.0.0.0:8888",undefined,undefined,undefined,undefined,
undefined,undefined,undefined,
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0",
undefined,[],undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
[{http_header,0,"Dnt",undefined,"1"},
{http_header,10,'Accept-Encoding',undefined,
"gzip, deflate"},
{http_header,11,'Accept-Language',undefined,"null"}]},
{http_request,'GET',{abs_path,"/demo"},{1,1}},
{http_request,'GET',{abs_path,"/demo"},{1,1}},
undefined,"/demo",undefined,undefined,
"/Users/someuser/yawstest","/",
"/Users/someuser/yawstest/demo",undefined,undefined,
<0.63.0>,[],"/","/",undefined}],
[]},
{yaws_server,deliver_dyn_part,8,
[{file,"yaws_server.erl"},{line,2818}]},
{yaws_server,aloop,4,[{file,"yaws_server.erl"},{line,1232}]},
{yaws_server,acceptor0,2,[{file,"yaws_server.erl"},{line,1068}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]
The stack trace shows that your my_app:out/1 function is getting called, but you're getting an undef exception. This is occurring because the runtime can't find the my_app:out/1 function, which means either it can't find the module or the module exists but does not export an out/1 function. For example, I was able to duplicate the error using the example code by not providing a my_app module.
First, make sure your my_app.erl file exports an out/1 function. Here's a trivial one that just returns a 405 error for all requests:
-module(my_app).
-export([out/1]).
out(_Arg) ->
{status, 405}.
Compile your my_app.erl file and put the compiled my_app.beam file either in a load path already known to the Erlang runtime, or in a directory you add to the load path. In your code it appears you're trying the latter approach, since you're specifically adding an ebin_dir with this Yaws global configuration directive:
GconfList = [{ebin_dir, ["/Users/someuser/yawsembedded/ebin"]}],
You need to verify that the /Users/someuser/yawsembedded/ebin directory exists, and that the compiled my_app.beam file is located there.

[Cowboy-Erlang]: Error when pin-pointing to localhost:8080 using provided cowboy example web_server

I am trying out a cowboy example provided by this github repository:
https://github.com/ninenines/cowboy/tree/master/examples/web_server
I build the release successfully using erlang.mk and run the following command, which opens the Erlang shell in my linux terminal:
$ ./_rel/web_server_example/bin/web_server_example console
But when I then open http://localhost:8080 in my web-browser, I get the following error report:
=ERROR REPORT==== 26-Nov-2014::14:33:48 === Error in process <0.166.0> on node 'web_server_example#127.0.0.1' with exit value:
{function_clause,[{cowboy_req,ensure_response,[{ok,{http_req,#Port<0.454>,ranch_tcp,keepalive,<0.166.0>,<<3
bytes>>,'HTTP/1.1',{{127,0,0,1},57150},<<9 bytes>>,undefined,8080,<<1
byte>>,undefined,<<0 bytes>>,undefined,undefined,[{<<4 bytes>>,<<14
bytes>>},{<<10 bytes>>,<<10 bytes>>},{<<13 bytes>>,<<9 bytes>>},{<<6
bytes>>,<<74 bytes>>},{<<10 bytes>>,<<104 bytes>>},{<<15 bytes>>,<<19
bytes>>},{<<15 bytes>>,<<35 bytes>>}],[{<<10 bytes>>,[<<10
bytes>>]}],undefined,[],waiting,<<0 bytes>>,undefined...
=ERROR REPORT==== 26-Nov-2014::14:33:48 === Ranch listener http had connection process started with cowboy_protocol:start_link/4 at
<0.166.0> exit with reason:
{function_clause,[{cowboy_req,ensure_response,[{ok,{http_req,#Port<0.454>,ranch_tcp,keepalive,<0.166.0>,<<"GET">>,'HTTP/1.1',{{127,0,0,1},57150},<<"localhost">>,undefined,8080,<<"/">>,undefined,<<>>,undefined,undefined,[{<<"host">>,<<"localhost:8080">>},{<<"connection">>,<<"keep-alive">>},{<<"cache-control">>,<<"max-age=0">>},{<<"accept">>,<<"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8">>},{<<"user-agent">>,<<"Mozilla/5.0
(X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/39.0.2171.65 Safari/537.36">>},{<<"accept-encoding">>,<<"gzip,
deflate,
sdch">>},{<<"accept-language">>,<<"sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4">>}],[{<<"connection">>,[<<"keep-alive">>]}],undefined,[],waiting,<<>>,undefined,false,done,[],<<>>,undefined}},204],[{file,"src/cowboy_req.erl"},{line,1009}]},{cowboy_protocol,next_request,3,[{file,"src/cowboy_protocol.erl"},{line,454}]}]}
This is "src/cowboy_protocol.erl" around line 454:
-spec next_request(cowboy_req:req(), #state{}, any()) -> ok.
next_request(Req, State=#state{req_keepalive=Keepalive, timeout=Timeout},
HandlerRes) ->
cowboy_req:ensure_response(Req, 204),
%% If we are going to close the connection,
%% we do not want to attempt to skip the body.
case cowboy_req:get(connection, Req) of
close ->
terminate(State);
_ ->
%% Skip the body if it is reasonably sized. Close otherwise.
Buffer = case cowboy_req:body(Req) of
{ok, _, Req2} -> cowboy_req:get(buffer, Req2);
_ -> close
end,
%% Flush the resp_sent message before moving on.
if HandlerRes =:= ok, Buffer =/= close ->
receive {cowboy_req, resp_sent} -> ok after 0 -> ok end,
?MODULE:parse_request(Buffer,
State#state{req_keepalive=Keepalive + 1,
until=until(Timeout)}, 0);
true ->
terminate(State)
end
end.
And the webb_server_app.erl file:
%% Feel free to use, reuse and abuse the code in this file.
%% #private
-module(web_server_app).
-behaviour(application).
%% API.
-export([start/2]).
-export([stop/1]).
%% API.
start(_Type, _Args) ->
Dispatch = cowboy_router:compile([
{'_', [
{"/[...]", cowboy_static, {priv_dir, web_server, "", [
{mimetypes, cow_mimetypes, all},
{dir_handler, directory_handler}
]}}
]}
]),
{ok, _} = cowboy:start_http(http, 100, [{port, 8080}], [
{env, [{dispatch, Dispatch}]},
{middlewares, [cowboy_router, directory_lister, cowboy_handler]}
]),
web_server_sup:start_link().
stop(_State) ->
ok.
Does anyone have any suggestion as to what is exactly causing this problem, and how to solve it? Thanks.
EDIT:
I can confirm that the fault was in the Erlang OTP version R16B02 in my case. Changing to the latest Erlang release (17.3), as well as resolving missing file dependencies that arose during the configuration stage (with the solutions in the following link):
https://sites.google.com/site/comptekkia/erlang/how-to-install-erlang-on-ubuntu-10-10
Solved the problem(s). The web_server example runs without error now.
The error says function clause, so the arguments to cowboy_req:ensure_response/2 must be wrong. And indeed they are, because first argument is {ok, Request} instead of Request. You have to trace back, which function called next_request/3 with bad argument, because it clearly should be called without ok.
Probably somewhere at the end, you will find something like:
Req = some_function(...)
And you will need to change it to:
{ok, Req} = some_function(...)
Good luck and happy bug hunting :D
UPDATE: I just cloned the repo and it works fine for me. I got the directory listing, so it is not bug in cowboy, but somewhere in user code.

Resources