Streaming a file using hackney - erlang

I'm trying to stream a multipart form using hackney.
This is my current code:
method = :post
path = "https://httpbin.org/post"
req_headers = [
{"Content-Type", "multipart/form-data"}
]
{:ok, pid} = :hackney.request(method, path, req_headers, :stream, [])
:hackney.send_multipart_body(pid, {:file, "/Users/gmile/.vimrc"})
{:ok, _status, _headers, pid} = :hackney.start_response(pid)
{:ok, body} = :hackney.body(pid)
On line with :hackney.send_multipart_body/2, I see get exception:
** (ArgumentError) argument error
:erlang.byte_size(nil)
(hackney) src/hackney_multipart.erl:134: :hackney_multipart.mp_header/2
(hackney) src/hackney_multipart.erl:239: :hackney_multipart.mp_file_header/2
(hackney) src/hackney_request.erl:222: :hackney_request.stream_multipart/2
(hackney) src/hackney.erl:432: anonymous fn/2 in :hackney.send_multipart_body/2
read_and_stream.exs:13: (file)
Question: what am I doing wrong?
There's clearly a bug in my code, but since there's little documentation about :hackney.send_multipart_body/2 I can't wrap my head around this.
I started debugging.
Here's the execution path from the stacktrace, in the call order:
https://github.com/benoitc/hackney/blob/master/src/hackney.erl#L432
https://github.com/benoitc/hackney/blob/master/src/hackney_request.erl#L222
https://github.com/benoitc/hackney/blob/master/src/hackney_multipart.erl#L239
https://github.com/benoitc/hackney/blob/master/src/hackney_multipart.erl#L134
If I understand correctly, there should a Boundary variable set in the HTTP client process. However I can't seem to understand where it is being set.

In order to use :hackney.send_multipart_body/2, your request's body (fourth argument) should be :stream_multipart, not :stream:
{:ok, pid} = :hackney.request(method, path, req_headers, :stream_multipart, [])

Related

Erlang's dets doesn't create file with open_file

It's my first attempt to write anything in Erlang, so maybe the question is silly.
I'm writing a quite simple HTTP server using cowboy
db_name() -> "DB_test".
timestamp() ->
calendar:datetime_to_gregorian_seconds(calendar:universal_time()).
sha(Str) ->
<<X:256/big-unsigned-integer>> = crypto:hash(sha256, Str),
lists:flatten(io_lib:format("~64.16.0b", [X])).
handle_post(Req0, State) ->
Link = binary_to_list(cowboy_req:header(<<"link">>, Req0)),
dets:open_file(db_name(), []),
dets:insert(db_name(), {hashed_url(Link), Link, timestamp()}),
Req = cowboy_req:reply(200,
#{<<"content-type">> => <<"text/plain">>},
sha(Link),
Req0),
{ok, Req, State}.
The idea is that a POST HTTP request contains a 'link' header with some link. After recieving such request my server should store it's hash in dets table along with the link and its timestamp. The problem is that the "DB_test" file isn't created. Why?
Based on your example code, it's hard to say exactly why, since you're ignoring the return values from both dets:open_file/2 and dets:insert/2.
Both of them return different values for the success and failure cases; but do not throw exceptions.
See the official documentation for more details: http://erlang.org/doc/man/dets.html
The simplest solution to this is to crash the cowboy connection handling process in non-success cases. You can do that by doing something like the following:
{ok, Ref} = dets:open_file(db_name(), []),
ok = dets:insert(Ref, {hashed_url(Link), Link, timestamp()}),
This will crash with a badmatch exception in the failure cases, when the value returned cannot be pattern matched to the left-hand side of the statement, subsequently causing cowboy to return HTTP 500 to the client.
You'll then see details on what the actual error was in the stacktrace logged
A second solution would be to explicitly handle the failure cases, you can use the 'case' keyword for that.
An example would be something like:
case dets:open_file(db_name(), []) of
{ok, Ref} ->
do_success_things();
{error, Reason}=E ->
io:format("Unable to open database file: ~p~n", [E]),
do_failure_things();
end
For further reading, I'd highly recommend the Syntax in functions and Errors and exceptions chapters of Learn you some Erlang for great good: http://learnyousomeerlang.com/

What is this Elixir message means

I am writing Elixir to get record from remote nodes, I have write a module,
defmodule Connect do
def connect do
node_ap_dev_ejd = :'abc#abc.com'
:net_adm.ping(node_ap)
fbUsersFun = fn(x) -> :binary.part(x,{0,3}) == <<"*ab">> end
f = fn()-> :mnesia.dirty_select(:'cz_map',[{{:cz_map, :'$1', :'_',:'_',:'_'},[],[:'$1']}]) end
fbUserList = :rpc.call(node_ap_dev_ejd,:mnesia,:activity,[:async_dirty,f])
list = Enum.filter(fbUserList ,fbUsersFun)
length(list)
end
end
I can run the code if I put it in iex shell line by line, however if I compile the code and run Connect.connect , this error appear, I have no idea of it, please suggest
** (Protocol.UndefinedError) protocol Enumerable not implemented for
{:badrpc, {:EXIT, {:undef, [{#Function<1.96315226/0 in Connect.connect/0>, [], []}, {:mnesia_tm, :non_transaction, 5, [file: 'mnesia_tm.erl', line: 738]}]}}}
(elixir) lib/enum.ex:1: Enumerable.impl_for!/1
(elixir) lib/enum.ex:112: Enumerable.reduce/3
(elixir) lib/enum.ex:666: Enum.filter/2
second_function.ex:10: Connect.connect/0
It means that the Enumerable protocol is not implemented for the data {:badrpc, ...}.
Most likely, that error comes from this line:
list = Enum.filter(fbUserList ,fbUsersFun)
In that line, you're trying to filter fbUserList which I guess is {:badrpc, ...} instead of an enumerable. Tuples are not enumerables; lists and maps (and other things) are.
The solution probably lies in a case expression which checks the result returned by :rpc.call/4 in order to defend from errors:
case :rpc.call(node_ap_dev_ejd, :mnesia, :activity, [:async_dirty, f]) do
{:badrpc, _} -> raise "bad rpc error"
fbUserList -> Enum.filter(fbUserList, ...) # and so on
end
I'm having the same issue working mnesia with erlang, the error is because of the anonymous function "f", the thing is that the remote node does not recognize that function because it was created in another node.
EDIT:
I managed to solve the problem in erlang, I will show you how I did it in erlang, I don't know much about elixir but I´m sure if it can be done in erlang it will in elixir.
So this segment
f = fn()-> :mnesia.dirty_select(:'cz_map',[{{:cz_map, :'$1', :'_',:'_',:'_'},[],[:'$1']}]) end
fbUserList = :rpc.call(node_ap_dev_ejd,:mnesia,:activity,[:async_dirty,f])
In erlang is like this
f = fun()-> mnesia:dirty_select(cz_map,[{{cz_map, '$1', '_', '_', '_'},[],['$1']}]) end,
fbUserList = rpc:call(node_ap_dev_ejd, mnesia, activity, [async_dirty, f])
Instead declaring an anonymous fun you have to do something like this
fbUserList = rpc:call(node_ap_dev_ejd, mnesia, activity, [async_dirty, mnesia:dirty_select/2, [cz_map, [{{cz_map, '$1', '_', '_', '_'},[],['$1']}]]])
You can find a clear explanation here what kind of types can be sent on an erlang message?
I hope this information helps you.

How to get status code from result in Elixir

I'm using elixir and the httpc erlang module to ping a website. I can hit the website just fine but when I try to use status_code I get an argument error. I can see that the data is there in the error but I'm not sure why the error is being thrown.
What arguments am I supposed to be providing? I thought that this function only returned an integer but it looks like it's expecting something.
:inets.start
{:ok, result} = :httpc.request(:get, {'http://www.erlang.org', []}, [], [])
result.status_code()
** (ArgumentError) argument error
:erlang.apply({'HTTP/1.1', 200, 'OK'}, :status_code, [])
:httpc.request returns tuple which contains various information. The status code can be retrieved as follows.
:inets.start
{:ok, result} = :httpc.request(:get, {'http://www.erlang.org', []}, [], [])
{status_line, headers, body} = result
{http_version, status_code, reason_phrase} = status_line
The tuple structure is described as in the httpc doc. (http://erlang.org/doc/man/httpc.html#request-4)
If you want to use more elixir-style syntax (like result.status_code), one option would be to use some of the elixir libraries like httpoison or httpotion.

strange issue with simple rabbitmq erlang message direct queue getter

A basic RabbitMQ install with user guest/guest.
Given the following simple erlang test code for RabbitMQ (erlang client), I am getting the error bellow. The queue TEST_DIRECT_QUEUE exists and has 7 messages in it, and the RabbitMQ server is up and running.
If I try to create a new with a declare API command, I also get a similar error.
Overall the error appears during any the << channel:call >> command
Any thoughts ? Thanks.
=ERROR REPORT==== 16-Feb-2013::10:39:42 ===
Connection (<0.38.0>) closing: internal error in channel (<0.50.0>): shutdown
** exception exit: {shutdown,{gen_server,call,
[<0.50.0>,
{call,{'queue.declare',0,"TEST_DIRECT_QUEUE",false,false,
false,false,false,[]},
none,<0.31.0>},
infinity]}}
in function gen_server:call/3 (gen_server.erl, line 188)
in call from test:test_message/0 (test.erl, line 12)
==============================================
-module(test).
-export([test_message/0]).
-include_lib("amqp_client/include/amqp_client.hrl").
-record(state, {channel}).
test_message() ->
{ok, Connection} = amqp_connection:start(#amqp_params_network{}),
{ok, Channel} = amqp_connection:open_channel(Connection),
Get = #'basic.get'{queue = "TEST_DIRECT_QUEUE"},
{#'basic.get_ok'{}, Content} = amqp_channel:call(Channel, Get), <=== error here
#'basic.get_empty'{} = amqp_channel:call(Channel, Get),
amqp_channel:call(Channel, #'channel.close'{}).
I have identified the issue myself after some frustrating hours. Overall, let me confess to be upset with the vague tutorias and documentation about RabbitMQ.... Anyways, here is what the problem was:
1) Queue Names are supposed to be in binary form, therefore preceded by "<<" and superceded by ">>". For example : <<"my queue name">> ( quotes included as well )
2) In a different scenario where I was trying to create the queue with queue.declare, the fact that the queue already existed was not a problem, but the fact that the queue was durable and the queue.declare did not specify that set of parameters caused the program to throw an error and interrupt execution. This is an unfortunate behavior where normally, developers would expect the queue matching to be done simply by name and then proceed. So to fix that I had to specify the durable value.
Here is a simple working code:
-module(test).
-export([test/0]).
-include_lib("amqp_client/include/amqp_client.hrl").
test() ->
{ok, Connection} = amqp_connection:start(#amqp_params_network{}),
{ok, Channel} = amqp_connection:open_channel(Connection),
Declare = #'queue.declare'{queue = <<"TEST_DIRECT_QUEUE">>, durable = true},
#'queue.declare_ok'{} = amqp_channel:call(Channel, Declare),
Get = #'basic.get'{queue = <<"TEST_DIRECT_QUEUE">>, no_ack = true},
{#'basic.get_ok'{}, Content} = amqp_channel:call(Channel, Get),
#amqp_msg{payload = Payload} = Content.

can't update a customer with put in erlang

I'm trying to update a customer in my riak database and i get the error message below:
And I don't know what causing this error and what this error message means.
And the module I use is:
allowed_methods(Request, State) ->
{['PUT'], Request, State}.
content_types_accepted(Request, State) ->
{[{"application/json",to_json}], Request, State}.
The error
webmachine error: path="/customer/cus/update" {error, {error,undef, [{customer_update,to_json, [{wm_reqdata,'PUT',http,
{1,1}, "127.0.0.1", {wm_reqstate,#Port<0.6513>, {dict,4,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],
[[mediaparams,{"charset","UTF-8"}]], [],
[[resource_module|customer_update],
['content-type',116,101,120,116,47,104,116,109, 108]], [],
[['content-encoding',105,100,101,110,116,105,116, 121]],
[],[],[],[],[],[],[],[]}}},
undefined,"127.0.0.1",'REQDATA',undefined,undefined,
{wm_log_data,undefined, {1322,989559,450145}, 'PUT', {6,
{"content-length", {'Content-Length',"121"},
{"connection",{'Connection',"Keep-Alive"},nil,nil}, {"content-type",
{'Content-Type', "application/json; charset=UTF-8"}, nil, {"host",
{'Host',"localhost:8000"},
{"expect",{"Expect","100-Continue"},nil,nil}, {"user-agent",
{'User-Agent', "Apache-HttpClient/4.0.1 (java 1.5)"}, nil,nil}}}}},
"127.0.0.1","/updatecustomer", {1,1},
404,0,undefined,undefined,undefined}},
[],"/customer/cus/update","//customer/cus/update",
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
[],".",500,1073741824,67108864,[],[], {6, {"content-length",
{'Content-Length',"121"},
{"connection",{'Connection',"Keep-Alive"},nil,nil}, {"content-type",
{'Content-Type',"application/json; charset=UTF-8"}, nil, {"host",
{'Host',"localhost:8000"},
{"expect",{"Expect","100-Continue"},nil,nil}, {"user-agent",
{'User-Agent',"Apache-HttpClient/4.0.1 (java 1.5)"}, nil,nil}}}}},
not_fetched_yet,false,
{1,{"content-type",{"Content-Type","text/html"},nil,nil}}, <<>>,
["localhost"], 8000,[]}, undefined]},
{webmachine_resource,resource_call,3}, {webmachine_resource,do,3},
{webmachine_decision_core,resource_call,1},
{webmachine_decision_core,accept_helper,0},
{webmachine_decision_core,decision,1},
{webmachine_decision_core,handle_request,2},
{webmachine_mochiweb,loop,1}]}}
You should define to_json/2 function.
For example:
to_json(RD, Result) ->
{mochijson:encode(Result), RD, Result}.
Unfortunately I lack the reputation to comment on the answer by Ilya.
TLDR: prefix to_json with the name of the module where you defined it
Longer answer:
I'm defining to_json in another module
Looking at your content_types_accepted/2 call, you aren't specifying which module to_json resides in, hence the undef error. Erlang function calls are always MFA -> module:function(arguments), you can only omit the module if the function is in the same module.
See also the documentation on Erlang packages
The key to understanding this error is the part:
{error, {error,undef, [{customer_update,to_json, ...
Which reports an undef error. Errors of these kinds are described at:
http://www.erlang.org/doc/reference_manual/errors.html#id81244
And you can see that undef means we have an undefined function. The error is due to a call customer_update:to_json(..) which then did not exist. That is the problem you have here.

Resources