Related
So I got a hook in my code that says if an offline message is received, it will start up this function.
offline_msg_received(From, To, Packet) ->
The Packet has code looking like this:
{message,<<"purple2d957668">>,chat,<<>>,undefined,{jid,<<"admin">>,<<"localhost">>,<<"5280">>,<<"admin">>,<<"localhost">>,<<"5280">>},[],[{text,<<>>,<<"Hello">>}],undefined,[{xmlel,<<"active">>,[{<<"xmlns">>,<<"http://jabber.org/protocol/chatstates">>}],[]}],#{}}
What I wanted out of this is the part containing my message. Which is:
[{text,<<>>,<<"Hello">>}]
I could get this part out with a pattern matching against the whole package variable. However, when I tried a pattern matching with my message, it says bad argument. I know that if I specify a variable with just
{text,<<>>,<<"Hello">>}
then I can pattern match against that. From my understanding, seeing [ ] around something means it is a list, or a string.
I can show some of the code here.
{P1, P2, P3, P4, P5, P6, P7, P8, P9, P10, P11} = Packet, %% P8 is the message I want.
?INFO_MSG("----------------------P8: ~p~n", P8), %% This shows {text,<<>>,<<"Hello">>}
{A, B, C} = P8, %% This gives me a badmatch error.
Now, if I were to instead do this manually in the code shell:
{text,<<>>,<<"Hello">>} = P8,
{A, B, C} = P8,
Then the code works, and puts <<"Hello">> in the C variable.
My guess is that the [ ] around it in the "Package" variable has something to do with my badmatch error, but for some reason those symbols don't appear when I print out P8. Does anyone have a good suggestion why it doesn't work? I am guessing it has something to do with the hidden [ ], since otherwise it looks identical to my shell code test.
Thanks. This is my first time asking questions here, so I hope I didn't do anything wrong.
The reason you're not seeing the square brackets around the value in your INFO_MSG call is that INFO_MSG expects a list of values, not just a single value. In the code above it happens to work out, since the format string only contains a single directive, and the value is a list of one element, so that's why the brackets are silently stripped out.
The INFO_MSG macro works the same as io:format, so you could try this out in the shell:
%% two arguments
io:format("~p ~p\n", [foo, bar]).
%% one argument - we still need square brackets
io:format("~p\n", [foo]).
You're doing this the hard way :) message and text are records defined in xmpp_codec.hrl. If you don't have the record definitions loaded in the shell, records look like a tuple with lots of elements, but if you load the record definitions, your message looks like this instead:
1> rr("include/xmpp_codec.hrl").
[address,addresses,adhoc_actions,adhoc_command,adhoc_note,
bind,block,block_list,bob_data,bookmark_conference,
bookmark_storage,bookmark_url,bytestreams,caps,
carbons_disable,carbons_enable,carbons_private,
carbons_received,carbons_sent,chatstate,client_id,compress,
compress_failure,compressed,compression,csi,db_feature,
db_result,db_verify|...]
2> {message,<<"purple2d957668">>,chat,<<>>,undefined,{jid,<<"admin">>,<<"localhost">>,<<"5280">>,<<"admin">>,<<"localhost">>,<<"5280">>},[],[{text,<<>>,<<"Hello">>}],undefined,[{xmlel,<<"active">>,[{<<"xmlns">>,<<"http://jabber.org/protocol/chatstates">>}],[]}],#{}}.
#message{id = <<"purple2d957668">>,type = chat,lang = <<>>,
from = undefined,
to = {jid,<<"admin">>,<<"localhost">>,<<"5280">>,
<<"admin">>,<<"localhost">>,<<"5280">>},
subject = [],
body = [#text{lang = <<>>,data = <<"Hello">>}],
thread = undefined,
sub_els = [{xmlel,<<"active">>,
[{<<"xmlns">>,
<<"http://jabber.org/protocol/chatstates">>}],
[]}],
meta = #{}}
The data you are looking for is in the body field.
Let's assign this record to the variable M and extract the body field:
4> Body = M#message.body.
[#text{lang = <<>>,data = <<"Hello">>}]
So that's a list containing one text record. Let's extract the data field from that:
6> [#text{data = Data}] = Body.
[#text{lang = <<>>,data = <<"Hello">>}]
7> Data.
<<"Hello">>
That's how it works in the shell - load the record definitions with the rr function. When writing a module that uses record definitions, you'd need to include the hrl file:
-include_lib("xmpp/include/xmpp_codec.hrl").
The -include_lib directory will look for the xmpp application on the load path, and search for the file xmpp_codec.hrl inside its include directory. You may need to specify the code path to the compiler, something like erlc -pa path/to/xmpp/ebin my_module.erl - note that the ebin directory inside the application should be on the code path.
I'm trying to save offline ejabberd messages in Riak. I earlier had problems connecting to Riak but those are resolved now with the help of this forum. But now, with my limited Erlang / ejabberd understanding, I'm failing to get the ejabberd packet saved as a string and then put on Riak. Essentially, when the offline_message_hook is latched, I take the Packet argument and then want to put a backslash for each double quote, so that I can then take this revised string and save as a string value on Riak. However I seem to be struggling with modifying the incoming packet to replace the " chars with \".
Is this the right approach? Or am I missing something here in my design? My application relies on the xml format, so should I instead parse the packet using the p1_xml module and reconstruct the xml using the extracted data elements before storing it on Riak.
Apologies for the very basic and multiple questions but appreciate if someone can throw some light here!
The code that I use to try and replace the " with \" in the incoming packet is: (it doesnt quite work):
NewPacket = re:replace(Packet, "\"", "\\\"", [{return, list}, global]),
So essentially, I would be passing the NewPacket as a value to my Riak calls.
ejabberd is compliant with Riak and it is already storing packets in Riak. For example, mod_offline does that.
You can look directly in ejabberd code to find how to do that. For example, in mod_offline, here is how ejabberd store the offline message:
store_offline_msg(Host, {User, _}, Msgs, Len, MaxOfflineMsgs,
riak) ->
Count = if MaxOfflineMsgs =/= infinity ->
Len + count_offline_messages(User, Host);
true -> 0
end,
if
Count > MaxOfflineMsgs ->
discard_warn_sender(Msgs);
true ->
lists:foreach(
fun(#offline_msg{us = US,
timestamp = TS} = M) ->
ejabberd_riak:put(M, offline_msg_schema(),
[{i, TS}, {'2i', [{<<"us">>, US}]}])
end, Msgs)
end.
The code of ejabberd_riak:put/3 is:
put(Rec, RecSchema, IndexInfo) ->
Key = encode_key(proplists:get_value(i, IndexInfo, element(2, Rec))),
SecIdxs = [encode_index_key(K, V) ||
{K, V} <- proplists:get_value('2i', IndexInfo, [])],
Table = element(1, Rec),
Value = encode_record(Rec, RecSchema),
case put_raw(Table, Key, Value, SecIdxs) of
ok ->
ok;
{error, _} = Error ->
log_error(Error, put, [{record, Rec},
{index_info, IndexInfo}]),
Error
end.
put_raw(Table, Key, Value, Indexes) ->
Bucket = make_bucket(Table),
Obj = riakc_obj:new(Bucket, Key, Value, "application/x-erlang-term"),
Obj1 = if Indexes /= [] ->
MetaData = dict:store(<<"index">>, Indexes, dict:new()),
riakc_obj:update_metadata(Obj, MetaData);
true ->
Obj
end,
catch riakc_pb_socket:put(get_random_pid(), Obj1).
You should have already the proper API to do what you want in ejabberd regarding Riak packet storage.
This code is an excerpt from this book.
count_characters(Str) ->
count_characters(Str, #{}).
count_characters([H|T], #{ H => N }=X) ->
count_characters(T, X#{ H := N+1 });
count_characters([H|T], X) ->
count_characters(T, X#{ H => 1 });
count_characters([], X) ->
X.
So,
1> count_characters("hello").
#{101=>1,104=>1,108=>2,111=>1}
What I understand from this is that, count_characters() takes an argument hello, and place it to the first function, i.e count_characters(Str).
What I don't understand is, how the string characters are converted into ascii value without using $, and got incremented. I am very new to erlang, and would really appreciate if you could help me understand the above. Thank you.
In erlang the string literal "hello" is just a more convenient way of writing the list [104,101,108,108,111]. The string format is syntactic sugar and nothing erlang knows about internally. An ascii string is internally string is internally stored as a list of 32-bit integers.
This also becomes confusing when printing lists where the values happen to be within the ascii range:
io:format("~p~n", [[65,66]]).
will print
"AB"
even if you didn't expect a string as a result.
As said previously, there is no string data type in Erlang, it uses the internal representation of an integer list, so
"hello" == [$h,$e,$l,$l,$o] == [104|[101|[108|[108|[111|[]]]]]]
Which are each a valid representation of an integer list.
To make the count of characters, the function use a new Erlang data type: a map. (available only since R17)
A map is a collection of key/value pairs, in your case the keys will be the characters, and the values the occurrence of each characters.
The function is called with an empty map:count_characters(Str, #{}).
Then it goes recursively through the list, and for each head H, 2 cases are posible:
The character H was already found, then the current map X will match with the pattern #{ H => N } telling us that we already found N times H, so we continue the recursion with the rest of the list and a new map where the value associated to H is now N+1: count_characters(T, X#{ H := N+1 }.
The character H is found for the first time, then we continue the recursion with the rest of the list and a new map where the key/value pair H/1 is added: count_characters(T, X#{ H => 1 }).
When the end of the list is reached, simply return the map: count_characters([], X) -> X.
I want to do two things: Convert IP Address inputs into CIDR
Here are some example inputs:
1.1.1.1
192.168.*.* #=> 192.168.0-255.0-255
192.168.1.2-20
1.1.1-10.1-100
Check if a given IP Address falls into any CIDR. This must be a very fast query, as it's a very common lookup in my web app. I'm thinking of doing something like this:
def matches?(request)
valid = #ips.select {|cidr| cidr.contains?(request.remote_ip) }
!valid.empty?
end
I think converting IP ranges into CIDR will let lookups be faster than what we're doing now, which is breaking the IP's into integer octets. We then index the first two sets of octets to partially match against IP's. Another option might be converting everything to ints and doing comparisons that way. I'd convert to ints with something like this IPAddr.new("1.1.1.1").to_i but then I'd need to store an upper and lower IP for each range instead of just a single CIDR.
Please let me know if I am overlooking any mainstream approaches, popular gems or repo's. Thanks!
Well, to get the CIDR notation of a range, you need an IP and the number of network bits (calculated from the netmask).
To enumerate the addresses of a given range, you can use the NetAddr (< 2.x) gem.
p NetAddr::CIDR.create('192.168.1.0/24').enumerate
=> ['192.168.1.0', '192.168.1.1', '192.168.1.2'... '192.168.1.255']
You can also calculate the bits from the netmask on the fly:
mask_int = NetAddr.netmask_to_i('255.255.255.0')
p NetAddr.mask_to_bits(mask_int)
=> 24
And to create a range based on two IPs:
lower = NetAddr::CIDR.create('192.168.1.1')
upper = NetAddr::CIDR.create('192.168.1.10')
p NetAddr.range(lower, upper)
=> ['192.168.1.2', '192.168.1.3'... '192.168.1.9']
So now that you can create a CIDR range, you can check to see if an IP is a part of it:
cidr = NetAddr::CIDR.create('192.168.1.0/24')
p cidr.contains?('192.168.1.10')
=> true
I suspect everything you need is in IPAddr. I use this to see if the remote IP is coming from a private network:
['127.0.0.0/8', '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16', '192.168.10.0/8'
].none?{|block| IPAddr.new(block) === request.remote_ip}
Maybe I'm misunderstanding the question, but it seems that one aspect of this question has not been addressed, that is, converting a range of ip addresses to one or more CIDR entries.
I use the following approach to lookup suspicious ip activity on my firewall, and if it is in a country that I'm not interested in allowing access (you know who you are) I use whois to lookup the address range, and then calculate the merged CIDRs as follows,
whois xxx.yyy.zzz.123
# find address range for this ip
range="xxx.yyy.zzz.0-xxx.yyy.zzz.255".split(/\s*-\s*/)
lower=range[0]
upper=range[1]
ip_net_range = NetAddr.range(lower, upper, :Inclusive => true, :Objectify => true)
cidrs = NetAddr.merge(ip_net_range, :Objectify => true)
This is a example on an internal network, but it is trivial to extend to a public ip block,
whois 192.168.1.3
range="192.168.0.0 - 192.168.255.255".split(/\s*-\s*/)
upper=range[0]
lower=range[1]
ip_net_range = NetAddr.range(lower, upper, :Inclusive => true, :Objectify => true)
cidrs = NetAddr.merge(ip_net_range, :Objectify => true)
p cidrs
[192.168.0.0/16]
Then I can pass that CIDR to my firewall software (shorewall) to have it dynamically drop that cidr(s).
You can use NetAddr (> 2.x.x) gem.
Use parse class method to create a IPv4Net object from its string representation. It will default is a /32 netmask if not specified.
private_ips = NetAddr::IPv4Net.parse('192.168.0.0/16')
Now you use rel method to determine the relationship with another IPv4Net object.
For example, if we would like to know if '2.3.4.5/31' is part of private_ips
private_ips.rel(NetAddr::IPv4Net.parse('2.3.4.5/31')).eql?(1) || private_ips.rel(NetAddr::IPv4Net.parse('2.3.4.5/31')).eql?(0)
=> false
private_ips.rel(NetAddr::IPv4Net.parse('192.168.0.0/31')).eql?(1) || private_ips.rel(NetAddr::IPv4Net.parse('192.168.0.0/31')).eql?(0)
=> true
Note: rel method determines the relationship to another IPv4Net.
Returns:
1 if this IPv4Net is the supernet of other.
0 if the two are equal.
-1 if this IPv4Net is a subnet of other.
nil if the networks are unrelated.
Gem Documentation Link
I would like to convert a string containing a valid Erlang expression to its abstract syntax tree representation, without any success so far.
Below is an example of what I would like to do. After compiling, alling z:z(). generates module zed, which by calling zed:zed(). returns the result of applying lists:reverse on the given list.
-module(z).
-export([z/0]).
z() ->
ModuleAST = erl_syntax:attribute(erl_syntax:atom(module),
[erl_syntax:atom("zed")]),
ExportAST = erl_syntax:attribute(erl_syntax:atom(export),
[erl_syntax:list(
[erl_syntax:arity_qualifier(
erl_syntax:atom("zed"),
erl_syntax:integer(0))])]),
%ListAST = ?(String), % This is where I would put my AST
ListAST = erl_syntax:list([erl_syntax:integer(1), erl_syntax:integer(2)]),
FunctionAST = erl_syntax:function(erl_syntax:atom("zed"),
[erl_syntax:clause(
[], none,
[erl_syntax:application(
erl_syntax:atom(lists),
erl_syntax:atom(reverse),
[ListAST]
)])]),
Forms = [erl_syntax:revert(AST) || AST <- [ModuleAST, ExportAST, FunctionAST]],
case compile:forms(Forms) of
{ok,ModuleName,Binary} -> code:load_binary(ModuleName, "z", Binary);
{ok,ModuleName,Binary,_Warnings} -> code:load_binary(ModuleName, "z", Binary)
end.
String could be "[1,2,3].", or "begin A=4, B=2+3, [A,B] end.", or anything alike.
(Note that this is just an example of what I would like to do, so evaluating String is not an option for me.)
EDIT:
Specifying ListAST as below generates a huge dict-digraph-error-monster, and says "internal error in lint_module".
String = "[1,2,3].",
{ok, Ts, _} = erl_scan:string(String),
{ok, ListAST} = erl_parse:parse_exprs(Ts),
EDIT2:
This solution works for simple terms:
{ok, Ts, _} = erl_scan:string(String),
{ok, Term} = erl_parse:parse_term(Ts),
ListAST = erl_syntax:abstract(Term),
In your EDIT example:
String = "[1,2,3].",
{ok, Ts, _} = erl_scan:string(String),
{ok, ListAST} = erl_parse:parse_exprs(Ts),
the ListAST is actually a list of AST:s (because parse_exprs, as the name indicates, parses multiple expressions (each terminated by a period). Since your string contained a single expression, you got a list of one element. All you need to do is match that out:
{ok, [ListAST]} = erl_parse:parse_exprs(Ts),
so it has nothing to do with erl_syntax (which accepts all erl_parse trees); it's just that you had an extra list wrapper around the ListAST, which caused the compiler to puke.
Some comments of the top of my head.
I have not really used the erl_syntax libraries but I do think they make it difficult to read and "see" what you are trying to build. I would probably import the functions or define my own API to make it shorter and more legible. But then I generally tend to prefer shorter function and variable names.
The AST created by erl_syntax and the "standard" one created by erl_parse and used in the compiler are different and cannot be mixed. So you have to choose one of them and stick with it.
The example in your second EDIT will work for terms but not in the more general case:
{ok, Ts, _} = erl_scan:string(String),
{ok, Term} = erl_parse:parse_term(Ts),
ListAST = erl_syntax:abstract(Term),
This because erl_parse:parse_term/1 returns the actual term represented by the tokens while the other erl_parse functions parse_form and parse_exprs return the ASTs. Putting them into erl_syntax:abstract will do funny things.
Depending on what you are trying to do it might actually be easier to actually write out and erlang file and compile it rather than working directly with the abstract forms. This goes against my ingrained feelings but generating the erlang ASTs is not trivial. What type of code do you intend to produce?
<shameless_plug>
If you are not scared of lists you might try using LFE (lisp flavoured erlang) to generate code as with all lisps there is no special abstract form, it's all homoiconic and much easier to work with.
</shameless_plug>
Zoltan
This is how we get the AST:
11> String = "fun() -> io:format(\"blah~n\") end.".
"fun() -> io:format(\"blah~n\") end."
12> {ok, Tokens, _} = erl_scan:string(String).
{ok,[{'fun',1},
{'(',1},
{')',1},
{'->',1},
{atom,1,io},
{':',1},
{atom,1,format},
{'(',1},
{string,1,"blah~n"},
{')',1},
{'end',1},
{dot,1}],
1}
13> {ok, AbsForm} = erl_parse:parse_exprs(Tokens).
{ok,[{'fun',1,
{clauses,[{clause,1,[],[],
[{call,1,
{remote,1,{atom,1,io},{atom,1,format}},
[{string,1,"blah~n"}]}]}]}}]}
14>