Can we use Elixir plugins with EMQ X Broker? - erlang

EMQ X has the provision to build with Elixir based plugins as per https://github.com/emqx/emqx-rel#build-with-elixir-plugins. But, it's not compiling and throwing the following error
➜ make
ln -snf _build/emqx/lib ./_checkouts
EMQX_DESC="EMQ X Broker" /Users/XXXXX/Programs/emqx-rel/rebar3 as emqx release
===> Compiling rebar_mix
===> Compiling _build/default/plugins/rebar_mix/src/rebar_mix_builder.erl failed
_build/default/plugins/rebar_mix/src/rebar_mix_builder.erl:none: undefined parse transform 'mod_vsn'
===> Plugin rebar_mix not available. It will not be used.
===> Compiling rebar3_elixir_compile
===> Compiling _build/default/plugins/rebar_mix/src/rebar_mix_hook.erl failed
_build/default/plugins/rebar_mix/src/rebar_mix_hook.erl:none: undefined parse transform 'mod_vsn'
===> Plugin {rebar3_elixir_compile,
{git,
"https://github.com/barrel-db/rebar3_elixir_compile.git",
{branch,"master"}}} not available. It will not be used.
===> Compiling relup_helper
===> Compiling _build/default/plugins/rebar_mix/src/rebar_mix_elixir_finder_hook.erl failed
_build/default/plugins/rebar_mix/src/rebar_mix_elixir_finder_hook.erl:none: undefined parse transform 'mod_vsn'
===> Plugin {relup_helper,{git,"https://github.com/emqx/relup_helper",
{branch,"master"}}} not available. It will not be used.
===> Compiling pc
===> Compiling _build/default/plugins/rebar_mix/src/rebar_mix.erl failed
_build/default/plugins/rebar_mix/src/rebar_mix.erl:none: undefined parse transform 'mod_vsn'
===> Plugin {pc,{git,"https://github.com/emqx/port_compiler.git",
{tag,"v1.11.1"}}} not available. It will not be used.
===> Compiling rebar3_run
===> Plugin {rebar3_run,{git,"https://github.com/emqx/rebar3_run",
{tag,"0.2.3"}}} not available. It will not be used.
and rebar.config diff from the latest emqx-rel's master
diff --git a/rebar.config b/rebar.config
index 81fd886..571b07b 100644
--- a/rebar.config
+++ b/rebar.config
## -1,5 +1,7 ##
%% NOTE: Order of the deps matters!
-{elixir_deps, []}.
+{elixir_deps, [
+ {emq_elixir_plugin, {git, "git#github.com:emqx/emqx-elixir-plugin.git", {branch, "master"}}}
+]}.
{deps,
[emqx,
## -113,11 +115,14 ##
{template,"{{output_dir}}/../../lib/emqx/etc/{{vm_args_file}}","etc/vm.args"},
{copy, "{{output_dir}}/../../lib/emqx/etc/certs","etc/"},
{copy, "{{output_dir}}/../../lib/cuttlefish/cuttlefish","bin/"},
- {copy, "{{output_dir}}/../../lib/cuttlefish/cuttlefish","bin/cuttlefish-{{rel_vsn}}"}
+ {copy, "{{output_dir}}/../../lib/cuttlefish/cuttlefish","bin/cuttlefish-{{rel_vsn}}",
+ {copy, "{{base_dir}}/consolidated", "releases/{{rel_vsn}}/consolidated"}}
]}
]}.
-{elixir_relx_apps, []}.
+{elixir_relx_apps, [
+ {emq_elixir_plugin, load}
+]}.
{edge_relx_apps, []}.
and rebar.config diff from the latest emqx-rel's master
diff --git a/rebar.config b/rebar.config
index 81fd886..571b07b 100644
--- a/rebar.config
+++ b/rebar.config
## -1,5 +1,7 ##
%% NOTE: Order of the deps matters!
-{elixir_deps, []}.
+{elixir_deps, [
+ {emq_elixir_plugin, {git, "git#github.com:emqx/emqx-elixir-plugin.git", {branch, "master"}}}
+]}.
{deps,
[emqx,
## -113,11 +115,14 ##
{template,"{{output_dir}}/../../lib/emqx/etc/{{vm_args_file}}","etc/vm.args"},
{copy, "{{output_dir}}/../../lib/emqx/etc/certs","etc/"},
{copy, "{{output_dir}}/../../lib/cuttlefish/cuttlefish","bin/"},
- {copy, "{{output_dir}}/../../lib/cuttlefish/cuttlefish","bin/cuttlefish-{{rel_vsn}}"}
+ {copy, "{{output_dir}}/../../lib/cuttlefish/cuttlefish","bin/cuttlefish-{{rel_vsn}}",
+ {copy, "{{base_dir}}/consolidated", "releases/{{rel_vsn}}/consolidated"}}
]}
]}.
-{elixir_relx_apps, []}.
+{elixir_relx_apps, [
+ {emq_elixir_plugin, load}
+]}.
{edge_relx_apps, []}.
➜ erl
Erlang/OTP 23 [erts-11.1.7] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe] [dtrace]
Eshell V11.1.7 (abort with ^G)
1>
➜ rebar3 -v
rebar 3.14.3 on Erlang/OTP 23 Erts 11.1.7
➜ iex -v
Erlang/OTP 23 [erts-11.1.7] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe] [dtrace]
IEx 1.11.3 (compiled with Erlang/OTP 23)

Related

Erlang simple release with cowboy 2.0.0 do not work

I'm using Erlang/OTP 20.0 and rebar3. When I start a new release that use cowboy 2.0.0, the release fail to start.
Here are the steps I do to build the project. What's wrong?
create the release project
$ rebar3 new release cowboy2
add cowboy package
{deps, [{cowboy, {git, "https://github.com/ninenines/cowboy.git", {tag, "2.0.0"}}}]}.
add a basic dispatcher
start(_StartType, _StartArgs) ->
Dispatch = cowboy_router:compile([
{'_', [
{"/", toppage_handler, []}
]}
]),
{ok, _} = cowboy:start_clear(http, [{port, 8080}], #{
env => #{dispatch => Dispatch}
}),
cowboy2_sup:start_link().
add handler. for the purpose of this example I use (https://raw.githubusercontent.com/ninenines/cowboy/master/examples/hello_world/src/toppage_handler.erl)
compile and release
$ rebar3 compile && rebar3 release
run the application
$ ./_build/default/rel/cowboy2/bin/cowboy2-0.1.0 console
The output
$ ./_build/default/rel/cowboy2/bin/cowboy2-0.1.0 console
Exec: /home/deimos/.asdf/installs/erlang/20.0/lib/erlang/erts-9.0/bin/erlexec
-boot /home/deimos/Dev/personal/cowboy2/_build/default/rel/cowboy2/releases/0.1.0/cowboy2
-mode embedded -boot_var ERTS_LIB_DIR /home/deimos/.asdf/installs/erlang/20.0/lib/erlang/lib
-config /home/deimos/Dev/personal/cowboy2/_build/default/rel/cowboy2/releases/0.1.0/sys.config
-args_file /home/deimos/Dev/personal/cowboy2/_build/default/rel/cowboy2/releases/0.1.0/vm.args
-pa -- console
Root: /home/deimos/Dev/personal/cowboy2/_build/default/rel/cowboy2
/home/deimos/Dev/personal/cowboy2/_build/default/rel/cowboy2
Erlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:30] [hipe] [kernel-poll:true]
=INFO REPORT==== 11-Oct-2017::21:00:52 ===
application: cowboy2
exited: {bad_return,
{{cowboy2_app,start,[normal,[]]},
{'EXIT',
{undef,
[{cowboy_router,compile,
[[{'_',[{"/",toppage_handler,[]}]}]],
[]},
{cowboy2_app,start,2,
[{file,
"/home/deimos/Dev/personal/cowboy2/_build/default/lib/cowboy2/src/cowboy2_app.erl"},
{line,18}]},
{application_master,start_it_old,4,
[{file,"application_master.erl"},
{line,273}]}]}}}}
type: permanent
(...)
Kernel pid terminated (application_controller)
({application_start_failure,cowboy2,{bad_return,{{cowboy2_app,start,[normal,[]]},{'EXIT',{undef,[{cowboy_router,compile,[[{'_',[{"/",toppage_handler,[]}]
Crash dump is being written to: erl_crash.dump...done
As #maze-le points out, the problem was that the cowboy application was not started. The solution is to add cowboy to the file apps/cowboy2/src/cowboy2.app.src also generated by the command $rebar3 new release cowboy2. I add the file for completeness.
apps/cowboy2/src/cowboy2.app.src
{application, cowboy2,
[{description, "An OTP application"},
{vsn, "0.1.0"},
{registered, []},
{mod, { cowboy2_app, []}},
{applications,
[kernel,
stdlib,
cowboy
]},
{env,[]},
{modules, []},
{maintainers, []},
{licenses, ["Apache 2.0"]},
{links, []}
]}.

My Erlang application don't see ranch module dependency

(Newbie here) I am having an error when trying to run ranch example.
Via rebar I created application and node (please see on github).
But when I am trying to build and run it I am getting undef ranch,start_listener.
Please see full console output:
$ rebar get-deps compile generate && sh rel/reverse/bin/reverse console
WARN: Expected reverse/deps/ranch to be an app dir (containing ebin/*.app), but no .app found.
==> rel (get-deps)
==> reverse (get-deps)
WARN: Expected reverse/deps/ranch to be an app dir (containing ebin/*.app), but no .app found.
Pulling ranch from {git,"git#github.com:ninenines/ranch.git",{tag,"1.1.0"}}
Cloning into 'ranch'...
==> ranch (get-deps)
==> ranch (compile)
Compiled src/ranch_transport.erl
Compiled src/ranch_sup.erl
Compiled src/ranch_ssl.erl
Compiled src/ranch_tcp.erl
Compiled src/ranch_protocol.erl
Compiled src/ranch_listener_sup.erl
Compiled src/ranch_app.erl
Compiled src/ranch_acceptors_sup.erl
Compiled src/ranch_acceptor.erl
Compiled src/ranch_server.erl
Compiled src/ranch.erl
Compiled src/ranch_conns_sup.erl
==> rel (compile)
==> reverse (compile)
Compiled src/reverse_sup.erl
Compiled src/reverse_app.erl
Compiled src/reverse_protocol.erl
==> rel (generate)
WARN: 'generate' command does not apply to directory reverse
Exec: reverse/rel/reverse/erts-6.3/bin/erlexec -boot reverse/rel/reverse/releases/1/reverse -mode embedded -config reverse/rel/reverse/releases/1/sys.config -args_file reverse/rel/reverse/releases/1/vm.args -- console
Root: reverse/rel/reverse
Erlang/OTP 17 [erts-6.3] [source-f9282c6] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]
Eshell V6.3 (abort with ^G)
(reverse#127.0.0.1)1>
=INFO REPORT==== 30-Dec-2014::22:47:08 ===
application: reverse
exited: {bad_return,
{{reverse_app,start,[normal,[]]},
{'EXIT',
{undef,
[{ranch,start_listener,
[reverse,10,ranch_tcp,
[{port,5555}],
reverse_protocol,[]],
[]},
{reverse_app,start,2,
[{file,"src/reverse_app.erl"},{line,13}]},
{application_master,start_it_old,4,
[{file,"application_master.erl"},
{line,272}]}]}}}}
type: permanent
{"Kernel pid terminated",application_controller,"{application_start_failure,reverse,{bad_return,{{reverse_app,start,[normal,[]]},{'EXIT',{undef,[{ranch,start_listener,[reverse,10,ranch_tcp,[{port,5555}],reverse_protocol,[]],[]},{reverse_app,start,2,[{file,\"src/reverse_app.erl\"},{line,13}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,272}]}]}}}}}"}
Crash dump was written to: erl_crash.dump
I am not sure I correctly added ranch to reltool.config (please see on github). But If I remove deps from libs_dir path I will get rebar generate error Application version clash. Multiple directories contain version ....
UPDATE if I remove failing call and run it, application:which_applications(). gives me {ranch,[],[]} as one of the running ones.
UPDATE versions
$ erl
Erlang/OTP 17 [erts-6.3] [source-f9282c6] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]
$ rebar --version
rebar 2.5.1 17 20141223_141030 git 2.5.1-84-gdd9125e
What I am doing wrong?
Thanks in advance!
In the reltool.config, remove the ebin directory from the path to Ranch:
{app, ranch, [{mod_cond, app}, {incl_cond, include}, {lib_dir, "../deps/ranch/"}]}
If you have lots of dependencies it's more convenient to do:
{lib_dirs, ["../deps"]}
instead of having a separate {app, <dep_name>, [...]} for each dependency.
The Application version clash. Multiple directories contain version XXX error indicates that the Ranch application is already installed, probably in the Erlang lib directory, creating conflicts with the deps version downloaded by Rebar.

Why can't I run this Erlang demo application?

I am having a similar problem as here for running an rebar app
Basically, I want to run the demo code from: https://github.com/hukl/fancyapi
When I am in the root directory, and I do:
→ erl [...] -pa ebin -pa deps/*/ebin
Erlang R15B02 (erts-5.9.2) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] [dtrace]
Eshell V5.9.2 (abort with ^G)
1> application
1> .
application
2> application:start(fancyapi).
{error,{"no such file or directory","fancyapi.app"}}
3> application:start("fancyapi").
{error,{bad_application,"fancyapi"}}
4>
How would ERL find the fancyapi code?
[g#dev1 ~/work]$ git clone git://github.com/hukl/fancyapi.git
Cloning into 'fancyapi'...
remote: Counting objects: 12, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 12 (delta 0), reused 12 (delta 0)
Receiving objects: 100% (12/12), done.
[g#dev1 ~/work]$ cd fancyapi/
[g#dev1 ~/work/fancyapi]$ rebar get-deps
==> fancyapi (get-deps)
Pulling etest from {git,"git://github.com/wooga/etest.git"}
Cloning into 'etest'...
Pulling etest_http from {git,"git://github.com/wooga/etest_http.git"}
Cloning into 'etest_http'...
Pulling elli from {git,"git://github.com/knutin/elli.git"}
Cloning into 'elli'...
==> etest (get-deps)
==> etest_http (get-deps)
Pulling jiffy from {git,"git://github.com/davisp/jiffy.git","HEAD"}
Cloning into 'jiffy'...
==> jiffy (get-deps)
==> elli (get-deps)
[g#dev1 ~/work/fancyapi]$ rebar compile
==> etest (compile)
Compiled src/etest_runner.erl
==> jiffy (compile)
Compiled test/jiffy_tests.erl
Compiled test/util.erl
Compiled src/jiffy_utf8.erl
Compiled src/jiffy.erl
Compiled test/etap.erl
Compiling c_src/decoder.c
Compiling c_src/encoder.c
Compiling c_src/jiffy.c
Compiling c_src/utf8.c
Compiling c_src/util.c
==> etest_http (compile)
Compiled src/etest_http.erl
Compiled src/etest_http_json.erl
Compiled test/etest_http_test.erl
==> elli (compile)
Compiled src/elli_handler.erl
Compiled src/elli_middleware_compress.erl
Compiled src/elli_example_middleware.erl
Compiled src/elli_middleware.erl
Compiled src/elli_util.erl
Compiled src/elli_request.erl
Compiled src/elli.erl
Compiled src/elli_test.erl
Compiled src/elli_example_callback.erl
Compiled src/elli_http.erl
==> fancyapi (compile)
Compiled src/fancyapi_app.erl
Compiled src/fancyapi_callback.erl
Compiled src/fancyapi_sup.erl
Compiled test/my_fancy_test.erl
[g#dev1 ~/work/fancyapi]$ erl -pa ebin -pa ebin deps/*/ebin
Erlang R15B03 (erts-5.9.3.1) [source] [64-bit] [smp:2:2] [async-threads:0] [hipe] [kernel-poll:false]
Eshell V5.9.3.1 (abort with ^G)
1> application:start(fancyapi).
ok

rebar: Missing application directory

I'm testing rebar on
Windows 8 64Bis
Erlang 64bits R15B02
I've compiled rebar from github code and created a basic app
$ mkdir testapp; cd testapp
$ mkdir rel
$ rebar create-app appid=testapp
$ echo "{sub_dirs, ["rel"]}." > rebar.config
$ cd rel
$ rebar create-node nodeid=testnode
$ cd -
$ rebar compile
$ rebar generate
ERROR: generate failed while processing testapp/rel: {'EXIT',{{badmatch,{error,"testapp: : Missing application directory."}
...
I'm reading the reltool documentation but i cant found anything about application dir the only relevant option is incl_cond but is defined by default by rebar command
src/testapp.app.src
{application, testapp,
[
{description, ""},
{vsn, "1"},
{registered, []},
{applications, [
kernel,
stdlib
]},
{mod, { testapp_app, []}},
{env, []}
]}.
rel/reltool.config
{sys, [
{lib_dirs, []},
{erts, [{mod_cond, derived}, {app_file, strip}]},
{app_file, strip},
{rel, "testapp", "1",
[
kernel,
stdlib,
sasl,
testapp
]},
{rel, "start_clean", "",
[
kernel,
stdlib
]},
{boot_rel, "testapp"},
{profile, embedded},
{incl_cond, derived},
{mod_cond, derived},
{excl_archive_filters, [".*"]}, %% Do not archive built libs
{excl_sys_filters, ["^bin/.*", "^erts.*/bin/(dialyzer|typer)",
"^erts.*/(doc|info|include|lib|man|src)"]},
{excl_app_filters, ["\.gitignore"]},
{app, testapp, [{mod_cond, app}, {incl_cond, include}]}
]}.
{target_dir, "testapp"}.
{overlay, [
{mkdir, "log/sasl"},
{copy, "files/erl", "\{\{erts_vsn\}\}/bin/erl"},
{copy, "files/nodetool", "\{\{erts_vsn\}\}/bin/nodetool"},
{copy, "files/testapp", "bin/testapp"},
{copy, "files/testapp.cmd", "bin/testapp.cmd"},
{copy, "files/start_erl.cmd", "bin/start_erl.cmd"},
{copy, "files/install_upgrade.escript", "bin/install_upgrade.escript"},
{copy, "files/sys.config", "releases/\{\{rel_vsn\}\}/sys.config"},
{copy, "files/vm.args", "releases/\{\{rel_vsn\}\}/vm.args"}
]}.
In my case I had to modify the default rel/reltool.config line
{lib_dirs, []},
to
{lib_dirs, ["../../"]},
Where the layout of my project is:
Makefile
ebin/
rebar.config
rel/
src/
This might help you. (quoted below in case the original link disappears)
Not exactly sure what's the reason for that change for versions following R15B01 though.
Thanks to Siri, reltool in R15B01 includes a really useful feature.
Many Erlang projects are using the following structure:
./src/foo.app.src
./src/foo.erl
./rel/reltool.config
When you wanted to reference the application foo in
rel/reltool.config, you previously had three options:
* project-local libs/ or apps/ directory
* unsafe {lib_dirs, "../.."} reltool.config setting
* rebar_reltool_link fake_lib_dir plugin
Starting with R15B01 all you have to do now is specify
{app, foo, [{incl_cond, include}, {lib_dir, ".."}]}
in reltool.config for applications not in lib_dirs.
Although R15B01 is still in development, this feature is complete and
can be used in the otp.git maint branch.
Refer this guideline,
https://github.com/rebar/rebar/wiki/Release-handling
If you're using R15B01 or newer change,
{app, exemplar, [{mod_cond, app}, {incl_cond, include}, {lib_dir, ".."}]}
above way of mentioning path solved the application directory not found issue for me.

Erlang releases with Rebar: What am I missing?

Thanks to much help here, I'm well on my way toward building my first Erlang release. No real code yet, but I want to understand how it's done. I've consulted and followed several web tutorials as well Martin et. al., but still seem to be missing something.
When I try to start my release I get:
lloyd#Reliance:~/Programming/Erlang/learn$ sh rel/learn/bin/learn start
[: 129: Node 'learn#127.0.0.1' not responding to pings.: unexpected operator
Under the project directory "learn" I have:
apps rebar rebar.config rel
In rebar.config, I have:
{cover_enabled, true}.
{sub_dirs, ["rel","apps/zzz", "apps/zzz_lib"]}.
In ...learn/apps, I have:
zzz zzz_lib
zzz and zzz_lib have all the right stuff in them so far as I can tell. From lean, I can clean, compile, and create docs.
In .../rel,I have:
files learn reltool.config
See reltool.config below.
I'm missing magic sauce, but what?
Many thanks,
LRP
{sys, [
{lib_dirs, []},
{rel, "learn", "1",
[
kernel,
stdlib,
sasl
]},
{rel, "start_clean", "",
[
kernel,
stdlib
]},
{boot_rel, "learn"},
{profile, embedded},
{excl_sys_filters, ["^bin/.*",
"^erts.*/bin/(dialyzer|typer)"]},
{app, sasl, [{incl_cond, include}]}
]}.
{target_dir, "learn"}.
{overlay, [
{mkdir, "log/sasl"},
{copy, "files/erl", "{{erts_vsn}}/bin/erl"},
{copy, "files/nodetool", "{{erts_vsn}}/bin/nodetool"},
{copy, "files/learn", "bin/learn"},
{copy, "files/app.config", "etc/app.config"},
{copy, "files/vm.args", "etc/vm.args"}
]}.
Looks like your retool.config file is missing some entries for the application you have written.
The first part should look something like this.
{sys, [
{lib_dirs, ["../apps"]}, <--- point to where your applications are
{rel, "learn", "1",
[
<your application here> <---- add your application(s) here
kernel,
stdlib,
sasl
]},
{rel, "start_clean", "",
[
kernel,
stdlib
]},
{boot_rel, "learn"},
{profile, embedded},
{excl_sys_filters, ["^bin/.*",
"^erts.*/bin/(dialyzer|typer)"]},
{app, <your application here>, [{incl_cond, include}]}, <-- and here
{app, sasl, [{incl_cond, include}]}
]}.
Here is a sample application from Erlang and OTP in Action that I packaged up using rebar.
https://github.com/tmcgilchrist/simple_cache
The general layout I follow is
simple_cache
|-> apps
| \-> simple_cache
| |-> src
| \-> ebin
|
|-> rebar.config
|-> rel
|-> files
|-> reltool.config
\-> simple_cache
Also rather that doing
sh rel/learn/bin/learn start
use
sh rel/learn/bin/learn console
and enter
application:which_applications().
Which should list a bunch of things plus your application.
eg
[{mysample_app,[],[]},
{sasl,"SASL CXC 138 11","2.1.10"},
{stdlib,"ERTS CXC 138 10","1.17.5"},
{kernel,"ERTS CXC 138 10","2.14.5"}]

Resources