I am in the process or learning erlang OTP and rebar and I have put together a small example app using a couple of dependencies: cowboy and lager.
I have issued
rebar get-deps
rebar compile
And things went smoothly. Now I want to fire up my console to test things around but it is not obvious to me how to start the dependency applications.
I tried issuing a
rebar generate
In order to get all the orchestration of firing up the apps, even though it's overkill for just development tests, but I miserably failed getting the following dump
Crash dump was written to: erl_crash.dump
eheap_alloc: Cannot allocate 1459620480 bytes of memory (of type "old_heap").
Aborted
The ebin dir only has beam files for the app I wrote but not the dependencies, I see the dependencies have their own ebin directory inside the main app deps directory, how would I go about having them available in a console to start them up?
I would appreciate if someone can shed some light as to what the common practice is for the dev env with multiple OTP apps.
I have read a couple of tutorials but they are mostly targeted at the rebar release cycle and not the development process.
TIA
In your case, the modules you pull into the deps directory should typically be called from within your application code - and your application can be typically invoked from the Erlang shell using the application:start/1 function. If you haven't yet, I strongly suggest that you read Chapter 12, "OTP Behaviors", of Francesco Cesarini's excellent book Erlang Programming - it's a great practical introduction for what you're attempting.
Related
I'm trying to figure out how to properly setup an ejabberd project that allows for easy compilation of custom beam files- so far, we've been using an existing project that is cumbersome to manage, and uses erlide as the IDE.
I would like to set up the project in a way that I can use a more helpful IDE like vscode, and somehow streamline the compiling and copying of the beam files and updating the module on the server.
Writing code in Elixir is fine as well- I just want the project to be set up in a way that is dev friendly.
Apologies if the question is too broad, but I'm not exactly sure how else to best phrase it. If you feel like I'm missing something in my current flow, please let me know, as I've basically inherited this project. If there are any clarifications required, let me know as well.
Thanks.
easy compilation of custom beam files
somehow streamline the compiling and copying of the beam files and updating the module on the server.
If the task is about compiling and loading additional modules, a running ejabberd node can compile, load and start additional modules in runtime, see
https://docs.ejabberd.im/developer/extending-ejabberd/modules/#ejabberd-contrib
Usually the modules come from
https://github.com/processone/ejabberd-contrib
but you can tell ejabberd to download other modules from other git repositories, or you can copy modules source code and tell ejabberd to install them. And those modules can be written in Erlang or Elixir. Full example: https://docs.ejabberd.im/developer/extending-ejabberd/elixir/#elixir-module-in-ejabberd-contrib
Basically:
you write the module in your development machine, test it...
when happy with the source code, copy mod_whatever.erl to the production machine, $HOME/.ejabberd-modules/sources/mod_whatever as explained in the example mentioned earlier
run ejabberdctl module_install mod_whatever
In step 2, instead of copying the source code yourself, you can have a git repository just for your module, tell ejabberd the module's git URL, similarly to https://github.com/processone/ejabberd-contrib/tree/master/extra
BTW, for step 3, starting in ejabberd 22.10, there's a page in ejabberd webadmin to install and uninstall those modules (copying the files requires manual administration of course).
I would like to set up the project in a way that I can use a more helpful IDE like vscode
What a coincidence, these days I'm playing with VSCode variants (VSCode, VSCodium, Coder's code-server and Github Codespaces) and how to develop ejabberd using them. This is useful for step 1 that I mentioned earlier (write module and test it). If you are interested in ejabberd + VSCode, tell me.
The problem
Most of the articles and books about Erlang I could find focus on creating long running server-like applications, leaving the process of command line tools creation not covered.
I have a multi-app rebar3 project consisting of 3 applications:
myweb - a Cowboy-based web service;
mycli - a command line tool to prepare assets for myweb;
mylib - a library used by both myweb and mycli, depends on a NIF.
As a result of the build I want to get such artifacts:
an executable for the web part that is going to serve HTTP requests;
an executable command line tool for the assets preparation;
a set of libraries used by the above.
Requirements
CLI should behave like a sane non-interactive command line tool: handle arguments, deal with stdin/stdout, return non-zero exit code on error, etc;
both server and CLI should be able to use NIFs;
it should be easy to package the artifacts as a set of deb/rpm packages, so both server and CLI should reuse common dependencies.
Things tried so far
Building an escript
One of the ways I've seen in the wild is to create a self-contained escript file. At least rebar3 and relx do so. So I gave it a try.
Pros:
has support for command line arguments;
in case of errors, it returns non-zero exit code.
Cons:
embeds all the dependencies in a single file making it impossible to reuse mylib;
since *.so files get embedded into the resulting escript file, they cannot be loaded at runtime, thus NIFs don't work (see erlang rebar escriptize & nifs);
rebar3 escriptize doesn't handle dependencies well (see bug 1139).
Unknowns:
Should the CLI app become a proper OTP application?
Should it have a supervision tree?
Should it be started at all?
If so, how do I stop it when the assets have been processed?
Building a release
Another way to build a command line tool was described in a How I start: Erlang article by Fred Hebert.
Pros:
Each of the dependency applications get into their own directory, making it easy to share and package them.
Cons:
there's no defined entry point like escript's main/1;
as a consequence both command line arguments and exit code must be handled manually.
Unknowns:
How to model the CLI OTP app in a non-interactive way?
How to stop the app when the assets have been processed?
Neither of the approaches above seem to work for me.
It would be nice to get the best of both worlds: get the infrastructure that is provided by escript such as main/1 entry point, command line parameters, and exit code handling while still having a nice directory structure that is easy to package and which doesn't hinder the use of NIFs.
Regardless if you are starting a long-running daemon-like application in Erlang, or a CLI command, you always need the following:
erts application - the VM and kernel in a particular version
Erlang OTP applications
Your applications' dependencies
CLI entry point
Then in either case the CLI entry point has to start the Erlang VM and execute the code that it supposed to execute in a given situation. Then it will either exit or continue running - the later for a long-running application.
The CLI entry point can be anything that starts an Erlang VM, e.g. an escript script, sh, bash, etc. The obvious advantage of escript over generic shell is that escript is already being executed in the context of an Erlang VM, so no need to handle starting/stopping the VM.
You can start Erlang VM in two ways:
Use system-wide Erlang VM
Use an embedded Erlang release
In the first case you don't supply erts nor any OTP application with your package, you only make a particular Erlang version a dependency for your application. In the second case you supply erts and all required OTP applications along with your application's dependencies in your package.
In the second case you also need to handle setting the code root properly when starting the VM. But this is quite easy, see the erl script that Erlang uses to start the system-wide VM:
# location: /usr/local/lib/erlang/bin/erl
ROOTDIR="/usr/local/lib/erlang"
BINDIR=$ROOTDIR/erts-7.2.1/bin
EMU=beam
PROGNAME=`echo $0 | sed 's/.*\///'`
export EMU
export ROOTDIR
export BINDIR
export PROGNAME
exec "$BINDIR/erlexec" ${1+"$#"}
This can be handled by scripts, for example the node_package tool that Basho uses to package their Riak database for all major operating systems. I am maintaining my own fork of it which I am using with my own build tool called builderl. I just say that so you know that if I managed to customize it you will well be able to do that as well :)
Once the Erlang VM is started, your application should be able to load and start any application, either supplied with Erlang or with your application (and that includes the mylib library that you mentioned). Here are some examples how this could be achieved:
escript example
See this builderl.esh example how I handle loading other Erlang applications from builderl. That escript script assumes that the Erlang installation is relative to the folder from which it's executed. When it's a part of another application, like for example humbundee, the load_builderl.hrl include file compiles and loads bld_load, which in turn loads all remaining modules with bld_load:boot/3. Notice how I can use standard OTP applications without specifying where they are - builderl is being executed by escript and so all the applications are loaded from where they were installed (/usr/local/lib/erlang/lib/ on my system). If libraries used by your application, e.g. mylib, are installed somewhere else, all you need to do is add that location to the Erlang path, e.g. with code:add_path. Erlang will automatically load modules used in the code from folders added to the code path list.
embedded Erlang
However, the same would hold if the application was a proper OTP release installed independently from the system-wide Erlang installation. That's because in that case the script is executed by escript belonging to that embedded Erlang release rather than the system-wide version (even if it's installed). So it knows the location of all applications belonging to that release (including your applications). For example riak does exactly that - in their package they supply an embedded Erlang release that contains its own erts and all dependent Erlang applications. That way riak can be started without Erlang being even installed on the host operating system. This is an excerpt from a riak package on FreeBSD:
% tar -tf riak2-2.1.1_1.txz
/usr/local/sbin/riak
/usr/local/lib/riak/releases/start_erl.data
/usr/local/lib/riak/releases/2.1.0/riak.rel
/usr/local/lib/riak/releases/RELEASES
/usr/local/lib/riak/erts-5.10.3/bin/erl
/usr/local/lib/riak/erts-5.10.3/bin/beam
/usr/local/lib/riak/erts-5.10.3/bin/erlc
/usr/local/lib/riak/lib/stdlib-1.19.3/ebin/re.beam
/usr/local/lib/riak/lib/ssl-5.3.1/ebin/tls_v1.beam
/usr/local/lib/riak/lib/crypto-3.1/ebin/crypto.beam
/usr/local/lib/riak/lib/inets-5.9.6/ebin/inets.beam
/usr/local/lib/riak/lib/bitcask-1.7.0/ebin/bitcask.app
/usr/local/lib/riak/lib/bitcask-1.7.0/ebin/bitcask.beam
(...)
sh/bash
This doesn't differ much in principle from the above apart from having to explicitly call the function that you want to execute when starting the Erlang VM (the entry point or the main function as you called it).
Consider this script that builderl generates to start an Erlang application just to execute a specified task (generate the RELEASES file), after which the node shuts down:
#!/bin/sh
START_ERL=`cat releases/start_erl.data`
APP_VSN=${START_ERL#* }
run_erl -daemon ../hbd/shell/ ../hbd/log "exec erl ../hbd releases releases/start_erl.data -config releases/$APP_VSN/hbd.config -args_file ../hbd/etc/vm.args -boot releases/$APP_VSN/humbundee -noshell -noinput -eval \"{ok, Cwd} = file:get_cwd(), release_handler:create_RELEASES(Cwd, \\\"releases\\\", \\\"releases/$APP_VSN/humbundee.rel\\\", []), init:stop()\""
This is a similar script but doesn't start any specific code or application. Instead, it starts a proper OTP release, so which applications are started and in what order depends on the release (specified by the -boot option).
#!/bin/sh
START_ERL=`cat releases/start_erl.data`
APP_VSN=${START_ERL#* }
run_erl -daemon ../hbd/shell/ ../hbd/log "exec erl ../hbd releases releases/start_erl.data -config releases/$APP_VSN/hbd.config -args_file ../hbd/etc/vm.args -boot releases/$APP_VSN/humbundee"
In the vm.args file you can provide additional paths to your applications if required, e.g.:
-pa lib/humbundee/ebin lib/yolf/ebin deps/goldrush/ebin deps/lager/ebin deps/yajler/ebin
In this example these are relative, but could be absolute if your application is installed into a standard well-known location. Also, this would be only required if you are using the system-wide Erlang installation and need to add the additional paths to locate your Erlang applications, or if your Erlang applications are located in non-standard location (e.g. not in lib folder, as Erlang OTP requires). In a proper embedded Erlang release, where the applications are located in the code root/lib folder, Erlang is able to load those applications without specifying any additional paths.
Summing up and other considerations
The deployment of Erlang applications doesn't differ much from other projects written in scripting languages, e.g. ruby or python projects. All those projects have to deal with similar issues and I believe each operating system's package management deals with them one way or another:
Get to know how your operating system deals with packaging projects that have run-time dependencies.
See how other Erlang applications are packaged for your operating system, there are plenty of them that are usually distributed by all major systems: RabbitMQ, Ejabberd, Riak among others. Just download the package and unpack it to a folder, then you will see where all the files are placed.
EDIT - reference the requirements
Coming back to your requirements, you have the following choices:
Install Erlang as an OTP release system-wide, as an embedded Erlang, or as a bag with applications in some random folders (sorry Rebar)
You can have multiple entry points in the form of sh or escript scripts executing a selection of applications from the installed release. Both will work as long as you configured the code root and paths to those applications correctly (as outlined above).
Then each of your applications: myweb and mycli, would need to be executed in its own new context, e.g. start a new VM instance and execute the required application (from the same Erlang release). In case of myweb the entry point can be a sh scripts that starts a new node according to the release (similar to Riak). In case of mycli the entry point can be an escript, which finishes executing once the task is completed.
But it's entirely possible to create a short-running task that exits the VM even if it's started from sh - see the example above. In that case mycli would require separate release files - the script and boot to boot the VM. And of course it's also possible to start a long-running Erlang VM from escript.
I provided an example project that uses all these methods at once, humbundee. Once it's compiled it provides three access points:
The cmd release.
The humbundee release.
The builder.esh escript.
The first one is used to start the node for installation and then shut it down. The second is used to start a long-running Erlang application. The third is a build tool to install/configure the node. This is how the project looks like once the release has been created:
$:~/work/humbundee/tmp/rel % ls | tr " " "\n"
bin
erts-7.3
etc
lib
releases
$:~/work/humbundee/tmp/rel % ls bin | tr " " "\n"
builderl.esh
cmd.boot
humbundee.boot
epmd
erl
escript
run_erl
to_erl
(...)
$:~/work/humbundee/tmp/rel % ls lib | tr " " "\n"
builderl-0.2.7
compiler-6.0.3
deploy-0.0.1
goldrush-0.1.7
humbundee-0.0.1
kernel-4.2
lager-3.0.1
mnesia-4.13.3
sasl-2.7
stdlib-2.8
syntax_tools-1.7
yajler-0.0.1
yolf-0.1.1
$:~/work/humbundee/tmp/rel % ls releases/hbd-0.0.1 | tr " " "\n"
builderl.config
cmd.boot
cmd.rel
cmd.script
humbundee.boot
humbundee.rel
humbundee.script
sys.config.src
The cmd entry point will use application deploy-0.0.1 and builderl-0.2.7 as well as release files cmd.boot, cmd.script, and some OTP applications. The standard humbundee entry point will use all applications apart from builderl and deploy. Then the builderl.esh escript will use application deploy-0.0.1 and builderl-0.2.7. All from the same embedded Erlang OTP installation.
A small escript that then goes into code from 'conventional' modules could be a solution.
As an example, Concuerror is expected to be used as a command line tool and uses an escript as its entry point. It handles command-line arguments via getopt. All the main code is in regular Erlang modules, which are included in the path with simple arguments to the escript.
As far as I understand, NIFs can then be loaded with regular -onload attributes (Concuerror does not use NIFs).
My project is using a websocket server. Only for testing purposes, I also have an erlang websocket client implementation which resides in the test/ folder along with the tests.
Now, when I run the tests via rebar with {cover_enabled, true} in my rebar.config, I also get coverage reported for the modules of the websocket client. I don't want this in my reports.
Cover documentation says I should create a cover specification file containing {excl_mods, [websocket_client]}.
But how do I convince rebar to use this file?
Cover documentation:
http://www.erlang.org/doc/apps/common_test/cover_chapter.html
http://www.erlang.org/doc/man/cover.html
rebar:
https://github.com/basho/rebar
Ok, I asked rebar author #dizzyd.
It turns out that this cover option is not supported by eunit, which I use to execute my tests.
I would like to build and deploy an application which has Django as frontend, YAWS (appmods) or Mochiweb/Webmachine as a backend and CouchDB as a datastore. Furthermore, I plan to extensively use CouchDB's ability to replicate in order to provide high fault tolerance for the whole application.
I tend to think that in order to achieve this, I must create a single OTP release which has YAWS and CouchDB as Erlang/OTP applications.
Does this approach seem to be correct? How can I organize YAWS and CouchDB in terms of OTP applications in order to create a solid production setup? Are there any best practices for doing that?
Erlang releases are a good way to package software, because they include all of the dependent libraries. This way, different erlang applications can run different versions of their required libraries without conflicting with each other.
A popular way of dealing with the complicated release process Erlang has is to let Rebar do it for you. They have a quick start guide to get you on the right path. I don't know if you're currently using Rebar to manage your project, but it makes things a lot easier. It is definitely worth looking into if you aren't using it yet.
However, rebar won't include your dependencies right out of the box. In order to do that, you should modify the reltool.config file and all your required applications.
First step is to add the directory where all your dependencies are located, e.g.:
{sys, [
...
{lib_dirs, ["../deps"]},
...
}.
Then, add your dependencies as applications to include in the release:
{sys, [
...
{app, jiffy, [{incl_cond, include}]},
{app, cowboy, [{incl_cond, include}]},
...
}.
Now, when you run the rebar generate command, your applications should be in the target directory under lib:
find brawl_server -type d -maxdepth 2
brawl_server
brawl_server/bin
brawl_server/erts-5.9.1
brawl_server/erts-5.9.1/bin
brawl_server/lib
brawl_server/lib/brawl_server-1.1
brawl_server/lib/cowboy-0.6.0
brawl_server/lib/jiffy-0.6.1
brawl_server/lib/kernel-2.15.1
brawl_server/lib/sasl-2.2.1
brawl_server/lib/stdlib-1.18.1
brawl_server/log
brawl_server/log/sasl
brawl_server/releases
brawl_server/releases/1
You also need to make sure your own OTP application knows it needs to start the applications it depends on. If you have a generated Rebar project, modify your <appname>.app.src file to include them:
{application, app, [
...
{applications, [
jiffy,
cowboy,
kernel,
stdlib
]},
...
}.
You should be able to compile and generate the release and run it with the included scripts, as laid out in the Rebar release handling article.
At this point, though, there's an added wrinkle, because apparently vanilla CouchDB isn't OTP compliant, and you can't include it this way. There is another distribution you may be able to use, instead: rcouch. I haven't tried it myself, hopefully it will work for you.
Further reading:
Learn you some erlang for great good: Releases
I used my own project, brawl-online, as an example
I would recommend to create DEB/RPM/you-name-it package or packages. CheckInstall is the simplest solution for this.
I'm attempting to understand how to build a multiple-application Erlang OTP release using Rebar.
My applications include Webmachine, Riak, and a few applications of my own devising. So far, I have been running Rebar successfully in the application root directory. But the Rebar bootstrap program suggests that there may be a better way. E.g.:
"You now have a self-contained script called "rebar" in
your current working directory. Place this script anywhere in your path
and you can use rebar to build OTP-compliant apps."
I can think of several ways to do this. But I'm wondering:
Is there is a preferred convention: e.g., placing Rebar in an .erlang file, /usr/local/bin, etc., etc.?
Many thanks,
LRP
I keep a copy in /usr/local/bin, which I use when creating new apps. For example:
$ rebar create-app appid=myapp
All of my projects are built with rebar by placing the binary in the root directory of my project and referencing it locally in my Makefile as seen here. As of this writing, this seems to be the convention the community has adopted. For example, nitrogen, mochiweb, ibrowse, and erlydtl are all built this way.
Personally, I'm not a big fan of including the rebar binary with every project but I don't see a good alternative. It makes life easier for people that either don't have rebar in their path, or don't have an updated version in their path.
I suspect this is a temporary situation. Eventually the rebar project will start to have fewer commits, and we'll all agree to keep a copy in our path. At that point we can all stop including it. Of course, if your projects are only going to be used by you, this is a non-issue.