I'm attempting to understand how to build a multiple-application Erlang OTP release using Rebar.
My applications include Webmachine, Riak, and a few applications of my own devising. So far, I have been running Rebar successfully in the application root directory. But the Rebar bootstrap program suggests that there may be a better way. E.g.:
"You now have a self-contained script called "rebar" in
your current working directory. Place this script anywhere in your path
and you can use rebar to build OTP-compliant apps."
I can think of several ways to do this. But I'm wondering:
Is there is a preferred convention: e.g., placing Rebar in an .erlang file, /usr/local/bin, etc., etc.?
Many thanks,
LRP
I keep a copy in /usr/local/bin, which I use when creating new apps. For example:
$ rebar create-app appid=myapp
All of my projects are built with rebar by placing the binary in the root directory of my project and referencing it locally in my Makefile as seen here. As of this writing, this seems to be the convention the community has adopted. For example, nitrogen, mochiweb, ibrowse, and erlydtl are all built this way.
Personally, I'm not a big fan of including the rebar binary with every project but I don't see a good alternative. It makes life easier for people that either don't have rebar in their path, or don't have an updated version in their path.
I suspect this is a temporary situation. Eventually the rebar project will start to have fewer commits, and we'll all agree to keep a copy in our path. At that point we can all stop including it. Of course, if your projects are only going to be used by you, this is a non-issue.
Related
I'm trying to figure out how to properly setup an ejabberd project that allows for easy compilation of custom beam files- so far, we've been using an existing project that is cumbersome to manage, and uses erlide as the IDE.
I would like to set up the project in a way that I can use a more helpful IDE like vscode, and somehow streamline the compiling and copying of the beam files and updating the module on the server.
Writing code in Elixir is fine as well- I just want the project to be set up in a way that is dev friendly.
Apologies if the question is too broad, but I'm not exactly sure how else to best phrase it. If you feel like I'm missing something in my current flow, please let me know, as I've basically inherited this project. If there are any clarifications required, let me know as well.
Thanks.
easy compilation of custom beam files
somehow streamline the compiling and copying of the beam files and updating the module on the server.
If the task is about compiling and loading additional modules, a running ejabberd node can compile, load and start additional modules in runtime, see
https://docs.ejabberd.im/developer/extending-ejabberd/modules/#ejabberd-contrib
Usually the modules come from
https://github.com/processone/ejabberd-contrib
but you can tell ejabberd to download other modules from other git repositories, or you can copy modules source code and tell ejabberd to install them. And those modules can be written in Erlang or Elixir. Full example: https://docs.ejabberd.im/developer/extending-ejabberd/elixir/#elixir-module-in-ejabberd-contrib
Basically:
you write the module in your development machine, test it...
when happy with the source code, copy mod_whatever.erl to the production machine, $HOME/.ejabberd-modules/sources/mod_whatever as explained in the example mentioned earlier
run ejabberdctl module_install mod_whatever
In step 2, instead of copying the source code yourself, you can have a git repository just for your module, tell ejabberd the module's git URL, similarly to https://github.com/processone/ejabberd-contrib/tree/master/extra
BTW, for step 3, starting in ejabberd 22.10, there's a page in ejabberd webadmin to install and uninstall those modules (copying the files requires manual administration of course).
I would like to set up the project in a way that I can use a more helpful IDE like vscode
What a coincidence, these days I'm playing with VSCode variants (VSCode, VSCodium, Coder's code-server and Github Codespaces) and how to develop ejabberd using them. This is useful for step 1 that I mentioned earlier (write module and test it). If you are interested in ejabberd + VSCode, tell me.
When using:
$ rebar3 as test eunit
it compiles the code into ebin, but the other directories are symlinked in the _build/test/lib folder. I've tried using profile test by modifying:
{relx, [{dev_mode, false}]}
This works only for _build/test/rel directory but not the lib directory. So during tests they are referencing the lib symlink directories. Is there a way to have these directories not symlinked to the original, but actual copies provided like the release?
After a quick look at the rebar3 code there doesn't seem to be a way to force copying of these directories. It looks like the priv directory is always symlinked here. Even though the function used is called symlink_or_copy, it only ends up copying when there is an error while creating the symlink.
dev_mode is a relx option, that's why it doesn't affect rebar3 features.
If you want this feature added you can create a feature request explaining your use case, why you think it would be useful and it might get implemented.
I would like to build and deploy an application which has Django as frontend, YAWS (appmods) or Mochiweb/Webmachine as a backend and CouchDB as a datastore. Furthermore, I plan to extensively use CouchDB's ability to replicate in order to provide high fault tolerance for the whole application.
I tend to think that in order to achieve this, I must create a single OTP release which has YAWS and CouchDB as Erlang/OTP applications.
Does this approach seem to be correct? How can I organize YAWS and CouchDB in terms of OTP applications in order to create a solid production setup? Are there any best practices for doing that?
Erlang releases are a good way to package software, because they include all of the dependent libraries. This way, different erlang applications can run different versions of their required libraries without conflicting with each other.
A popular way of dealing with the complicated release process Erlang has is to let Rebar do it for you. They have a quick start guide to get you on the right path. I don't know if you're currently using Rebar to manage your project, but it makes things a lot easier. It is definitely worth looking into if you aren't using it yet.
However, rebar won't include your dependencies right out of the box. In order to do that, you should modify the reltool.config file and all your required applications.
First step is to add the directory where all your dependencies are located, e.g.:
{sys, [
...
{lib_dirs, ["../deps"]},
...
}.
Then, add your dependencies as applications to include in the release:
{sys, [
...
{app, jiffy, [{incl_cond, include}]},
{app, cowboy, [{incl_cond, include}]},
...
}.
Now, when you run the rebar generate command, your applications should be in the target directory under lib:
find brawl_server -type d -maxdepth 2
brawl_server
brawl_server/bin
brawl_server/erts-5.9.1
brawl_server/erts-5.9.1/bin
brawl_server/lib
brawl_server/lib/brawl_server-1.1
brawl_server/lib/cowboy-0.6.0
brawl_server/lib/jiffy-0.6.1
brawl_server/lib/kernel-2.15.1
brawl_server/lib/sasl-2.2.1
brawl_server/lib/stdlib-1.18.1
brawl_server/log
brawl_server/log/sasl
brawl_server/releases
brawl_server/releases/1
You also need to make sure your own OTP application knows it needs to start the applications it depends on. If you have a generated Rebar project, modify your <appname>.app.src file to include them:
{application, app, [
...
{applications, [
jiffy,
cowboy,
kernel,
stdlib
]},
...
}.
You should be able to compile and generate the release and run it with the included scripts, as laid out in the Rebar release handling article.
At this point, though, there's an added wrinkle, because apparently vanilla CouchDB isn't OTP compliant, and you can't include it this way. There is another distribution you may be able to use, instead: rcouch. I haven't tried it myself, hopefully it will work for you.
Further reading:
Learn you some erlang for great good: Releases
I used my own project, brawl-online, as an example
I would recommend to create DEB/RPM/you-name-it package or packages. CheckInstall is the simplest solution for this.
I am in the process or learning erlang OTP and rebar and I have put together a small example app using a couple of dependencies: cowboy and lager.
I have issued
rebar get-deps
rebar compile
And things went smoothly. Now I want to fire up my console to test things around but it is not obvious to me how to start the dependency applications.
I tried issuing a
rebar generate
In order to get all the orchestration of firing up the apps, even though it's overkill for just development tests, but I miserably failed getting the following dump
Crash dump was written to: erl_crash.dump
eheap_alloc: Cannot allocate 1459620480 bytes of memory (of type "old_heap").
Aborted
The ebin dir only has beam files for the app I wrote but not the dependencies, I see the dependencies have their own ebin directory inside the main app deps directory, how would I go about having them available in a console to start them up?
I would appreciate if someone can shed some light as to what the common practice is for the dev env with multiple OTP apps.
I have read a couple of tutorials but they are mostly targeted at the rebar release cycle and not the development process.
TIA
In your case, the modules you pull into the deps directory should typically be called from within your application code - and your application can be typically invoked from the Erlang shell using the application:start/1 function. If you haven't yet, I strongly suggest that you read Chapter 12, "OTP Behaviors", of Francesco Cesarini's excellent book Erlang Programming - it's a great practical introduction for what you're attempting.
I have an erlang application I have been writing which uses the erldis library for communicating with redis.
Being a bit of a newbie with actually deploying erlang applications to production, I wanted to know if there was anyway to 'bundle' these external libraries with the application rather than installing into my system wide /usr/lib/erlang/lib/ folder.
Currently my directory structure looks like...
\
--\conf
--\ebin
--\src
I have a basic Makefile that I stole from a friend's project, but I am unsure how to write them properly.
I suspect this answer could involve telling me how to write my Makefile properly rather than just which directory to plonk some external library code into.
You should really try to avoid project nesting whenever possible. It can lead to all sorts of problems because of how module/application version is structured within Erlang.
In my development environment, I do a few things to simplify dependancies and multiple developed projects. Specifically, I keep most of my projects sourced in a dev directory and create symlinks into an elibs dir that is set in the ERL_LIBS environmental variables.
~/dev/ngerakines-etap
~/dev/jacobvorreuter-log_roller
~/dev/elib/etap -> ~/dev/ngerakines-etap
~/dev/elib/log_roller -> ~/dev/jacobvorreuter-log_roller
For projects that are deployed, I've either had package-rpm or package-apt make targets that create individual packages per project. Applications get boot scripts and init.d scripts for easy start/stop controls but libraries and dependancy projects just get listed as package dependencies.
I use mochiweb-inspired style. To see example of this get your copy of mochiweb:
svn checkout http://mochiweb.googlecode.com/svn/trunk/ mochiweb
and use
path/to/mochiweb/scripts/new_mochiweb.erl new_project_name
to create sample project of the structure (feel free to delete everything inside src afterwards and use it for your project).
It looks like this:
/
/ebin/
/deps/
/src/
/include/
/support/
/support/include.mk
Makefile
start.sh
ebin contains *.beam files
src contains ***.erl files and local *.hrl files
include contains global *.hrl files
deps contains symlinks to root directories of dependencies
Makefile and include.mk takes care of including appropriate paths when project is built.
start.sh takes care of including appropriate paths when project is run.
So using symlinks in deps directory you are able to fine tune the versions of libraries you use for every project. It is advised to use relative paths, so afterwards it is enough to rsync this structure to the production server and run it.
On more global scale I use the following structure:
~/code/erlang/libs/*/
~/code/category/project/*/
~/code/category/project/*/deps/*/
Where every symlink in deps points to the library in ~/code/erlang/libs/ or to another project in the same category.
The simplest way to do this would be to just create a folder named erldir and put the beams you need into it and then in your start script just use the -pa flag to the erlang runtime to point out where it should fetch the beams.
The correct way (at least if you buy into the OTP distribution model) would be to create a release using reltool (http://www.erlang.org/doc/man/reltool.html) or systools (http://www.erlang.org/doc/man/systools.html) which includes both your application and erldis.
Add the external libraries that you need, anywhere you want them, and add them to your ERL_LIBS environment variable. Separate the paths with colon in unix or semicolon in dos.
Erlang will add the "ebin"-named subdirs to its code loading path.
Have your *.app file point out the other applications it depends on.
This is a good halfway-there approach for setting up larger applications.
Another way is put your lib path in ~/.erlang.
code:add_pathz("/Users/brucexin/sources/mochiweb/ebin").
code:add_pathz("/Users/brucexin/sources/webnesia/ebin").
code:add_pathz("./ebin").
code:add_pathz("/Users/brucexin/sources/erlang-history/ebin/2.15.2").