I have an erlang application I have been writing which uses the erldis library for communicating with redis.
Being a bit of a newbie with actually deploying erlang applications to production, I wanted to know if there was anyway to 'bundle' these external libraries with the application rather than installing into my system wide /usr/lib/erlang/lib/ folder.
Currently my directory structure looks like...
\
--\conf
--\ebin
--\src
I have a basic Makefile that I stole from a friend's project, but I am unsure how to write them properly.
I suspect this answer could involve telling me how to write my Makefile properly rather than just which directory to plonk some external library code into.
You should really try to avoid project nesting whenever possible. It can lead to all sorts of problems because of how module/application version is structured within Erlang.
In my development environment, I do a few things to simplify dependancies and multiple developed projects. Specifically, I keep most of my projects sourced in a dev directory and create symlinks into an elibs dir that is set in the ERL_LIBS environmental variables.
~/dev/ngerakines-etap
~/dev/jacobvorreuter-log_roller
~/dev/elib/etap -> ~/dev/ngerakines-etap
~/dev/elib/log_roller -> ~/dev/jacobvorreuter-log_roller
For projects that are deployed, I've either had package-rpm or package-apt make targets that create individual packages per project. Applications get boot scripts and init.d scripts for easy start/stop controls but libraries and dependancy projects just get listed as package dependencies.
I use mochiweb-inspired style. To see example of this get your copy of mochiweb:
svn checkout http://mochiweb.googlecode.com/svn/trunk/ mochiweb
and use
path/to/mochiweb/scripts/new_mochiweb.erl new_project_name
to create sample project of the structure (feel free to delete everything inside src afterwards and use it for your project).
It looks like this:
/
/ebin/
/deps/
/src/
/include/
/support/
/support/include.mk
Makefile
start.sh
ebin contains *.beam files
src contains ***.erl files and local *.hrl files
include contains global *.hrl files
deps contains symlinks to root directories of dependencies
Makefile and include.mk takes care of including appropriate paths when project is built.
start.sh takes care of including appropriate paths when project is run.
So using symlinks in deps directory you are able to fine tune the versions of libraries you use for every project. It is advised to use relative paths, so afterwards it is enough to rsync this structure to the production server and run it.
On more global scale I use the following structure:
~/code/erlang/libs/*/
~/code/category/project/*/
~/code/category/project/*/deps/*/
Where every symlink in deps points to the library in ~/code/erlang/libs/ or to another project in the same category.
The simplest way to do this would be to just create a folder named erldir and put the beams you need into it and then in your start script just use the -pa flag to the erlang runtime to point out where it should fetch the beams.
The correct way (at least if you buy into the OTP distribution model) would be to create a release using reltool (http://www.erlang.org/doc/man/reltool.html) or systools (http://www.erlang.org/doc/man/systools.html) which includes both your application and erldis.
Add the external libraries that you need, anywhere you want them, and add them to your ERL_LIBS environment variable. Separate the paths with colon in unix or semicolon in dos.
Erlang will add the "ebin"-named subdirs to its code loading path.
Have your *.app file point out the other applications it depends on.
This is a good halfway-there approach for setting up larger applications.
Another way is put your lib path in ~/.erlang.
code:add_pathz("/Users/brucexin/sources/mochiweb/ebin").
code:add_pathz("/Users/brucexin/sources/webnesia/ebin").
code:add_pathz("./ebin").
code:add_pathz("/Users/brucexin/sources/erlang-history/ebin/2.15.2").
Related
Let's say, I have a completely new VPS server which I've just rolled out, which I haven't installed anything on yet.
And I've compiled and build a production release of Phoenix application on my local machine which is identical to a VPS server Linux distributive- and version-wise.
In the directory _build/prod/rel/my_app123 there have been generated 4 subdirectories:
bin
erts-12.3
lib
releases
Will copying the content of rel/my_app123/, that is, these 4 subdirectories, over to a VPS will be absolutely enough in order to run an application?
Or will I have install something extra as well? Elixir and Erlang?
How about production dependencies from mix.exs? Or are these have been included and compiled into into a release?
P.S. Assume that my web application has no "js", "css" and the like files, and doesn't use a database.
When you run mix release, it bundles all of your Elixir/Erlang dependencies for the MIX_ENV in question into the release directory, the erlang BEAM runtime/VM that you were using in your build, and any files that you specify in your mix project in mix.exs.
Because the BEAM runtime and code that bootstraps loading your code are included in the release, you won't need to install Elixir or Erlang on the target machine.
Things that are not included include:
any non-Elixir dependencies. For example, if you rely on openssl, you'll need to make sure you have a binary-compatible version of that installed on the machine you plan to run on (typically, the equivalent major verson release).
Portable bytecode. BEAM isn't like the Java VM. The compiled BEAM code needs to run on a substantially similar architecture. Build on an Arm64 machine for deployment on an Arm64 virtual machine, or x86 for Intel-compatible hardware, for instance. And it's probably best to use the same major OS distribution. There may be cases where "Any Linux * Same CPU architecture" is fine, but for example, building on a Windows or MacOS install of Elixir/OTP and deploying on Linux is a non-starter; you'd need to use a sufficiently similar OS.
As an example, one of my projects has its releases built on Alpine using Docker, so we only really have to worry about CPU compatibility. In our case we do need to make sure some external non-Elixir dependencies our app binds to are included on the docker image.
RUN apk add --no-cache libstdc++ openssl ncurses-libs wkhtmltopdf xvfb \
fontconfig \
freetype \
ttf-dejavu
(ignore the fact that wkhtmltopdf is kind of deprecated, we're working on it. But for now it's a non-elixir dependency we rely on).
If you're building for a, say, an EC2 instance and not using Docker, you'd just need to make sure your release is built on a similar OS to what you're using for production, and make sure the production AMI (image) has those non-Elixir dependencies on it, or will at the time of deployment, perhaps using apt or another package manager. For a VPS, the solution for non-elixir dependencies will depend on whether they have the option for customizing the base machine image (maybe with Packer or Ansible)
Since you may seem to have been a bit confused about it in the comments, yes, MIX_ENV=prod mix release will build all of your production Elixir/Erlang dependencies and include them in the /_build/prod folder.
I include the whole ./prod folder in our release, but it looks like protocol consolidation binaries and the lib folder .Beam files are all in the rel folder so that's a bit unnecessary.
If you do a default build, the target will be inside your _build directory, with sub-directories for the config environment and your application, e.g. _build/dev/rel/your_app/. That directory should contain everything you need to run your app -- the prompt after running mix release provides some clues for this when it says something like:
Release created at _build/dev/rel/your_app!
I find it more useful, however, to zip up the app into a single portable file (and yes, I agree that the details about how to do this are not necessarily the first things you see when reading about Elixir releases). The trick is to customize your mix.exs by fleshing out the releases option -- this is usually done via a dedicated private function but the organization of how you supply the options is up to you.
What I find is often useful is the generation of a single zipped .tar.gz file. This can be accomplished by specifying the include_executables_for option along with steps. It looks something like this:
# mix.exs
defmodule YourApp.MixProject do
use Mix.Project
def project do
[
# ...
releases: releases()
# ...
]
end
defp releases do
[
my_app: [
include_executables_for: [:unix],
steps: [:assemble, :tar]
]
]
end
When you configure your application this way, running mix release will generate a nice portable file containing your app with everything it needs. Unzipping this file is education for understanding everything your app needs. By default this file will be created at a location like _build/dev/yourapp-1.0.0.tar.gz. You can configure the build path by specifying a path for your app. See Mix.Release for more options.
Background, I currently use Izpack for my Windows installer, I bundle a java runtime and use winrun4j as a wrapper both for the installation and the actual program once installed. It worked for a long time but there are a number of problems with the installer that I have not been able to solve and have been looking to replace it.
Oracle now provide the JPackage installer so it seems like a sensible choice. But the folder structure created by the installer is different to what I currently have, I have a number of config and non java files and I have not been able to get the .exe that JPackage creates to do anything.
So is it possible to use JPackage to create the installer but in a strcuture better matching my existing structure, and use continue to use WInRun4j to actyally run my application
Existing Folder Structure
ROOT
---App.exe
---Config Files
---lib
-------jar files
---JVM64
------- Java runtime
---help
JPackage structure
ROOT
---App.exe
---Runtime Dlls
---app
----- jar files
Config files
--runtime
------Java runtime
------Runtime Dlls (again)
The structure of directories generated by jpackage is mainly set up for you and does not seem possible to change, and makes installation of Java app dependencies very easy with self contained JRE. The basic structure for Windows is as you say:
ROOT
---App.exe (for --main-class parameter)
---xyz.exe (for each --add-launcher parameter)
---Runtime Dlls (these appear to be unused except for applauncher.dll, see SO 62607300)
---app/
------App.cfg (for --main-class)
------xyz.cfg (for per --add-launcher)
---runtime/
------Java runtime
------Runtime Dlls
With --input and --main-jar params you are free to setup additional directory structure under app/ folder for anything else you want for your application. So if you used lib/myappjar.jar it would add:
---app/
-------lib/
----------myappjar.jar
If you used --input build\mypath it would copy the entire tree of files under that folder, so if build\mypath dir contained
bin/
---Scripts
---xyz.properties
README.txt
Then app would also contain:
---app/
------bin/
---------Scripts
---------xyz.properties
------README.txt
By the way the Runtime DLLs placed at top level appear to be copies of some of the DLLs under runtime/bin
[https://stackoverflow.com/questions/62607300/why-is-java-jpackage-installing-windows-dll-files-in-two-places]
I have cloned and built the waf script using:
./waf-light configure
Then to build my project (provided by Gomspace) I need to add waf and the eclipse.py to my path. So far I haven't found better than this setenv script:
WAFROOT=~/git/waf/
export PYTHONPATH=$WAFROOT/waflib/extras/:$PYTHONPATH
export PATH=~/git/waf/:$PATH
Called with:
source setenv
This is somehow a pretty ugly solution. Is there a more elegant way to install waf?
You don't install waf. The command you found correctly builds waf: /waf-light configure build Then for each project you create, you put the built waf script into that projects root directory. I can't find a reference, but this is the way in which waf:s primary author Thomas Nagy wants the tool to be used. Projects that repackage waf to make the tool installable aren't "officially sanctioned."
There are advantages and disadvantages with non-installation:
Disadvantages:
You have to add the semi-binary 100kb large waf file to your repository.
Because the file contains binary code, people can have legal objections to distributing it.
Advantages:
It doesn't matter if new versions of waf break the old API.
Users don't need to install waf before compiling the project -- having Python on the system is enough.
Fedora (at least Fedora 22) has a yum package for waf, so you could see that it's possible to do a system install of waf, albeit with a hack.
After you run something like python3 ./waf-light configure build, you'll get a file called waf that's actually a Python script with some binary data at the end. If you put it into /usr/bin and run it as non-root, you'll get an error because it fails to create a directory in /usr/bin. If you run it as root, you'll get the new directory and /usr/bin/waf runs normally.
Here's the trick that I learned from examining the find_lib() function in the waf Python script.
Copy the waf to /usr/bin/waf
As root, run /usr/bin/waf. Notice that it creates a directory. You'll see something like /usr/bin/.waf-2.0.19-b2f63c807a4215294bf6005410c74c18
mv that directory to /usr/lib, dropping the . in the directory name, e.g. mv /usr/bin/.waf-2.0.19-b2f63c807a4215294bf6005410c74c18 /usr/lib/waf-2.0.19-b2f63c807a4215294bf6005410c74c18
If you want to use waf with Python3, repeat Steps 2-3 running the Python script /usr/bin/waf under Python3. Under Python3, the directory names will start with .waf3-/waf3- instead instead of .waf-/waf-.
(Optional) Remove the binary data at the end of /usr/bin/waf.
Now, non-root should be able to just use /usr/bin/waf.
That said, here's something to consider, like what another answer said: I believe waf's author intended waf to be embedded in projects so that each project can use its own version of waf without fear that a project will fail to build when there are newer versions of waf. Thus, the one-global-version use case seems to be not officially supported.
I'm attempting to understand how to build a multiple-application Erlang OTP release using Rebar.
My applications include Webmachine, Riak, and a few applications of my own devising. So far, I have been running Rebar successfully in the application root directory. But the Rebar bootstrap program suggests that there may be a better way. E.g.:
"You now have a self-contained script called "rebar" in
your current working directory. Place this script anywhere in your path
and you can use rebar to build OTP-compliant apps."
I can think of several ways to do this. But I'm wondering:
Is there is a preferred convention: e.g., placing Rebar in an .erlang file, /usr/local/bin, etc., etc.?
Many thanks,
LRP
I keep a copy in /usr/local/bin, which I use when creating new apps. For example:
$ rebar create-app appid=myapp
All of my projects are built with rebar by placing the binary in the root directory of my project and referencing it locally in my Makefile as seen here. As of this writing, this seems to be the convention the community has adopted. For example, nitrogen, mochiweb, ibrowse, and erlydtl are all built this way.
Personally, I'm not a big fan of including the rebar binary with every project but I don't see a good alternative. It makes life easier for people that either don't have rebar in their path, or don't have an updated version in their path.
I suspect this is a temporary situation. Eventually the rebar project will start to have fewer commits, and we'll all agree to keep a copy in our path. At that point we can all stop including it. Of course, if your projects are only going to be used by you, this is a non-issue.
I have a machine which doesn't give me local admin rights. Is it still possible to run erlang on it, as I cannot run a windows .exe installer to install erlang?
You can copy erl.exe (plus the runtime system and all the libraries you need) from another installation and run it without the need to install. As long as you are allowed to execute files it should be okay.
Forgive me for not being as smart as Zubair,
but I would like to know exactly how to do this.
I do not have admin privilege, cannot run installers, and cannot copy files to C:\WINDOWS.
In particular, I cannot write to C:\WINDOWS\WinSxS or C:\WINDOWS\system32.
How do I get a list of exactly what libraries are required by the various erlang executables ?
I have all the MS redistributable libraries and manifests,
but I don't know where to put them to make it work.
The redistributable library structure has directories such as
Microsoft.VC90.ATL, Microsoft.VC90.CRT, etc. Each directory contains relevant dlls and a manifest.
Do I copy all the contents into the ERL_HOME\bin directory or ERL_HOME\erts-x.y.z\bin or ERL_HOME\erts-x.y.z\lib ?
or leave them in some other directory and put those entries in the PATH ?
or do I need to build the paths implied by the manifests (i.e. where they would be copied into the WinSxS cache) using hashes and version numbers in the paths, then put those entries in the PATH ?