We are getting familiar with Erlang/OTP and Riak (Core, KV, Pipe, and so on) trying to build an open-source distributed application in turn. Our project is going to have a bunch of dependencies : tools like erlang_js, protobuffs, etc., but also services that we need to have running, such as Riak KV.
In Python/Ruby/Node.js, if modules are placed in a standard subdirectory relative to your project's, you can reference them, and later package them in releases. You can fire up a shell in the project's directory, play with your modules, do tests and so on, all just easily if good practices are followed.
What are the best practices for organizing a development environment in Erlang/OTP, with all dependencies reachable (and easily updatable to newest version), shell access to running nodes, testing, making releases, and so forth?
Check out rebar3: used for packaging, templating and managing releases of Erlang/OTP applications. You will find there an entire tutorial on how to use it.
Check out this tutorial on OTP first, before you can switch to this one which shows you releases and how they are handled in Erlang. Use this entire book as reference as you develop your project and because the author is still progressively adding more advanced stuff that you may need.
You may also want to keep reading about Erlang Applications and then do check out these quick links below:
Erlang Packaging, Process OneOTP Packaging Video by Chicago Boss GuysErlang Dependency ManagementRichard Jones Advice and examples on Packaging Erlang Projects with DependenciesMaven Tool and how it is used in packaging Erlang Projects
Most importantly, take rebar very seriously. It is very important in managing Erlang applications which have dependencies. You can keep posting your questions here on Stack Overflow for any assistance as you progress.
Before I forget, do check out a lot of stuff from the Riak Community.
Also, its important to check out the system documentation on creating target systems and also how to use Reltool to handle releases. The good thing with erlang is that it has several options of how to do something, as long as its easy to maintain your application in that way. With target systems, you will learn how to embed the Erlang VM, how to run Erlang applications on Solaris, VxWorks, and creating Erlang applications as Services on Windows NT using erlsrv.
Normally, we make sure that as an operating system is booting a Server, our application starts with it. Solaris so far has more customizations than any other OS as regards embedding Erlang/OTP VM. You can always communicate with (an) embedded Erlang VM(s) using Escript where by the escript creates a an erlang node which is allowed to connect to the embedded VM (so they have to share same cookie) and the embedded VM must have allowed the temporarily created Node to connect by executing net_kernel:allow(List_of_nodes)
Make sure to call this method to make your embedded VM allow connections only from an known strictly specified number of Nodes.
Related
Suppose I have a CLI application with subcommands and arguments (like application foo --bar baz). How can I package it for distribution without requiring user to install Erlang?
I know there's mix escript Mix task, but it builds a binary that requires Erlang to be installed, also Mix reference states that escripts should be used only for development purposes.
mix release, however, produces redundant shell scripts that I don't want to see in dist.
So, is there a way to make a standalone distributable of an Elixir CLI application?
P. S. This is actually my first experience with Elixir (and the whole Erlang ecosystem)
Elixir applications can be packaged as Erlang releases, see also here. These can include Erlang VM. Elixir since 1.9 supports building releases without extra tooling, using included mix task mix release - please check the documentation for greasy details: https://hexdocs.pm/mix/Mix.Tasks.Release.html
You might benefit from a quick look at this blog post for inspiration and conceptual overview, noticing that for simple CLI app it is much simpler: https://www.cogini.com/blog/best-practices-for-deploying-elixir-apps/
Bakeware generates a single executable file.
They have added a CLI app example here.
How to integrate an already created erlang OTP application with its dependencies into a single unit so that when the unit is started everything(processes) get started behind the scene?
You specify in the .app file which other applications your application depends on, like in this example. Then you build a release using reltool:create_target/2 like in this example. That function takes the definition of a release as a parameter which looks something like in this file. When Erlang VM boots the release it starts all the applications in the correct order.
There are a few third-party tools that make creating releases easier, like rebar, relx, mad, erlang.mk, or builderl (no documentation in the last one yet, used by the application that I pointed out earlier).
I am very new to Erlang programming language. Is there a standard build tool in Erlang?
I have googled out these, not sure which one I should use. I don't know that what kind of occasion is it used for?
Rebar2: It is first usable and de facto build tool that most of the Erlang projects are using it. It uses Erlang script for getting dependencies, compiling, testing and making release of your project. However it is not a modern build tool and suffers from slowness of compiling in developing phase, difficulty for using in larger projects and a bit hard to understand for newcomers.
Rebar3: It is a successor to rebar2 with an attempt to improving its mechanism and providing new features which is compatible with modern build tools. Also it is easier to use for newcomers
Erlang.mk: It is a big Makefile. As Makefile is fast and is available by default in every unix system, so you can benefit from these features for your Erlang application build tool. It has a package index of most well-known Erlang projects and other standard features like Rebar. Also it is faster that rebar2 in developing phase (preliminary results show that rebar3 is notably faster than Erlang.mk)
I myself use Rebar and this possible duplicate of your question has two answers that recommend Rebar as well. But it is a matter of taste and I recommend to consider two different approaches and choose what is closer to your purposes.
I have an Erlang application which has a dependency in its deps directory on another application.
From what I understand I can either;
a) start my dependent application from my including application by calling application:start(some_other_app) which starts the application and shows it running standalone within Observer.
b) include my dependent application in my .app file with {included_applications, [some_other_app]} so that the application is loaded and not started and then start the included application from my own top level supervisor. This again starts the included application and shows its running below my own supervision hierarchy in Observer.
My question is when should I use either approach? If I use option “a” and my dependent application exits will it be restarted or should I be using approach “b” so that any dependencies I have are monitored accordingly?
On a side note I use Rebar to package and manage my dependencies.
Thanks,
Andy.
You probably shouldn't do either a) nor b)
From Learn You Some Erlang
Look at the chapter: Included Applications
It is more and more recommended not to use included applications for a
simple reason: they seriously limit code reuse. Think of it this way.
We've spent a lot of time working on ppool's architecture to make it
so anybody can use it, get their own pool and be free to do whatever
they want with it. If we were to push it into an included application,
then it can no longer be included in any other application on this VM,
and if erlcount dies, then ppool will be taken down with it, ruining
the work of any third party application that wanted to use ppool.
For these reasons, included applications are usually excluded from
many Erlang programmers' toolbox. As we will see in the following
chapter, releases can basically help us do the same (and much more) in
a more generic manner.
In the chapter Release is the Word you can read about how several applications are bundled into a relase and how they are started.
Having your dependencies declared in your application descriptor is the way to go, so you should use option B in most of the scenarios.
The application controller will ensure that all your dependencies are present and started (in order) before starting your application and will also make your app fail if those terminate with errors. Also, the application controller will shutdown everything when needed.
Other than that, if you choose option A, when starting an application with application:start/1, you will get a temporary application by default, so you should use application:start/2, passing the permanent atom as the second argument.
EDIT: Having your dependencies in the application descriptor also helps visibility, it's easy to know your deps without scanning the source code.
I had a similar question, How do you include dependencies in an erlang project, and then how do you release them?
I had some help from various friends, and the erlang mailing list...and after re-reading some docs and more trial and error... I figured some stuff out. It's long, so check out the gist:
https://gist.github.com/3728780
-Todd
Apache Maven is a very popular build and dependency management tool in the Java open source ecosphere. I did some tests to find out if it can handle compiled Free Pascal / Delphi units and found it easy to implement. So it would be possible to
release open source libraries precompiled for Free Pascal (or Delphi) in a public Maven repository
include metadata in this repository which contains dependency information
use Maven on the command line to download the open source library from the public repository, and automatically resolve all dependencies
local repositories, working as proxies, could be used to cache frequently used binaries
automatic checksum generation and verification (provided by Maven) would reduce the risk of downloading corrupted binaries
source code and even documentation files could be provided with the binaries
binaries can be provided with or without debug information
continuous integration servers like Hudson, TeamCity or CruiseControl can be used to build projects whenever changes have been submitted to the source control system and notify developers about build errors
This way of dependency management could be very beneficial for open source projects which use many third party libraries with complex dependencies. It would avoid typical conflicts caused by using wrong versions.
For the developer, the workflow for editing and building a project would be reduced to a minimum:
checkout the project source from internal version control system
edit source file(s)
run mvn package to automatically download all required third party libraries (precompiled units) if they are not yet in the workstation's local repository
compile and run
The only additional file for Apache Maven which is required in the project folder is the POM.XML file containing the project information.
Edit: while Maven is usable for some of the required tasks, implementing a solution like Maven in native Free Pascal would have some advantages: no Java SDK required, support for all development platforms where Free Pascal is available, maintenance and plugin development in Pascal.
Usage of a Maven-like tool would not be helpful for open source projects only - commercial projects could access and use the artifacts in public Maven repositories in the same way as well.
Maven features are listed at http://maven.apache.org/maven-features.html
Update:
one use case could be the build of Lazarus, where Maven would download all required libraries and invoke the compiler with the necessary build path arguments. Changes in the dependencies on lower levels would be propagated automatically up to the parent build.
Possible benefits:
less time needed to set up a new work
station, no manual installation of
third party libraries required
less errors caused by wrong library
versions, detection of version
conflicts (for example if two
libraries depend on different
versions of a third library)
artifacts which are created inhouse
can be added to the local maven
repository and shared between
developers and project, central
storage of all artifacts with
metadata
builds are reproducible, just by
using the same source and project
metadata file (pom.xml)
can reduce development time and
increase project stability
Update #2: FPMake
the FPMake build system for Free Pascal seems to be a tool with much potential, in many details it is quite similar to Maven:
FPMake is a pascal based build system developed for and distributed with FPC
FPMake standardizes the building by defining some limits like standard directories
the command fppkg <packagename> will look in a database for the package, extract it, and then compile fpmake.pp and run it
it has standard build targets (clean, build, install, ...)
it can create a 'manifest' file suitable for import into a repository (like mvn deploy or mvn install), the manifest is an XML file which looks very similar to a pom.xml in Maven:
FPMake manifest file:
<packages>
<package name="my-package">
<version major="0" minor="7" micro="6" build="1"/>
<filename>my-package-0.7.6-1.zip</filename>
<author>my name</author>
<license>GPL</license>
<homepageurl>http://www.freepascal.org/</homepageurl>
<email>myname#freepascal.org</email>
<description>this is the package description</description>
<dependencies>
<dependency>
<package packagename="rtl"/>
</dependency>
</dependencies>
</package>
</packages>
Freepascal has been working on a package system of its own in a cross between apt-get and freebsd ports style. (download source/build/install automatically), called fppkg.
However work has stalled. People investing time are the bottleneck, not people wanting to choose tools.
As far as Maven goes, I don't like auxilary tools that need installation of huge external runtimes. It might be fine for a big major app (like Open Office), but not for an util.
I also prefer a tool that is designed to the FPC reality and workflow.
Documentation tools, build tools, download systems, testsuite systems are already all there, it just need a person that dedicates a lot of time into it to make it happen.
Some typical problems when introducing a new technology in a project as FPC, and why it has a tendency to make its own tools:
need to train 20+ committers in parttime.
The only COMMON programming language you can assume is Free Pascal. Even Delphi inner workings can't be taken for granted to be known (many committers came directly to FPC or even still via TP or a Mac Pascal)
Obviously that makes something with plugins in a different language annoying.
Bash script is a close second. (g)make third, but already a magnitude less.
All servers are *nix-like (FreeBSD, OS X, Linux), but not all run Apache. (e.g. my FreeBSD mirror runs XSHTTPD)
somebody most knowledgable must be dedicated maintainer for a long time. Fix problems, update/ do migrations etc. Perferably more than one for obvious reasons.
a major pain are Linux distributions (and FreeBSD to a lesser degree), most maintainers of *nix packages are not capable of more than "./configure;make;make install", and must be spoonfed with a near buildable repository and auxilary files.
In-distribution packaging of FPC/Lazarus has always been important, and is still increasing
All distributions have their own special rules about metadata, depedancies, and how sources must be published. Particularly Debian/Ubuntu is very bureaucratic and slow.
Most don't like third party auto-installers on top of their systems (since that bypasses their dependancy control)
This all leads to the effective practice that own tools in Pascal with minimal scripting work best. Some tools used:
Gmake is mainly used to parameterise the build process on a per directory level, a successor, fpcmake (not really a make derivative despite the name) has begun, but the migration hasn't completed.
Latex and a latex to html conversion (tex4ht, but debian uses hevea) are used in the documentation building (the non library documentation)
The community site (netscape community server which uses TCL scripting, a heavy complex application server) has been a trouble ever since it started, but specially lately since the maintainer became less active.
Mantis has been a problem (specially the email module would crash or lame the server due to the volume), but it has been whipped into shape during successive updates and hard work of several lazarus devels. Currently it is a decent workhorse.
lazarus.freepascal.org PHPBB forum OTOH is relatively painless since a lot of younger people know how to deal with it.
The same goes for subversions (though the more advanced scale needs some adjusting, not everybody is deep into the ins and outs of mergetracking)
If somebody was really serious about Maven, I usually would ask him:
to CRITICIALLY investigate the use for the project. In a very concrete way, with schedule and time estimates. Birds-eye level "everything's possible" overviews are essentialy worthless.
Give some thought on future change of used technologies. Every technology is eventually replaced, even the in-house ones, in 18 year+ projects. A new technology must not make migrations of other infrastructural components hard or involved. The new technology to end all new technologies doesn't exist.
Make a migration plan. Migration is often underrated and underestimated.
And in the end, there is always the 1000000 Euro question, who will do the daily maintenance?
Keep in mind that in a company you just kick the person responsible for the application server. But in an informal environment this is way harder, specially long term, since people's lives, occupations and time spent on the project vary.
Sounds like an interesting plan, but the Delphi community (and FPC even more so, I'd imagine!) values libraries as source far more than precompiled libraries. The general consensus is that anyone who uses a binary-only library is a fool, for two reasons: You can't fix any bugs you find in it, and compiler changes will break compatibility.