I am about to build a system with Erlang (my problem can be solved better with multiple CPUs) and I have skimmed through:
Learn You Some Erlang;
Erlang/OTP in Action
Programming Erlang (Armstrong);
Erlang Programming (Cesarini).
The Erlang/OTP in Action book (2) says: "...writing non-OTP Erlang code is really an advanced topic, and something that should be done only when you really really have to. So perhaps not ever having done it the non-OTP way is a blessing because you will pick up the right habits straight away with regard to things OTP..."
(2) insists that commercial systems should be built with OTP only, whilst (4) gently shows that OTP is not the only way to do this.
Additionally, by reading (4) I have found out that building the knowledge from the very basics of Erlang helps to understand how OTP works (but not in the opposite way).
So, the question is:
Should I choose Erlang/OTP to build a commercial system or can it be done with pure Erlang?
Learn and play with pure Erlang.
Build systems in Erlang/OTP.
can it be done with pure Erlang?
Yes, it can, subject to the analogue of Greenspun's Tenth Rule: your system will contain an ad hoc, informally-specified, bug-ridden, slow implementation of half of OTP.
OTP is written in Erlang so it definitely can be done in pure Erlang. That being said OTP provides you with a lot of useful functionality which you would otherwise need to write yourself. #Alexey is right than you will end up implementing a portion of OTP along the way. The problem is that the framework you need for building fault-tolerant systems, which OTP provides, has to itself be very fault-free.
I will also echo #Mazen in that experimenting with doing it in pure Erlang gives you a good feel for what is going on internally in OTP and how you can build fault-tolerant systems in Erlang. You will make better use of OTP afterwards. You will come to a better understanding when not to use OTP, and why.
If you read through the "Learn You Some Erlang" (like you said you have) you'll notice the author spends the Designing a Concurrent Application chapter building something half-useful in "plain Erlang". Then in the next 2 chapters, he basically throws it all away, and uses OTP to achieve the same thing with far less code (add to that the fact that because it's OTP code, it's standard - so other developers with familiarity with OTP will have a much better idea of where to look to understand it!)
The OTP has been developed and refined over a lot of years (initial release 1998 according to Wikipedia) with a large number of developers, and the backing of a large international company (Ericsson), and peer reviewed as it's all open source. I think it'd be a little audacious to assume you could single-handledly build something equally robust and scalable!
So use the OTP. Even for "play" projects, still use the OTP as it can only help you learn to make your "commercial" code better :)
Related
This question is in the context of the Beam VM and the capabilities that it provides, not in the general context of what a Turing complete language can do. I want to invest some time to learn either pure Erlang or Elixir. I get the basic differences between the two and I am leaning towards Elixir because of the macros, better syntax and faster development of the language this day.
My question is: if I choose Elixir, will I stumble on something that I cannot do in it, but can do in Erlang? Can I use all the OTP stuff, all the Erlang libraries, code reload, etc. with Elixir?
I am not asking for someone's preference between the two; just facts about the possibilities of the languages. Preferably from someone who used both in production.
You shouldn't stumble on anything you can do in one that you can't in the other, since you can freely call Elixir code from Erlang and vice-versa. You can even easily mix Erlang and Elixir files in one project.
In Elixir:
:erlang_module.erlang_function(args)
in Erlang:
'Elixir.ElixirModule':elixir_function(args)
TL;DR - Start with Elixir
Erlang has a steeper learning curve compared to Elixir. Once you start learning Elixir, you will automagically start learning Erlang. Hence, start out with Elixir. Elixir is written in Erlang and Elixir. See distribution on Github (since Elixir is full of macros aka meta-programming).
You can use Elixir with Erlang and vice-versa, hence the full Erlang eco-system of 20+ years of libraries.
More details from Erlang Solutions
Elixir’s ‘out of the box’ productivity is accomplished by a strong focus on tooling and convenience in expression of data manipulation. System design is the same in Elixir and Erlang, but Elixir removes a lot of boilerplate code and is easier to extend. The removal of boilerplate raises productivity and allows programmers to get feedback faster – crucial when you want to launch your product to market as fast as possible. Less boilerplate also makes for happy developers, and happy developers are unsuprisingly productive developers.
Joe Armstrong's (Erlang inventor's) blog post about Elixir
Start here to learn about Elixir - Getting Started
Once you feel its going well, work your way towards practicing on Exercism and other resources.
Just to preface - I have only used Elixir in production and not Erlang.
I would honestly recommend Elixir. This is my opinion and not necessarily the right one for you, but below I will list my reasons why.
Productivity: I come from a Ruby/Rails background so the Elixir syntax and style is something that was very familiar to me. One of the main factors that helps me determine whether or not to learn a language is how productive I can be in it - mainly why I chose Ruby. Elixir is the same. I can work just as fast in it as I can Ruby with all the added benefits of concurrency and pattern matching.
Erlang: Since Elixir is built on top of Erlang and compiles down to erlang and the beam vm, you have access to every erlang module and package. So if you are worried about using elixir and missing out on all of the features of Erlang, you shouldn't be. Elixir even has it's own implementations of most of the bigger Erlang/OTP features such as GenServer, GenEvent etc.
Community/Resources: The Elixir community is really a great one. The slack channel is really popular and great way to get some answers for beginner questions. There are already some really good books written on the language (Programming Elixir 1.2 - Dave Thomas, Author of the Ruby Pickaxe) also that make getting into the language really easy.
Really, if you have a mess around with the two you will probably come to the same decision that Elixir is a much nicer language with all the features of Erlang + More. It's on the rise as well, I can't remember the exact numbers but I remember reading something from the Hex website (package manager) about a considerable increase in package downloads.
There are a few things. I think you can't make a recursive anon function in Elixir. Now to be fair this is something that in 8+ years of Erlang I don't think I have ever needed to do, and if I did it could be done easily in some other way. There are probably a few other things like that that quite honestly you can ignore.
In general, for things that most people care about anything you can Do in Erlang, you can do in Elixir and vice versa. The community on the Elixir side seems to be more active so I would suggest starting there. I have recently moved from Erlang to Elixir and with the exception of a few fancy things in Proper I can say that it is a pretty easy transition
in the early phase of design of erlang small app - how do you do prototyping?
Is it better to first prototype without OTP just to prove all main mechanics in plain erlang and in further elaboration add what OTP offers with refined requirements / aspects or use OTP from the beginning?
(The answer below is not trying to plug my instructional, it just happens to apply directly to the OP's question; were it possible I would just send the OP a private message or email. At the time of this answer my demonstration system is only barely worth even reading, aside from basic architecture concepts.)
I start with a slew of function stubs. I do this in most languages (even something like this in assembler). The special thing about this in Erlang is that my initial stubs represent supervisors or logical managers, not one-off solutions to elements of my fundamental problem.
Beyond that, I like to do something most people abhor these days: talking the problem out in prose to discover inconsistencies in the way I view the problem. I've just started on an example of this here (as in, I'm still working on this before and after work daily as of today, 2014.11.06): http://zxq9.com/erlmud.
Some system stubs (conceptual, not OTP -- which is integral to the idea I'm trying to demonstrate in the project, actually) are here: https://github.com/zxq9/erlmud/tree/be7c6a8ae0d91aac37850083091ae4d15f1369a4/erlmud-0.1 for example. Over the next few days they will change significantly until there is a prototype system that works instead of just stubs. If you're really curious about this, follow the commits from the one I linked over the next two weeks or so (paid work schedule permitting, of course).
One positive thing I've noticed about prototyping with stubs and not jumping straight into OTP behaviors is that very often the behavior that is assumed to be a proper fit for a component turns out not to be. There are many cases where I anticipate I will want a gen_server, but after writing some stubs and messing around a bit I find myself beginning to manually implement an FSM. Sometimes that happens in reverse, too, I think I need an FSM and wind up writing a server, or realize I could benefit from a proper gen_event. Once you've ironed out what you're doing it is pretty easy to convert pure Erlang into OTP. It is much less easy to edit your mental model of how a component works once you've written a gen_fsm or gen_server, because you start to feel invested in the idea of thinking of it in OTP terms prematurely.
Remember: typing is the easy part, the real battle is figuring out what to type. So begin boldly by writing executable stubs and toy with them.
There is no special recipe to do prototypes in Erlang. How would you do a prototype in Java, C#, Scala, (put any language here) ?
When prototyping, you need to achieve your proof of concept as fast as possible and deliver a minimal vital project.
In your case, does OTP helps you to deliver your minimal vital project or not?
If yes, then use it. And of course don't use it if it isn't.
Are you familiar with OTP concepts in the first place? If not, then you need to learn them. And thats mean that you need to invest more time in learning OTP. Is that ok for your prototyping purpose?
I'm only trying to highlight the fact that prototyping in Erlang isn't different from any other language.
I'm starting to dive into Erlang for the first time, and OTP is held aloft by lovers and critics alike as being the gold standard for highly available, distributed processing.
Given that OTP has been around for decades and is openly documented, why is it that other languages supporting lightweight threads/processes haven't adopted versions of their own? Are there technical/political challenges? Or does everyone just shrug and learn Erlang?
Thanks!
The largest issue is that most language runtimes don't have built-in lightweight concurrency and error isolation with exit signal propagation. Without those things you would have a really hard time properly porting OTP.
For the languages that do have the right kind of runtime, I am seeing some effort or at least plans to build OTP inspired frameworks. Cloud Haskell is the first that comes to mind. I also expect that Go and Rust will eventually have something like OTP if they don't already.
There are technical challenges, as Erlang itself is designed for the same features OTP is known for. Case in point, Basho Riak is a distributed fault-tolerant key/value store written in Erlang. One might be able to port it to Haskell or some similar functional language, but it would probably be a lot of work. Just for fun, you might look into OTP stuff written in the Elixir language.
Actually, it has been (tried).
Akka is the library which takes some OTP features and implements them in Scala for JVM.
Given the principles underlying JVM and BEAM (the Erlang VM) are very different (mainly GC, scheduling and message passing are radically different), I can't say how successful that implementation is and how many benefits of the original OTP it preserves. There's a lot of (heated) debate on that in the internets.
I want to write actor style code for embedded processors and I am trying to decide between writing everything in Erlang vs writing everything in zeromq+any language. Using zeromq looks to be very powerful in the sense that I can use any programming language and make my development a lot easier(many available libs) but then I am not sure if there is any gotcha in this power? I understand that Erlang represent actor model much better especially with OTP concepts but then it seems easy to represent similar actor model with zeromq? Am I looking at this correctly?
1.What do I really lose not using Erlang for embedded applications (where distributed processing, a power point of Erlang, is NOT required) and just build things on top a generic messaging framework like zeromq?
2.Is Erlang offering more than a coordinated messaging framework for a non-distributed embedded application?
3.What specific capabilities of Erlang could took too long to implement with zeromq?
You're comparing apples and oranges. Part of the advantage of using Erlang is the language; if you're going to put it up against zmq + some other language, the other language in that comparison really matters. zmq + ARM assembly? Erlang brings all the wonderful advantages of not hand-coding ASM.
As for what else Erlang brings to the table, Embedded Erlang? Absolutely argues that Erlang has advantages in fault tolerance, hot code loading, rapid development by leveraging Erlang and OTP, easy interaction with C libraries, and simple debugging by live REPL and copy-paste of terms.
Some of those things, such as hot reload, on-device REPL, and established libraries, will definitely take some real hacking to reproduce from the ground up.
My point would be that you will have to work very hard to get the same kind of error handling in Zmq. Erlang has some really nice built-in error handling when things begins to go bad. There has been considerable time spent in Erlang optimizing that part and making it robust.
Zmq on the other hand, is probably faster in some combinations with some languages when you make simple benchmarks. There is less overhead, so it may process messages faster than what Erlang can provide.
But chances are that you will end up re-implementing large parts of Erlang in the language of your choice. And you will probably not do a job as good as 6-10 developers working on Erlang/OTP for 15 years.
On the other hand, Erlang is not a simple language to learn. There is way more to it than just learning how to program in a functional style. Especially the concurrency patterns and failure handling can take some time getting used to.
ZeroMQ =/= Erlang covers many differences. The claim there is that ZeroMQ only provides the messaging aspect, not the light-weight processes, process monitoring and other aspects.
I think Erlang is very well suited for server systems developed in my workplace (currently developed in Java). I am a bit skeptical how this would be accepted both by developers (who have no idea about functional or Erlang) and by managers.
Any ideas on how to approach the issue? I am thinking about some hybrid system, where the hardcore highly reliable infra uses Elrang, and app specific stuff developed in Java (as nodes?)
There are a few approaches, and neither have any guarantees to actually work
Implement something substantial in a short time frame, perhaps using your own time. Don't tell anyone until you have something to display that works. Unless you have a colleague in on it.
Pull up lots of Erlang projects that are good demonstrations of the features you want. Present it to your managers and try to frame them about the risk in keeping using Java with this kind of technology available.
If the company you work for actually have a working code base in Java already, they're not likely to take you seriously when you suggest to rewrite it in another language.
The true test that you believe in Erlang being a much better choice: Quit and start up a competing company and bring the technology insight you have in your current industry. Your managers are really comparing a similar risk-scenario as you would do if you were to quit your job, and they are looking for the same assuring facts for success as you would do, to consider leaving a "safe" paycheck.
As for how to integrate, check out the jinterface application in Erlang. It allows Java code to send messages to Erlang nodes, and it allows Java to expose mailboxes to the Erlang nodes as if there were Erlang processes.
It's all about ROI (Return On Investment) to a manager: a manager will be concerned about performance (of the company). In order to appeal to his business nature, you'll have to make a case for it using dollar$ (or whatever appropriate currency).
Beware that undertaking a "skunkwork" project on the side to "prove" your solution based on Erlang might backfire: "so you had time to play with Erlang, why didn't you spend the time on the project then?" (Of course, not all managers/companies would think this way).
You have to take into account the whole proposal e.g. impact on the team, skills to be developed etc. It's all about money.
If I have an advice for you: start small, plant a seed, nurture it and watch it grow.
A wise man once said to me:
"It's not about technology, it's about
the product & market".
Start by not targetting a rewrite but using erlang for a new feature/project. Rewrites can be expensive and taking a chance on erlang for something that is already a time consuming and costly undertaking is a hard sell. But if there is a new piece that could be done in erlang and java, you stand a better chance. The project will be small enough hopefully that you can discover early if erlang is a good fit and adapt accordingly. And when erlang proves itself in that project you will have better data to make your case with.
We're introducing RabbitMQ into our infrastructure, which currently runs a combination of C++, Java and Python applications. I'm not specifically intending to move the team towards Erlang, but if I were, introducing a well-written third-party tool that just happens to use Erlang is a very good way to get the foot in the door.
One major caveat is that while Erlang is a wonderful language to learn, the surrounding technology (OTP in particular) has a huge learning curve and is extremely primitive in many ways (debugging, IDE's, etc.). It is getting better all the time, but reluctant converts will crucify you if you don't warn them about the pain of learning to program in a radically different environment. Even simple things like the lack of code-sense technology (E.g., type 'foo.' and the IDE tells you what methods you can call on foo) can leave a really bad taste in the mouth.