I'm starting to dive into Erlang for the first time, and OTP is held aloft by lovers and critics alike as being the gold standard for highly available, distributed processing.
Given that OTP has been around for decades and is openly documented, why is it that other languages supporting lightweight threads/processes haven't adopted versions of their own? Are there technical/political challenges? Or does everyone just shrug and learn Erlang?
Thanks!
The largest issue is that most language runtimes don't have built-in lightweight concurrency and error isolation with exit signal propagation. Without those things you would have a really hard time properly porting OTP.
For the languages that do have the right kind of runtime, I am seeing some effort or at least plans to build OTP inspired frameworks. Cloud Haskell is the first that comes to mind. I also expect that Go and Rust will eventually have something like OTP if they don't already.
There are technical challenges, as Erlang itself is designed for the same features OTP is known for. Case in point, Basho Riak is a distributed fault-tolerant key/value store written in Erlang. One might be able to port it to Haskell or some similar functional language, but it would probably be a lot of work. Just for fun, you might look into OTP stuff written in the Elixir language.
Actually, it has been (tried).
Akka is the library which takes some OTP features and implements them in Scala for JVM.
Given the principles underlying JVM and BEAM (the Erlang VM) are very different (mainly GC, scheduling and message passing are radically different), I can't say how successful that implementation is and how many benefits of the original OTP it preserves. There's a lot of (heated) debate on that in the internets.
Related
We are hoping to develop a static code analyser for Elixir in order to detect concurrency issues (mainly deadlock and race conditions). We have some basic idea about the structure for the analyser, but our problem is which AST would be better suited for this task. As we have understood the Elixir compilation process creates an Expanded Elixir AST, Abstract Erlang Format and Core Erlang.
My question is out of these whether the Expanded Elixir AST or Core Erlang would be better for creating the Call Graph and Control flow graph. And if we use Core Erlang is it possible to work our way backwards from Core Erlang to find the source within the Elixir code for the issues identified by the analyser?
If anyone has some idea about this, your help would be really appreciated. :)
If the purpose is educational I would probably go with erlang. The compiling of erlang to beam will probably be a little more straightforward (if not a lot more) and as you develop your tool, you will probably find more resources/documentation on erlang ASTs. It's lower level than what most people do and you'll find more answers in the erlang community (probably won't be true in a few years). Overall, your tool would be simpler in erlang, with less moving parts.
I also found that project : https://github.com/rrrene/credo
More specifically, I think it will be practically difficult to work you way back to elixir code. From your perspective, after having implemented your tool, it will make sense. But that probably would not be the case for your average elixir developer. The more elixir matures, the more it tends to depart from core erlang concepts. It builds a lot of features on top of erlang, to the point where you can build entire web applications without knowing a single bit of erlang. Probably not the best approach but the fact that it's possible tells you how large the gap between the two languages can become.
This question is in the context of the Beam VM and the capabilities that it provides, not in the general context of what a Turing complete language can do. I want to invest some time to learn either pure Erlang or Elixir. I get the basic differences between the two and I am leaning towards Elixir because of the macros, better syntax and faster development of the language this day.
My question is: if I choose Elixir, will I stumble on something that I cannot do in it, but can do in Erlang? Can I use all the OTP stuff, all the Erlang libraries, code reload, etc. with Elixir?
I am not asking for someone's preference between the two; just facts about the possibilities of the languages. Preferably from someone who used both in production.
You shouldn't stumble on anything you can do in one that you can't in the other, since you can freely call Elixir code from Erlang and vice-versa. You can even easily mix Erlang and Elixir files in one project.
In Elixir:
:erlang_module.erlang_function(args)
in Erlang:
'Elixir.ElixirModule':elixir_function(args)
TL;DR - Start with Elixir
Erlang has a steeper learning curve compared to Elixir. Once you start learning Elixir, you will automagically start learning Erlang. Hence, start out with Elixir. Elixir is written in Erlang and Elixir. See distribution on Github (since Elixir is full of macros aka meta-programming).
You can use Elixir with Erlang and vice-versa, hence the full Erlang eco-system of 20+ years of libraries.
More details from Erlang Solutions
Elixir’s ‘out of the box’ productivity is accomplished by a strong focus on tooling and convenience in expression of data manipulation. System design is the same in Elixir and Erlang, but Elixir removes a lot of boilerplate code and is easier to extend. The removal of boilerplate raises productivity and allows programmers to get feedback faster – crucial when you want to launch your product to market as fast as possible. Less boilerplate also makes for happy developers, and happy developers are unsuprisingly productive developers.
Joe Armstrong's (Erlang inventor's) blog post about Elixir
Start here to learn about Elixir - Getting Started
Once you feel its going well, work your way towards practicing on Exercism and other resources.
Just to preface - I have only used Elixir in production and not Erlang.
I would honestly recommend Elixir. This is my opinion and not necessarily the right one for you, but below I will list my reasons why.
Productivity: I come from a Ruby/Rails background so the Elixir syntax and style is something that was very familiar to me. One of the main factors that helps me determine whether or not to learn a language is how productive I can be in it - mainly why I chose Ruby. Elixir is the same. I can work just as fast in it as I can Ruby with all the added benefits of concurrency and pattern matching.
Erlang: Since Elixir is built on top of Erlang and compiles down to erlang and the beam vm, you have access to every erlang module and package. So if you are worried about using elixir and missing out on all of the features of Erlang, you shouldn't be. Elixir even has it's own implementations of most of the bigger Erlang/OTP features such as GenServer, GenEvent etc.
Community/Resources: The Elixir community is really a great one. The slack channel is really popular and great way to get some answers for beginner questions. There are already some really good books written on the language (Programming Elixir 1.2 - Dave Thomas, Author of the Ruby Pickaxe) also that make getting into the language really easy.
Really, if you have a mess around with the two you will probably come to the same decision that Elixir is a much nicer language with all the features of Erlang + More. It's on the rise as well, I can't remember the exact numbers but I remember reading something from the Hex website (package manager) about a considerable increase in package downloads.
There are a few things. I think you can't make a recursive anon function in Elixir. Now to be fair this is something that in 8+ years of Erlang I don't think I have ever needed to do, and if I did it could be done easily in some other way. There are probably a few other things like that that quite honestly you can ignore.
In general, for things that most people care about anything you can Do in Erlang, you can do in Elixir and vice versa. The community on the Elixir side seems to be more active so I would suggest starting there. I have recently moved from Erlang to Elixir and with the exception of a few fancy things in Proper I can say that it is a pretty easy transition
I want to write actor style code for embedded processors and I am trying to decide between writing everything in Erlang vs writing everything in zeromq+any language. Using zeromq looks to be very powerful in the sense that I can use any programming language and make my development a lot easier(many available libs) but then I am not sure if there is any gotcha in this power? I understand that Erlang represent actor model much better especially with OTP concepts but then it seems easy to represent similar actor model with zeromq? Am I looking at this correctly?
1.What do I really lose not using Erlang for embedded applications (where distributed processing, a power point of Erlang, is NOT required) and just build things on top a generic messaging framework like zeromq?
2.Is Erlang offering more than a coordinated messaging framework for a non-distributed embedded application?
3.What specific capabilities of Erlang could took too long to implement with zeromq?
You're comparing apples and oranges. Part of the advantage of using Erlang is the language; if you're going to put it up against zmq + some other language, the other language in that comparison really matters. zmq + ARM assembly? Erlang brings all the wonderful advantages of not hand-coding ASM.
As for what else Erlang brings to the table, Embedded Erlang? Absolutely argues that Erlang has advantages in fault tolerance, hot code loading, rapid development by leveraging Erlang and OTP, easy interaction with C libraries, and simple debugging by live REPL and copy-paste of terms.
Some of those things, such as hot reload, on-device REPL, and established libraries, will definitely take some real hacking to reproduce from the ground up.
My point would be that you will have to work very hard to get the same kind of error handling in Zmq. Erlang has some really nice built-in error handling when things begins to go bad. There has been considerable time spent in Erlang optimizing that part and making it robust.
Zmq on the other hand, is probably faster in some combinations with some languages when you make simple benchmarks. There is less overhead, so it may process messages faster than what Erlang can provide.
But chances are that you will end up re-implementing large parts of Erlang in the language of your choice. And you will probably not do a job as good as 6-10 developers working on Erlang/OTP for 15 years.
On the other hand, Erlang is not a simple language to learn. There is way more to it than just learning how to program in a functional style. Especially the concurrency patterns and failure handling can take some time getting used to.
ZeroMQ =/= Erlang covers many differences. The claim there is that ZeroMQ only provides the messaging aspect, not the light-weight processes, process monitoring and other aspects.
I am about to build a system with Erlang (my problem can be solved better with multiple CPUs) and I have skimmed through:
Learn You Some Erlang;
Erlang/OTP in Action
Programming Erlang (Armstrong);
Erlang Programming (Cesarini).
The Erlang/OTP in Action book (2) says: "...writing non-OTP Erlang code is really an advanced topic, and something that should be done only when you really really have to. So perhaps not ever having done it the non-OTP way is a blessing because you will pick up the right habits straight away with regard to things OTP..."
(2) insists that commercial systems should be built with OTP only, whilst (4) gently shows that OTP is not the only way to do this.
Additionally, by reading (4) I have found out that building the knowledge from the very basics of Erlang helps to understand how OTP works (but not in the opposite way).
So, the question is:
Should I choose Erlang/OTP to build a commercial system or can it be done with pure Erlang?
Learn and play with pure Erlang.
Build systems in Erlang/OTP.
can it be done with pure Erlang?
Yes, it can, subject to the analogue of Greenspun's Tenth Rule: your system will contain an ad hoc, informally-specified, bug-ridden, slow implementation of half of OTP.
OTP is written in Erlang so it definitely can be done in pure Erlang. That being said OTP provides you with a lot of useful functionality which you would otherwise need to write yourself. #Alexey is right than you will end up implementing a portion of OTP along the way. The problem is that the framework you need for building fault-tolerant systems, which OTP provides, has to itself be very fault-free.
I will also echo #Mazen in that experimenting with doing it in pure Erlang gives you a good feel for what is going on internally in OTP and how you can build fault-tolerant systems in Erlang. You will make better use of OTP afterwards. You will come to a better understanding when not to use OTP, and why.
If you read through the "Learn You Some Erlang" (like you said you have) you'll notice the author spends the Designing a Concurrent Application chapter building something half-useful in "plain Erlang". Then in the next 2 chapters, he basically throws it all away, and uses OTP to achieve the same thing with far less code (add to that the fact that because it's OTP code, it's standard - so other developers with familiarity with OTP will have a much better idea of where to look to understand it!)
The OTP has been developed and refined over a lot of years (initial release 1998 according to Wikipedia) with a large number of developers, and the backing of a large international company (Ericsson), and peer reviewed as it's all open source. I think it'd be a little audacious to assume you could single-handledly build something equally robust and scalable!
So use the OTP. Even for "play" projects, still use the OTP as it can only help you learn to make your "commercial" code better :)
I think Erlang is very well suited for server systems developed in my workplace (currently developed in Java). I am a bit skeptical how this would be accepted both by developers (who have no idea about functional or Erlang) and by managers.
Any ideas on how to approach the issue? I am thinking about some hybrid system, where the hardcore highly reliable infra uses Elrang, and app specific stuff developed in Java (as nodes?)
There are a few approaches, and neither have any guarantees to actually work
Implement something substantial in a short time frame, perhaps using your own time. Don't tell anyone until you have something to display that works. Unless you have a colleague in on it.
Pull up lots of Erlang projects that are good demonstrations of the features you want. Present it to your managers and try to frame them about the risk in keeping using Java with this kind of technology available.
If the company you work for actually have a working code base in Java already, they're not likely to take you seriously when you suggest to rewrite it in another language.
The true test that you believe in Erlang being a much better choice: Quit and start up a competing company and bring the technology insight you have in your current industry. Your managers are really comparing a similar risk-scenario as you would do if you were to quit your job, and they are looking for the same assuring facts for success as you would do, to consider leaving a "safe" paycheck.
As for how to integrate, check out the jinterface application in Erlang. It allows Java code to send messages to Erlang nodes, and it allows Java to expose mailboxes to the Erlang nodes as if there were Erlang processes.
It's all about ROI (Return On Investment) to a manager: a manager will be concerned about performance (of the company). In order to appeal to his business nature, you'll have to make a case for it using dollar$ (or whatever appropriate currency).
Beware that undertaking a "skunkwork" project on the side to "prove" your solution based on Erlang might backfire: "so you had time to play with Erlang, why didn't you spend the time on the project then?" (Of course, not all managers/companies would think this way).
You have to take into account the whole proposal e.g. impact on the team, skills to be developed etc. It's all about money.
If I have an advice for you: start small, plant a seed, nurture it and watch it grow.
A wise man once said to me:
"It's not about technology, it's about
the product & market".
Start by not targetting a rewrite but using erlang for a new feature/project. Rewrites can be expensive and taking a chance on erlang for something that is already a time consuming and costly undertaking is a hard sell. But if there is a new piece that could be done in erlang and java, you stand a better chance. The project will be small enough hopefully that you can discover early if erlang is a good fit and adapt accordingly. And when erlang proves itself in that project you will have better data to make your case with.
We're introducing RabbitMQ into our infrastructure, which currently runs a combination of C++, Java and Python applications. I'm not specifically intending to move the team towards Erlang, but if I were, introducing a well-written third-party tool that just happens to use Erlang is a very good way to get the foot in the door.
One major caveat is that while Erlang is a wonderful language to learn, the surrounding technology (OTP in particular) has a huge learning curve and is extremely primitive in many ways (debugging, IDE's, etc.). It is getting better all the time, but reluctant converts will crucify you if you don't warn them about the pain of learning to program in a radically different environment. Even simple things like the lack of code-sense technology (E.g., type 'foo.' and the IDE tells you what methods you can call on foo) can leave a really bad taste in the mouth.