This question is more of getting information about different HTTP Libraries in elixir.
I have been using HTTPoison for a long time in my all elixir projects. which is a wrapper around Hackney. I read some stuff on http://big-elephants.com/2019-05/gun/ about hackney as
Occasionally hackney could get stuck, so all the calls to HTTPoison
would be hanging and blocking caller processes.
Let me ask with an example, I was using Poison in my project and I considered it to be a best JSON Parser, then I saw Jason.
The primary reason is performance. Jason is about twice as fast and
uses half the memory while being almost 100% functionally compatible
with Poison (it covers all the features used directly in Phoenix).
I know the reason why I want to switch to Jason instead of Poison.
In HTTP, I can see there are Tesla, Mint and Gun. I just want to know the perspective of switching from something Famous to I may say Accurate one?
Related
I just saw this 2012 video from LinuxConf.au about Erlang in production.
There's a part on the video where Bernard says no big Erlang projects use Hot Code Loading apart from Ericsson, because it's really hard to guarantee things will work. It's around minute 29.
Is that still true? Are there tools to help test a hot code load or or make it easier nowadays?
This is not true. Every Erlang user uses hot code loading to his advantage in one way or another -- whether it is for development, testing, troubleshooting, one-off fixes, or full scale deployments. This is one of the major Erlang advantages. Rather unique too.
For example, WhatsApp, one of the biggest Erlang users, relies on hot code loading for almost all code pushes.
I have personally worked with hot code loading in scenarios where each change was well understood and often performed by the same person who made the change. It works extremely well and good engineers don't have any problems doing this. Speaking of tools, loading modules one by one from Erlang shell using l(...). or all at once using l(). (see here) works just fine. Some prefer release-based tools like relx.
Others, like Ericsson, use enterprise-style deployments with hot code loading after rigorous testing of clear-cut releases and patches. The goal here is to upgrade without using spare capacity and special procedures for draining and shifting load. Operationally this may be simpler and more efficient than restarts, but testing can be more expensive.
It is difficult to know whether it is a feature widely or scarcely used. Nowadays there are plenty of Erlang systems out there. I can however think of reasons why and why not to use it, since I have been working with bot options for quite some time.
In favour of using it:
It is obviously quite useful during development to ensure a fast feedback cycle. I always develop with an open shell, and with functions to load code automatically as son as compile.
In the rare case you need to implement a monolithic application with high availability requirements, it is basically the only option
The main reason not to use it, as the presentation states: it is hard. Even if you manage to understand exactly how it works (which is not the hardest part).
It is not, in my opinion just a problem of tooling, but rather that you are getting a lot of intrinsic complications just by the fact that now your code is part of the mutable running state of the system. You basically end up having a long running system that changes behaviour, so you add those to the problems you already had:
You are no longer sure that restarting the system will not change behaviour on any fundamental way. So you will probably need to put extra care on making sure that whatever code you load, it is also written to disk.
Changing the way your modules work (i.e. loading new code) is very tricky unless a) you never break compatibility, b) you somehow figure out the order in which the modules should change or c) you assume the worst that can happen is a few crashes due to undefined functions, function or case clauses, etc, and hope for the best (the actual worst is when the new and old modules interact in unexpected ways while you haven't finished loading all of the new ones and the actually run some impossible logic).
You will almost certainly will end up killing some process running old code when loading new code at some point. Maybe your supervisors will help you, maybe not. In any case that could be very confusing and difficult to debug.
As the presentation also states, is very hard to test (if not impossible).
Etc.
Adding to all those, you are running a long living server with long living state, which is far from ideal.
So my advise is always that, if you could get away with a distributed application and rolling upgrades, you should do it. That option is much easier to handle, and in my experience, performs better overall.
I'm going to develop a collaborative site, and one of the features will be collaborative editing with realtime changes. i.e. when two or more users are editing the same doc, they can see each other changes as soon as they happen.
I have some experience with Ruby on Rails, so I was thinking about using EventMachine, but with all this hype around Node.js, I am know considering using it instead. So, what would be the main benefits of using Node.js instead of EventMachine?
tl;dr
What are the main differences between EventMachine and Node.js (besides the language)?
EventMachine has nothing to do with Rails apart from them both being written in the same language. You can get EventMachine as bare as Node.js; all you have to do is not add libraries to your project. In my experience the EventMachine libraries (like em-http) are much nicer than anything for Node. And you can use fibers instead of callbacks to avoid callback hell. Complete exception handling is pretty much impossible in Node because of all the callbacks. Plus Ruby is a nicer, more complete language than Javascript.
I tend towards the "use what you know" (even if it's a heavier architecture). Because of that, I don't see it being quite as simple as "EventMachine vs NodeJS." Mainly, the difference can be summarized as this:
NodeJS is a framework/language that was written to handle event based programming in JavaScript. That is its driving force. It's not an after thought, or a third party mechanism. It's baked right in to the language. You create callbacks/events because that's how the language is built. It's not a third party plug in, and doesn't alter your workflow.
EventMachine is a gem in Ruby that gives developers access to some of the goodness of the event based programming model. It's heavily used and well tested, but not baked directly in to the language. Both are locked to one CPU, but with event programming at Nodes core, it still has a leg up. Ruby wasn't written with concurrency in mind.
That said, technical problems can be overcome. The more important questions (from my view) that should guide your decision are these:
What will your production environment look like? Do you have complete control over the server? Can you host it however you want? Or will it be on a shared system to start with, and then you have to expand on that?
Do all the developers on your team have the ability to learn a new language very fast? How fast will they be able to understand an event-based language like JavaScript for the middle tier?
Do you need all of the architecture that Rails gives you (full Testing framework, scaffolding, models, controllers, etc)? Or is that overkill?
There are quite a few technical differences between the two. One is a language, one is a framework. Really, how heavy of a stack you want to run? How much learning will your developers have to do? Do you want a full stack the gives you a lot of niceties, that you may not use, or do you want a bare bones set up that runs extremely fast and concurrent, even though you may have to write extra boiler plate code and learn a new lanugage?
While Rails is not as heavy as some web application architectures, you're still going to need more processor power than you would to handle a similar amount of throughput in NodeJS. Assuming quality code for both systems. Bad code written on either stack is going to prevent the stack from shining. It really comes down to- Do you really want to learn a whole new way of doing things, or utilize your current understanding of Ruby to get things off the ground fast?
I know it's not really a definitive answer, but I hope this helps guide you to a decision!
One thing worth mentioning is the production story. EM, like most Rack stuff, has plenty of testing and monitoring tools available that are well tested, whereas Node.js falls well short in this respect.
At the time of writing, it seems almost impossible to get clear metrics from Node to answer questions like 'Do I need to scale'. There are options starting to form out there from the likes of Joyent, and always the roll-your-own argument, but nothing anywhere near tools such as NewRelic.
Node.js is very good from a performance / configurability point of view, but personally I wouldn't host it in production just yet.
Node.js
You get far better control low level control over what's going in. You can include general libraries to build on top of node.js to tweak your level of abstraction to your own liking. For example you can use connect or express depending on whether you want a view engine written for you.
You can use socket.io or now depending on how much you want your client-server connection abstracted. You can opt to include any of numerous MVC libraries or write your own.
Event-Machine
An asynchronous IO library just like node.js
It comes down to a Ruby vs JavaScript preference, how much flexibility you want with abstractions or lack of abstractions and whether you want to use node as your actual web server.
a detailed view at confusion has already been proposed... just a personal view
[] node.js will be better, if you are ready to learn and experiment more than you think because:
it's thread mechanism is awesome (inspired from that of 'erlang')
you can build a purpose specific server (easily) which will be real productive
I've been checking out the Nitrogen Project which is supposed to be the most mature web development framework for Erlang.
Erlang, as a language, is extremely impressive. However, with regards to Nitrogen, what I am not too keen about is using Erlang's rather uncommon syntax (unless you're native in PROLOG) to build UIs.
What is your experience with it as opposed to other mainstream web frameworks such as Django or Rails?
I've done very little with Nitrogen so far, but I've been monitoring the mailing list for months, so I think I have something useful to say about it.
To your concern about the syntax of Erlang and the Nitrogen framework, I'd respond that that sounds like a pure case of unfamiliarity, rather than unsuitability. Objectively, HTML is not a beautiful language, and it has plenty of quirks. You're used to this now, so it doesn't seem so bad. Give Nitrogen/Erlang a chance and you may find that you get used to it soon enough, too.
To your question about comparison to other languages and frameworks, I'd say the biggest difference is that with Nitrogen, the entire web site is being served directly by the Erlang runtime. Ruby on Rails has such a mode, but it's intended only for testing. Many other frameworks don't even offer the option of running everything within a single long-running process.
Running the entire web application and its underlying infrastructure within a single long-running process has significant implications on how the site runs:
With Apache, each child gets killed off every N connections, where N=500 or so, and you can't say whether a given child will always handle all of a given client's requests. Because HTTP is stateless but web apps almost always require some client state, an Apache child must rebuild its view of client state as part of handling a new connection. By default, this means going back to disk for persistent data stored about that client. There are alternatives like memcached, but these aren't built into the core of a LAMP type stack. With Erlang, nothing is killed off periodically, and Erlang offers standard facilities like Mnesia which provide disk-backed in-memory DBs.
Incidentally, if you're familiar with nginx, it's built on the same principles as Erlang, and it's fast for the same reason. The main difference between nginx and an Erlang instance running a web server is that nginx isn't a programming environment, so it still has to delegate a lot of processing to outside code. That means it shares the same IPC and persistent state problems as Apache.
Because the runtime stays up continuously and is a fully-functional programming environment, you can probably build more parts of your system in Erlang than with a lashed-together LAMP type stack. This magnifies the above benefits. The various parts of your system can coordinate via message passing and Mnesia instead of heavyweight IPC and MySQL, and all the pieces stay up and running continually, leading to less time-consuming state reconstruction.
A dozen or so Apache children all accessing the persistent client state data store is a lock-based hairball. The frameworks all handle locking and such for you transparently, but what they can't hide is the time it takes to do all this correctly.
Erlang is an impure functional language, which implies but does not require data purity; it is also built with multiprocessing in mind, going clear down to the core of the runtime design. These two facts mean you're less likely to spend time waiting on locks in an Erlang based server than one naively built on one of the other frameworks. It is certainly possible to optimize away lock delays in the other systems, but is that really what you want to be doing? Do you want to be on the thousandth team that has to learn how to optimize its web stack after the service becomes popular, or would you rather leave it all up to the tooling so you can spend your time doing something no one else has done yet?
I, too, was once concerned about clunky Erlang syntax. I've built a couple of tools to alleviate its annoyances for everyday web programming, and perhaps you will find one or both of them helpful:
ErlyDTL is an Erlang implementation of the Django Template Language; it's not available in Nitrogen, but it is available in other frameworks, such as Zotonic, Erlang Web, BeepBeep, and Chicago Boss
Chicago Boss is a full-stack Erlang framework that does a lot of code generation so that you can access data fields with function calls instead of Erlang's rather verbose record syntax (e.g. Person:name() instead of Person#person.name)
Note that Nitrogen does not include a database layer, so it's not really comparable to Rails or Django. For a comprehensive comparison of the database-driven frameworks, check out my answer to this StackOverflow question:
https://stackoverflow.com/questions/1822518/current-state-of-erlang-web-development-frameworks-template-languages/2898271#2898271
I would check out Webmachine if I were you. It is quite simple, fast, and leaves the interface up to you.
Erlang Web should also be considered mature. It is an MVC framework, whereas Nitrogen is more event based. It's a matter of preference.
I haven't used the other tools mentioned here except Webmachine, which I think it's a wonderful tool, but it is not a web framework like the others. It is as HTTP processor, and is ideal for building a restful interfaces.
I would also suggest you give the Erlang syntax a chance. Erlang is one of my favourite languages to use.
I have a Rails webapp [deployed on Heroku] which makes a series of HTTP calls to other sites on a repeated basis, using Heroku's rake:cron feature. The current situation isn't ideal; the rake:cron process is executed in a single thread, which means HTTP calls are made sequentially; which means in turn that there's a long time between calls to the same site [typically 2 mins].
I'd like to execute this process in parallel, and reduce the time between calls to 10 secs. Having seen Kevin Smith's 'Erlang in Practice' I'm sold on the idea of using Erlang as a replacement backend. What I'm trying to figure out [given Damien Katz's comments], is whether I should a) re-write the entire webapp in Erlang, front end and all or b) maintain a split structure, with a Rails frontend / Erlang backend.
I like the idea of using a 100% Erlang stack for the project; I'll need to use some kind of Erlang web framework [Nitrogen ? Erlyweb ?]; I'm concerned they're not mature enough and I'll spend my time bogged down on the web part of the project with them.
Anyone any views ? Thanks.
What's the actual impact on your visitors (of the two-minute interval between HTTP backend calls)?
If there isn't much of a difference, I'd say this sounds like premature optimization and that you'd be much better off skipping Erlang for now.
The two previous posters have pretty much covered they philosophical aspects of your question. So I'll answer the framework maturity/getting bogged down part of your question.
In the event that you decide you do want to rewrite the webapp in erlang for whatever reason then I wouldn't be too concerned about the framework slowing you down. Both erlyweb and nitrogen are already feature complete enough that you can work pretty quickly with them. I've developed a fairly complex agile project management app in nitrogen and found it to be quite intuitive and not really lacking in features that I needed. A few hours in the evenings and a few weeks later and I had a working app up and running.
As to which one to use that depends on the type of app you want to build.
Nitrogen's target is extremely dyamic web applications. Most of the page is rendered using javascript and it is highly event driven.
ErlyWeb is more suited to a site where the content is the primary focus less so than a rich client type of application. It uses the MVC style of architecure.
Good luck on whatever you decide.
It depends. How much Erlang do you know? How much code have you already written?
How much project experience do you have? Is this for work or for fun?
Rewriting projects from scratch is often a recipe for disaster, especially if you are trying
to learn a new language along the way. It seems to me like you would not be asking this question if you were already fluent in both languages, in which case I would recommend that you just stick to Ruby if it's a work project.
I disagree with the above poster that changing the language is a premature optimization, if it is necessary.
Changing the language is a big deal. It can't be done at the last minute.
However, I would probably not change the language at all for the reason you outlined.
If you don't have any other reasons than performance for switching, you should probably just
look at multi-threading in Ruby or some other optimization.
I'm all about using the right tool for the job. Unless you have an absolutely dead on reason to port the front, there's absolutely nothing wrong with hooking the two together.
I am considering Erlang as a potential for my upcoming project. I need a "Highly scalable, highly reliable" (duh, what project doesn't?) web server to accept HTTP requests, but not really serve up HTML. We have thousands of distributed clients (other systems, not users) that will be submitting binary data to central cluster of servers for offline processing. Responses would be very short, success, fail, error code, minimal data. We want to use HTTP since it is our best chance of traversing firewalls.
Given this limited information about the project, can you provide any weaknesses that might pop up using a technology like Erlang? For instance, I understand Erlang's text processing capabilities might leave something to be desired.
You comments are appreciated.
Thanks.
This sounds like a perfect candidate for a language like Erlang. The scaling properties of the language are very good, but if you're worried about the data processing abilities, you shouldn't be. It's a very powerful language, with many libraries available for developers. It's an old language, and it's been heavily used/tested in the past, so everything you want to do has probably already been done to some degree.
Make sure you use erlang version R11B5 or newer! Earlier versions of erlang did not provide the ability to timeout tcp sends. This results in stalled or malicious clients being able to execute a DoS attack on your application by refusing to recv data you send them, thus locking up the sending process.
See issue OTP-6684 from R11B5's release notes.
With Erlang the scalability and reliability is there but from your project definition you don't outline what type of text processing you will need.
I think Erlang's main limitation might be finding experienced developers in your area. Do some research on the availability of Erlang architects and coders.
If you are going to teach yourself or have your developers learn it on the job keep in mind that it is a very different way of coding and that while the core documentation is good a lot of people do wish there were more examples. Of course the very active community easily makes up for that.
I understand Erlang's text processing
capabilities might leave something to
be desired.
The starling project already provides basic unicode support and there is a EEP (Erlang Enhancement Proposal) currently in draft, but going in to bring it into the mainstream of Erlang/OTP support.
I encountered some problems with Redis read performance from Erlang. Here is my question. I tend to think the reason is Erlang-written module, which has troubles while processing tons of strings during communication with Redis.