First, brief but profound thanks to all who have responded to my questions to date.
I'm climbing the Erlang OTP learning curve and enjoying every step. Now digging into Application design and integration. My question today is this:
Suppose I have a set of library modules: myLibrary. I want to employ these functions in an Erlang Application involving a set of application-specific modules: myApp. Mylibrary may well serve Applications beyond myApp.
I want, in turn, to deploy myApp over the web, over, on Webmachine, say. I also want need data persistence for myApp, could be Dets or Mnesia or Riak -- yet to be decided.
First, should I wrap myLibrary into the same Application as myApp? Or should they be separate Applications?
Second, how can I best integrate all these Apps into one functional Release? (I'm burning oil on the published docs re: this question, but finding them a bit confusing).
Thank you,
LRP
First of all, I'm thrilled that you are learning erlang and getting your questions answered by the community. I would like to suggest, however, that you reciprocate the good will by marking correct answers that we have provided as "accepted". Now that we have that out of the way, on to the actual answer...
myLibrary should be it its own repository if you believe it has potential for reuse. See edate as an example of such a library.
I agree that the business of packaging applications and their dependencies in erlang is, at the moment, a bit confusing. You may want to start with my answers here and here.
This set of three posts walks you through the process of integrating the holy trinity of webmachine, erlydtl, and riak. The material is a bit dated but as I recall the author touches on a number of points pertinent to your question.
This post is often given as a reference for using rebar and creating actual releases.
As of this writing I don't actually do real erlang releases in production. I use rebar to compile, test, and manage my dependencies and that's it - on my production servers I just clone my repository, run make, and away I go. But if you want the full-on release management experience, rebar can help you do that too.
Related
I am trying to ilustrate the concept of distributed applications using Erlang. My system currently has one server and one ATM. I try to keep it as simple as possible.
For the moment my application runs locally. I am using gen_server for a client-server relationship between the banking server and the ATM. I also have a gen_fsm module to suggest different states my ATM has. In order to store any data i use the dict module (I don't want to make things more complicated using databases). To keep processes alive (the gen_server and gen_fsm) I am using a supervisor process. I've wrapped all modules as an application but for the moment it's all local. Any ideas would be highly appreciated.
I was thinking to start the same application on two different nodes and to illustrate the distributed concept to use some kind of failover/takeover mechanism but I have no ideas on what modules to use.
Is it mandatory to use target systems?(at some point i must do a hot upgrade to the application)
What's the correct order to do these things: first to upgrade and then distribute?
I would be very grateful if someone could give me some ideas on how to accomplish all those things.
I never tried it myself, but docs seem to point to:
Erlang Release Handling (11.3 Distributed Systems).
It's a really short paragraph showing the sync_nodes command, I suggest you to read the whole chapter because I noticed that often the concepts of concurrency and distribution are so persavive in Erlang that problems like yours have already been solved and included in OTP.
BTW, Erlang user guide also has a whole chapter dedicated to Distributed Applications that seems related to distributed applications config options, I think that the two should do the trick.
Hope this helps, if you need more help just ask!
I have some community running Drupal 6. Drupal 6 have great numbers of modules for what I need, plus word "Community plumbing".
I've tried PHP frameworks like Yii, but I dont like PHP-CLI, because I feel that not well integrated with the PHP itself. So, I stay with Drupal 6 with lots of hacks.
But Drupal now become more "CMS", more "Enterprise" (just like lots of Java CMS).
With Drupal 7, out of the box, it easier to make a newspaper site, but to me, it more difficult to tailored to my needs, it more magic, and also slower.
The word "Community plumbing" has been replaced with "Open Source CMS", with statement "Use Drupal to build everything from personal blogs to enterprise applications". This is scares me as community-based application developer.
Django looks promising, but Rails have much more open source kick-ass app avaiable.
I dont know anything about Drupal 8. I dont know what to do for my next development.
It sounds like you are looking to move from a CMS to a framework. I am not going to give advise on frameworks here, since there are numerous topics on SO that cover this already.
A few things you should realize, coming from Drupal, moving to a (RAD)framework are the following:
Drupal is not OO, has its own ways of using PHP (hooks, template overrides) and such. Expect the experience to be very different. Your Drupal-gained-knowledge may come in handy, but is often of little use in framework-land where OO is the alpha and omega. :)
Drupal takes a lot from your hands. By installing some module, you have a new feature. With frameworks you are all on your own in this. You should select a framework that works with gems, packages or modules itself if you want to stand on shoulders of others peoples work. The main difference between these libraries and Drupal modules, is that you will have to implement and integrate the libs yourself. I consider that a very good thing, since it makes no assumptions up front and allows you to build exactly what you had in mind, yet having the same experience as Drupal-modules: many things are already done for you.
You know PHP. You probably don't know Python or Ruby as well. That means PHP-based frameworks have an advantage for you. But read various posts on SO about the downsides of PHP used in Frameworks to see some reasons why learning a new language and environemnt is not all that bad.
All in all, I think you will be surprised byt the increase in development-efficiency when coming from Drupal into a framework.
Make your own CMS in 'just PHP' and go back to being happy and having fun coding all day long :)
More people than you have noticed this behavior of leaving developers that have supported a system for years and years with a take it or leave it option.
It seems all the big companies can do with a product is make it 'bigger'. More convoluted, automated, with more layers of code and less modularity - all in the aim to make 'the ultimate megaglobular ultrauniversecorporate site'. Almost as if small clients with 'just a website' don't exist. Enterprise products belong in the enterprise domain. When you upgrade a non-'do the mightiest world-spanning web site/application' system, you shouldn't get a 'do the mightiest world-spanning web site/application'system.
The bigger system may be able to do more things and be more scalable to large systems, but they leave behind no small, accurate tool to do small and medium jobs quickly and with ease. Also: "Documentation is not scalable" (c) H. Erlandsson. To even find the doItAll() function or even know it exists, you have to wade through increasingly huge documentation. To then learn how to use it correctly, if you don't know all the component the function builds on, you have learn the behavior of many subsystems to understand how to apply some terse declaration to your problem.
The ultimate something can be several refined gems, and not an asteroid-size cluster of crystals.
Was a bit tricky to read out exactly what you're asking, hope I read you right. I can recommend some programming system gems, but they are not in the web dev domain, maybe others will know a few :)
I'm going to develop a collaborative site, and one of the features will be collaborative editing with realtime changes. i.e. when two or more users are editing the same doc, they can see each other changes as soon as they happen.
I have some experience with Ruby on Rails, so I was thinking about using EventMachine, but with all this hype around Node.js, I am know considering using it instead. So, what would be the main benefits of using Node.js instead of EventMachine?
tl;dr
What are the main differences between EventMachine and Node.js (besides the language)?
EventMachine has nothing to do with Rails apart from them both being written in the same language. You can get EventMachine as bare as Node.js; all you have to do is not add libraries to your project. In my experience the EventMachine libraries (like em-http) are much nicer than anything for Node. And you can use fibers instead of callbacks to avoid callback hell. Complete exception handling is pretty much impossible in Node because of all the callbacks. Plus Ruby is a nicer, more complete language than Javascript.
I tend towards the "use what you know" (even if it's a heavier architecture). Because of that, I don't see it being quite as simple as "EventMachine vs NodeJS." Mainly, the difference can be summarized as this:
NodeJS is a framework/language that was written to handle event based programming in JavaScript. That is its driving force. It's not an after thought, or a third party mechanism. It's baked right in to the language. You create callbacks/events because that's how the language is built. It's not a third party plug in, and doesn't alter your workflow.
EventMachine is a gem in Ruby that gives developers access to some of the goodness of the event based programming model. It's heavily used and well tested, but not baked directly in to the language. Both are locked to one CPU, but with event programming at Nodes core, it still has a leg up. Ruby wasn't written with concurrency in mind.
That said, technical problems can be overcome. The more important questions (from my view) that should guide your decision are these:
What will your production environment look like? Do you have complete control over the server? Can you host it however you want? Or will it be on a shared system to start with, and then you have to expand on that?
Do all the developers on your team have the ability to learn a new language very fast? How fast will they be able to understand an event-based language like JavaScript for the middle tier?
Do you need all of the architecture that Rails gives you (full Testing framework, scaffolding, models, controllers, etc)? Or is that overkill?
There are quite a few technical differences between the two. One is a language, one is a framework. Really, how heavy of a stack you want to run? How much learning will your developers have to do? Do you want a full stack the gives you a lot of niceties, that you may not use, or do you want a bare bones set up that runs extremely fast and concurrent, even though you may have to write extra boiler plate code and learn a new lanugage?
While Rails is not as heavy as some web application architectures, you're still going to need more processor power than you would to handle a similar amount of throughput in NodeJS. Assuming quality code for both systems. Bad code written on either stack is going to prevent the stack from shining. It really comes down to- Do you really want to learn a whole new way of doing things, or utilize your current understanding of Ruby to get things off the ground fast?
I know it's not really a definitive answer, but I hope this helps guide you to a decision!
One thing worth mentioning is the production story. EM, like most Rack stuff, has plenty of testing and monitoring tools available that are well tested, whereas Node.js falls well short in this respect.
At the time of writing, it seems almost impossible to get clear metrics from Node to answer questions like 'Do I need to scale'. There are options starting to form out there from the likes of Joyent, and always the roll-your-own argument, but nothing anywhere near tools such as NewRelic.
Node.js is very good from a performance / configurability point of view, but personally I wouldn't host it in production just yet.
Node.js
You get far better control low level control over what's going in. You can include general libraries to build on top of node.js to tweak your level of abstraction to your own liking. For example you can use connect or express depending on whether you want a view engine written for you.
You can use socket.io or now depending on how much you want your client-server connection abstracted. You can opt to include any of numerous MVC libraries or write your own.
Event-Machine
An asynchronous IO library just like node.js
It comes down to a Ruby vs JavaScript preference, how much flexibility you want with abstractions or lack of abstractions and whether you want to use node as your actual web server.
a detailed view at confusion has already been proposed... just a personal view
[] node.js will be better, if you are ready to learn and experiment more than you think because:
it's thread mechanism is awesome (inspired from that of 'erlang')
you can build a purpose specific server (easily) which will be real productive
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I am looking at Erlang for a future version of a distributed soft-real-time hosted web-based telephony app (i.e. Erlang looks like absolutely the perfect choice for this kind of app). I come from a .NET background and the current version of this app uses a combination of C#, WCF and JQuery to deliver the service. I now need Erlang to allow me to add extra 9s to my up-time and to allow me to get more bang for my server bucks.
Previously I'd set up a development process here combining VS.NET, GIT, TeamCity and auto-deployment of MSI files to the various environments we maintain. It's not perfect, but we're all now pretty comfortable with it. I'm wondering whether a process like we have is even appropriate for such a radically different technology stack (LYME)?
I'm confident that all of the programming challenges we previously solved using .NET can be better solved in less code with Erlang, so I'm completely sold on the language choice. What I don't yet understand from reading the Pragmatic and O'Reilly books on Erlang, is how I should adapt my software engineering and application life-cycle management (ALM) processes to suit the new platform. I see that in-place code updates could make my (and my testing and ops team's) life much easier (compared to the god-awful misery of trying to deploy MSI files across a windows network) but I am not sure how things should change when I use Erlang.
How would you:
do continuous integration in Erlang (is it commonly used?)
use it during a QA cycle (we often run concurrent topic branches using GIT, that get their own mini-QA cycle, so they all get deployed into a test environment)
build and distribute your code to DEV, TEST, UAT, STAGING, and PROD environments
integrate code generation phases into your build cycle (we currently use MSBUILD + T4 templates)
centralize logging for a bunch of different servers (we currently use Log4Net, MSMQ, etc)
do alerting with tools like SCOM
determine whether someone/something has misconfigured your production servers
allow production hot-fixes only after adequate QA (only by authorized personnel)
profile the performance (computation and communication) of your apps
interact with windows-based active directory servers
I guess I need to know what worked for you and why! What tools and frameworks did you use? What did you try that failed? What would you do differently if you could start over, knowing what you know now?
Whoa, what a long post. First, you should be aware that the 99.9% and better kool-aid is a bit dangerous to drink while blind. Yes, you can get some astounding stability figures, but you need to write your program in a way facilitating this. It does not come for free. It does not happen by magic either. Your application must be designed in a way such that other subsystems recover. OTP will help you a lot - but it still takes time to learn.
Continuous integration: Easily done. If you can call rebar or make through your build-bot you are probably set here already. Look into eunit, cover and Erlang QuickCheck (the mini variant is free for starters) - all can be run from rebar.
QA Cycle: I have not had any problems here. Again, if using rebar you can build embedded releases that are minimized erlang vm's you can copy anywhere and run (they are self-contained). You can even hot deploy fixes to such a system pretty easily by altering the code path a bit so you have an overlay of newer fixes. Your options are numerous. Git already help you here a lot.
Environmentalization: Easily done.
Logging centralization: Look into SASL and the error_logger. You can do anything you want here.
Alerting: The system can be probed for all you need (introspection is strong in Erlang). But you might have to code a bit to hook it up to the system of your choice.
Misconfiguration: Configuration files are Erlang terms. If it can be computed, it can be done.
Security: Limit who has access. It is a people problem, not a technical one in my opinion.
Profiling: cprof, cover, eprof, fprof, instrument + a couple of distributed systems for doing the same. Random sampling is also easy (introspection is strong in Erlang).
Windows interaction: Dunno. (Bias: last time I used windows professionally was in 1998 or so).
Some personal observations:
Your largest problem might end up being that you try to cram Erlang into your existing process and it might resist. It is a new environment, so new approaches will be needed in places and you should expect to adapt and workaround limitations you find along the way. The general consensus is that it can work (it is working for several big sites).
It looks like you have a well-established and strict process. How much is that process allowed to be sacrificed to give way to a new kind of thinking?
Are your programmers willing to throw out almost all of their OO knowledge? If not, you will end with a social problem rather than a technical one. If they are like me however, they will cheer, clap in their hands and get a constant high by working with an interesting language solving an interesting problem in a new way.
How many Erlang-experienced programmers do you have? If you have rather few, then better cut your teeth on some smaller subsystems first and then work towards the larger goal. Getting the full benefit of the system takes months if not years. Getting partial benefit can be had in weeks though.
I have an idea for a large web application, which in my mind would really be a number of reasonably independent application "components" (?), e.g. forum, blog etc.
It seems to me that it would be nice to open source some of the individual components for others, or myself in the future to use.
Now clearly, I could write a basic forum, make the code available, then copy it across into my large app and extend/integrate it. However, this doesn't feel very DRY. In an ideal world I could maintain the code in one place and drop it into my larger application.
Rails seems to include engines, which sound like solve my problem but it would appear that what I want to do is a bad idea!
So should I..
a) Just build one large, coupled app.
b) Create a rails forum app (for example), release it, then merge/copy it into my main app
c) Use engines
d) Other...
Cheers,
Adam
That warning is just the standard Uncle Ben Corollary. Paraphrased from "With great power, also comes great responsibility." to "Just because you can, doesn't mean you should."
However, just because there are a number of uses for engines that are bad ideas, that doesn't mean it is in your case. Blogs and forums are two reasonably independent components that have already been produced as plugins using Engines. You might not even need to make them.
The way I see it, if you're planning on reusing a functional component that you can feasibly segregate from the rest of the app, then that's a good enough reason to make it a plugin, whether or not it relies on engines.
My experience is that these plugins grow from existing application code I've written. It's much easier to conceive, write, and test them as part of an application than on their own. Engines being miniature apps, aren't that much harder to build and test.
In short. Only you can decide which path is best, because you are most familiar with the goals of your app. Before you start have a look at what others have done with engines and plugins. The components you want to build may already exist.
To answer the question, I would probably start by building a single app, and extract out the bits that could be useful in other apps and abstract them into [engine] plugins as they reach maturity.
What you might be looking for Git Submodules. I don't know a ton about them, but you can simply have a git repo within a git repo. And you can update all your inner git repos pretty easily. Here's an article talking about it.