We are planning to make a "large" website for I'd say 5000 up to many more users. We think of putting in lots of real time functionality, where data changes instantly propagate to all connected clients. New frameworks like Meteor and DerbyJS look really promising for this kind of stuff.
Now, I wonder if it is possible to do typical backend stuff like sending (bulk) emails, cleaning up the database, generating pdfs, etc. with those new frameworks. And in a way that is productive and doesn't suck. I also wonder how difficult it is to create complex forms with them. I got used to the convenient Rails view helpers and Ruby gems to handle those kind of things.
Meteor and DerbyJS are both quite new, so I do expect lots of functionality will be added in the near future. However, I also wonder if it might be a good idea to combine those frameworks with a "traditional" Rails app, that serves up certain complex pages which do not need realtime updates. And/or with a Rails or Sinatra app that provides an API to do the heavy backend processing. Those Rails apps could then access the same databases then the Meteor/DerbyJS app. Anyone thinks this is a good idea? Or rather not? Why?
It would be nice if anyone with sufficient experience with those new "single page app realtime" frameworks could comment on this. Where are they heading towards? Will they be able to handle "complete" web apps with authentication and backend processing? Will it be as productive/convenient to program with them as with Rails? Well, I guess no one can know that for sure yet ;-) Well, any thoughts, guesses and ideas are welcome!
For things like sending bulk emails and generating PDFs, Derby let's you simply use normal Node.js modules. npm now has over 10,000 packages, so there are packages for most things you might want to do on the server. Derby doesn't control your server, and it works on top of any normal Express server. You should probably stick with Node.js code as much as possible and not use Rails along with Derby. That is not to say that you can't send messages to a separate Rails app, but since you already have to have a Node.js app running to host Derby, you might as well use it for stuff like this.
To communicate with such server-side code, you can use Derby's model events. We are still exploring how this kind of code works and we don't have a lot of examples, but it is something that we will have a clear story around. We are building an app ourselves that communicates with an email server, so we should have some real experience with this pretty soon.
You can also just use a normal AJAX request or send a message over Socket.IO manually if you don't want to use the Derby model to do this kind of communication. You are free to make your own server-side only routes with Express along with your Derby app routes. We think it is nice to have this kind of flexibility in case there are any use cases that we didn't properly anticipate with the framework.
As far as creating forms goes, Derby has a very powerful templating system, and I am working on making it a lot better still. We are working on a new UI components feature that will make it possible to build libraries of self-contained UI widgets that can simply be dropped into a Derby app while still playing nicely with automatic view-model bindings and data syncing. Once this feature is completed, I think form component libraries will be written rather quickly.
We do expect to include all of the features needed for a normal app, much like Rails does. It won't look like Rails or work like Rails, but it will be similarly feature complete eventually.
For backend tasks (such as sending emails, cleaning up the database, generating pdfs) it's better to use resque or sidekiq
Now, I wonder if it is possible to do typical backend stuff like
sending (bulk) emails, cleaning up the database, generating pdfs, etc.
with those new frameworks. And in a way that is productive and doesn't
suck. I also wonder how difficult it is to create complex forms with
them. I got used to the convenient Rails view helpers and Ruby gems to
handle those kind of things.
Also, my question is not only about background jobs, but also about stuff one can might do during a request, like generating a pdf, or simply rendering complex views with rails helpers or code from gems. –
You're mixing metaphors here - a single page app is just a site where the content is loaded without doing a full page reload, be that a front end in pure js or you could use normal html and pjax.
The kind of things you are describing would be done in a background task regardless of the fornt-end framework you used. But +1 for sidekiq if you're using ruby.
As for notifying all the other users of things that have changed, you can look into using http://pusher.com or http://pubnub.com if you don't want to maintain a websocket server.
Related
Background
I have a fairly typical Rails application, which uses Devise for authentication management. While building this app, I realized that realtime chat would be a great feature to have. Ideally, of course, this would make use of Websockets, in order to reduce the polling load on the server (as well as making it marginally easier to implement, as you don't have to manage polling).
I realized quickly that Ruby isn't really a great fit for having a large number of concurrent connections open at one time. Phoenix, however, is written in Elixir, so I can make use of the Erlang VM, which is quite good at long connections. It also seems like it could be greatly beneficial if all the chat data was stored separate from the main application database, which should also reduce load in the future.
The Problem
I want to be able to make this separation completely invisible to the user. They visit www.example.com/chat, and it loads all the relevant data in from chat.example.com and starts up the websockets, without requiring them to login to a separate service. I think using an <iframe> is probably the way to go about doing this.
My problem is sharing authentication and data between the two applications. The Rails app needs to be able to create conversations on the Phoenix app in response to certain events. The Phoenix app needs to know what user is currently authenticated into Rails, as well as general data about the user.
An OAuth flow with the Rails app as the ID provider seemed like a good fit at first, but I can't figure out a way for the Phoenix app to automatically be granted access. I also have some concerns about user records existing inside the Phoenix app—it should be aware of all users on the main application, so you can start a chat with a user even if they haven't ever opened chat.
What would be the best way to go about doing this? My intuition says that this is going to involve window.postMessage and some kind of token system, but I wanted to ask what the generally accepted way of doing this was before I accidentally created an insecure mess.
Sharing the session isn't too hard, assuming you are running at least Rails 4.1 and using JSON serialization (default for apps created with >=4.1). A quick google search finds PlugRailsCookieSessionStore, which accomplishes this.
For more information on what it takes to share a session between Rails and another language, Matt Aimonetti has an excellent blog post with detailed information.
Lastly, if you would prefer to stay entirely in Ruby, it's definitely doable. Ryan Stout discusses scalability around persistent connections in the FAQ for Volt, which uses a persistent connection for every user. The article he links is also a great read. Just mentioning it to help you weigh the trade off of building a separate app in another language.
Hypothetical question (at the moment!)
Suppose I have a great idea for an application. It acts on data which can be well-represented by tables in a relational database, using interlinked objects which represent those tables. It supports a well-defined API for interacting with (Creating, Reading, Updating, Deleting) those objects, and viewing information about them.
In short, it's a perfect fit for Rails... except it doesn't want to be a web-app. Perhaps it wants a Command Line interface; or an OS-native dialog-based interface; or perhaps it wants to present itself as a resource to other apps. Whatever - it just isn't designed to present itself over HTTP.
These questions suggest it's certainly possible, but both approach the problem from the point of view of adapting an existing web-app to have an additional, non-web, interface.
I'm interested in knowing what the best way to create such an app would be. Would you be best to rails new non_web_app, in order to get the skeleton built "for free", then write some "normal" Ruby code that requires config/environment - but then you have a lot of web-centric cruft that you don't need? Or would it be better to roll up your sleeves and build it from whole cloth, taking just the libraries you need and manually writing any required configuration?
If the latter, what exactly is needed to make a Rails app, but without the web bits?
If you want to access the Rails ORM to develop a CRUD non-web application, just include ActiveRecord in your own Ruby script; you will avoid using a lot of Rails modules you probably don't need (routing, template generator, ...) Here is an example of how to do it.
If you prefer to have the full Rails stack, do not run your Rails web app in an application server (WEBrick, Passenger, Mongrel, ...) to avoid any HTTP exposure, and interact with your application using tasks or the rails console.
I would avoid taking Rails too far off the rails. If I were doing this and felt that the gains of rails w/o the web stuff I'd do the following:
rails new non_web_app
and ignore the webbish cruft and use rails to generate models. In this way you get the tight, comfortable database behavior and can tie various gems in as you want to augment those models. I'd not bother implementing views, of course, and I'd consider implementing controllers in which the various render bits are removed and to use you instantiate an instance of the controller and call the action directly. This means the controller represents your API into your business logic still but the "views" it now "renders" are simply the return of the data output.
Then you could simply strip out the bits you do not need...the public directory, the view structure under app, config/routes.rb, etc. You'll need to test those changes incrementally and make sure that removing some now extraneous bit doesn't throw the Rails world into chaos.
Rails is for Web apps. That means HTTP. Now, you could package a Web app so that it runs on the desktop instead, or you could use ActiveRecord with a desktop application framework like Monkeybars.
I'm going to be collaborating with a Python developer on a web
application. I'm going to be building a part of it in Ruby and he is
going to build another part of it using Django. I don't know much about
Django.
My plan for integrating the two parts is to simply map a certain URL
path prefix (say, any request that begins with /services) to the Python
code, while leaving Rails to process other requests.
The Python and Ruby parts of the app will share and make updates to the
same MySQL datastore.
My questions:
What do people think generally of this sort of integration strategy?
Is there a better alternative (short of writing it all in one language)?
What's the best way to share sensitive session data (i.e. a logged in
user's id) across the two parts of the app?
As I see it you can't use Django's auth, you can't use Django's ORM, you can't use Django's admin, you can't use Django's sessions - all you are left with is URL mapping to views and the template system. I'd not use Django, but a simpler Python framework. Time your Python programmer expanded his world...
One possible way that should be pretty clean is to decide which one of the apps is the "main" one and have the other one communicate with it over a well-defined API, rather than directly interacting with the underlying database.
If you're doing it right, you're already building your Rails application with a RESTful API. The Django app could act as a REST client to it.
I'm sure it could work the other way around too (with the rest-client gem, for instance).
That way, things like validations and other core business logic are enforced in one place, rather than two.
A project, product, whatever you call it, needs a leader.
This is the first proof that you don't have one. Someone should decide either you're doing ruby or python. I prefer ruby myself, but I understand those who prefer python.
I think starting a product asking yourself those kind of questions is a BAD start.
If your colleague only knows prototype, and you only know JQuery, are you going to mix the technologies too? Same for DB? And for testing frameworks?
This is a never ending arguing subject. One should decide, IMHO, if you want so;ething good to happen. I work with a lot of teams, as a consultant, Agile teams, very mature teams for some of them, and that's the kind of stuff they avoid at all cost.
Except if one of you is going to work on some specific part of the project, which REALLY needs one or other of the technologies, but still think the other one is best for the rest of the application.
I think, for example, at a batch computing. You have ALL your web app in ror or django, and you have a script, called by CRON or whatever, computing huge amounts of data outside the web app, filling a DB or whatever.
My2Cts.
This is a bit of a weird meta-programming question, but I've realized that my new project doesn't need a full MVC framework, and being a rails guy, I'm not sure what to use now.
To give you a gist of the necessary functionality; this website will display static pages, but users will be able to log in and 'edit their current plans'. All purchasing and credit card editing is being handled by a recurring payment subscriber, I just need a page to edit their current plan. All of that will be done through (dynamic) XML API calls, so no database is necessary.
Should I stick with my typical rails/nginx stack, or is there something I could use that would lighten the load, since I don't need the Rails heft. I'm familiar with python and PHP but would prefer not to go that route. Is Sinatra a good choice here?
tl;dr: What's a good way to quickly serve mostly static pages, preferably in Ruby, with some pages requiring dynamic XML rendering?
If you want to stick with Ruby, Sinatra would be fine, as would Rails Metal.
If you're feeling a bit adventurous and want to get some useful experience with the technology that rails uses you could try building a Rack application. It's a pretty simple API to be able to respond to generic HTTP queries, and from there you can quickly build static file handling and XML processing. It's also considerably faster to start up and serve pages than rails.
http://github.com/cloudhead/toto is an example of a decent Rack based application.
If you know Rails, then why not just stick with it? That way you can use all authentication features, etc. that you're used to without having to learn another platform and incur the implementation risks that that includes. If the application ever grows beyond what's expected you're already on a solid base.
We are developing a considerably big application using Ruby on Rails framework (CRM system) and are considering to rewrite it to use ExtJS so that Rails would just do the data handling, while ExtJS would do all the browser heavylifting in a desktop-like manner.
Anyone has some experience and hints about what would be the best approach? Is ExtJS mature enough to be used in relatively big (and complex) applications? And what about the Rails part - what would be the best approach here?
EDIT:
Just to make it clear. I would prefer to do it in such a way that all the javascript client side application code is loaded at once (at the start up of the application, optimally as one compressed js file) and then just use ajax to send data to and from Rails app. Also, it would be nice to have ERB available for dynamic generation of the Ext apliccation elements.
I currently have an extremely large, desktop style app written in ExtJS. It used to run on top of Perl's Catalyst MVC framework, but once the entire View layer was converted to an ExtJS based desktop I started migrating to Ruby on Rails models and controllers. It is equally as fast, if not faster, and easier to maintain and has a much smaller code base.
Make sure that you set your active record config to not include the root name of the model in the json, so that Ext's JsonStore has no problem reading the records. There is an option on ActiveRecord BASE called include_root_in_json you have to set to false.
Make sure that you properly define your Application classes in Ext and maximize code re-use and you are going to want some sort of method to clean up unused nodes in the DOM. Javascript performance can be a real pain unless you are using the latest versions of Safari or Firefox 3.1.
You probably will want some sort of caching method for data on the server to be served to your application in JSON format at the time the page is loaded. This will cut down on the number of round trips via Ajax.
Definitely make use of Ext's WindowManager and StoreManager objects, or roll your own from Ext.util.MixedCollection
Develop your code in separate, managable files, then have a build process which combines them into a single file, and then run YUI's compressor or Dean Edwards Packer on it to compress / obfuscate the file. Serve all JS and CSS in their own single files, including the Ext supplied ones.
[2012 update] ExtJS was acquired by Sencha, who offer a GPLv3 license, and two commercial licenses.
[2008-Oct comment] ExtJS is great on technical merits, but the fiasco with the licensing several months ago have led me to look at other frameworks - I don't trust the creators of ExtJS at all now. I don't like how they worded their license, and how they pretended to be open source advocates whilst obviously attempting to profit unfairly off those who believed them.
I'm only against using ExtJS on moral grounds.
This belongs in answer to Milan's comment on my previous answer, but as a newcomer here I don't have enough reputation points to reply there:
There was a problem with the "sp is undefined", which was a result of Rails' caching of the JavaScript files into one large file (there would be several hundred files otherwise). The caching introduced some weird bugs with newlines which threw the whole thing off. This had me pulling my hair out for a while, but the solution was to update Ruby from 1.8.6 (patch level 72) to the latest 1.8.7. This fixed the problem so please check it again if you want to have a look (you'll need to do a full refresh to beat the asset caching).
I'm glad you've come across the Ext MVC stuff before. At present I can fully believe it must be quite difficult to understand, mainly due to a lack of examples, tutorials and demos. The code itself is reasonably well documented however (at least the newer code anyway, there is a lot which needs clearing out).
I am currently in the process of refactoring a few key classes before it is ready for a proper 'release'. When that's ready (I'm thinking a couple of weeks), I will generate the documentation and set up a quick site with some demos and example code. When I've done so I'll put up a post on my blog (http://edspencer.net).
My aim with this is to try to provide a framework which will make writing this type of application much simpler, and to establish some conventions. Currently there is no consensus or default way of structuring ExtJS applications, so anything we can do to move that along will be a step in the right direction! Comments and contributions are more than welcome.
I've successfully deployed a large RoR/ExtJS app of the kind you describe ("single-page" client-side AJAX driven). Ext_scaffold is pretty much a red-herring.
It's not too taxing to get RoR and ExtJS working smoothly together. The fundamental choice is whether to extend ExtJS to "speak Rails", patch RoR to "speak ExtJS", or meet in the middle. It's going to depend where your team's skills are.
I adopted the meet-in-the-middle strategy, which includes:
Extend Ext.data.Store and Ext.data.Record to be aware of Rails routing conventions
Hack Ext.grid.EditorPanel and Ext.form.BasicForm to play well with ActiveRecord associations
Write some modules to extend ActiveRecord::Base and ApplicationController to simply commits from Ext.grid.EditorPanel and Ext.form.BasicForm
That's pretty much it.
Having said that, there are drawbacks to ExtJS.
You're going to have to get your hands dirty in the internals. Don't be beguiled by the demos.
The community documentation is poor and PHP-centric.
Coming from the Github/Lighthouse-centred RoR world, using VBulletin is like waking up in 1998. I mean, there's no public bugtracker just a forum post that's updated (WTF?).
The code is a bit over-engineered.
The team have lost Open Source credibility so they've lost Open Source oxygen.
The team appear to be focused integration with GWT (can anyone say "enterpri$ey"?).
You might want to have a look at the Netzke framework that is thought to do just that: facilitate creating complex one-page web-application with the emphasis on modular approach.
The advantages of Netzke are:
Reusability and extensibility of the code. Once you get your component (both client and server side) made, you can reuse it in any place, combine with other components, or event extend it with inheritance.
Efficiency. Class for every component is loaded from the server (and evaluated) only once, which saves a lot of time on server-client communication.
It's open source, and it's in active development. It has live demos and example code.
It has prebuilt components that you can use straight away without even touching Ext JS (just configure them in Rails)
It's been used (by its author) for real-life development of a complex logistics application.
Disadvantages of Netzke are:
The code is still young, and the community small.
If you're interested, have a look at the description and design details here: https://github.com/nomadcoder/netzke-core
Live demo/tutorials can be found here: http://netzke-demo.herokuapp.com and here: http://yanit.heroku.com
Ext is definitely mature enough to handle this situation. I'm currently working on a Rails project with a lot of Ext, and the hardest part has definitely been working with Rails's to_json to render JSON that Ext can read (for arrays, hashes, models, which failed validation, etc.)
Check out the ext_scaffold plugin for Rails. I started with this and hacked away at its ActiveRecord/ActionView extensions until it did what I needed it to do.
I has some experience using ExtJS with Rails too. Using the framework is a great way to get some nice looking widgets for free. REST convention should sit well with the framework too if you use it to develop single page applications. Works well with RJS too.
Here are my gripes with using the framework
You can't really make use of flash[:notice] since reloading a single page application is silly. This makes passing validation notices and messages a chore since you have to use RJS/ javascript methods to show them.
You can't use erb much thus you have to encapsulate a lot of the logic into the json callbacks.
I've deployed ExtJS and Rails for a number of applications and they certainly can be made to talk to each other. We've put together a quick demo of an app we're currently developing in Rails + Ext at http://demo.domine.co.uk/admin. Ignore the front end for now as it's not complete - the admin section is essentially finished and you can log in to it with:
username: edward
password: rarrar
As the demo's not completely finished yet I won't guarantee that it works correctly in anything other than Firefox at this stage. There's no reason for it not to work in other browsers, I just haven't spent any time testing them yet. The point is more about the integration with rails though.
Every application on the start menu is interacting with the Rails backend via JSON. I've written a basic Rails plugin to do most of the work for us there. I'll be releasing the code behind that shortly but for now hopefully that gives some idea of how well these two technologies can work together...
While I have no experience of ExtJS (besides reading about in the "Practical Rails Projects" book) I used a jQuery Flexigrid with jrails to get more of a desktop feel.
That worked pretty well.
Ok. I use extjs gxt gwt on many project, and it very easy for develop. But I want to tell you that I built my project with extjs+gwt (gxt), I don't sure about Ruby.
link text