I'm working on a Rails 2.3.x app that needs to run a long, slow task asynchronously.
Delaying the request until processing has completed is a bit of a pain. We're happy to just send a "Okay, we've received that and we're working on it"-style response back to the client, and then just crunch on the work once the client's gone away.
So I've seen a few methods for doing this (such as forking or using cron), but it seems that task queuing systems might be a better idea.
It seems like the three main ones are:
BackgroundFu,
BeanstalkD, and
Sparrow
Sparrow looks particularly interesting due to it's support for Memcached, but I don't want to just pick one without making an informed decision. Does anyone have any insights regarding best practices? I'd love to hear about your experiences with the above, especially any major gotchas with this approach or the libraries mentioned. Perhaps there's an even better way to do this that I've missed completely?
Just so it's out there: Rails 3.x isn't an option right now, this needs to work for Rails 2.3.x.
Thanks!
I recommend delayed_job, specifically this fork and branch. I'm using it myself on rails 2.3.5 and it works well.
It has a rails 3 branch for when you do want to upgrade, it works with multiple backends, and it has an optional pretty syntax:
Notifier.delay.deliver_signup #user
By the way, just forking is a mistake as you'll find the fork doesn't have a good database connection.
Related
I am using Ruby on Rails 3.2.2. and I would like to know if the following thought (born from my previous question) is correct:
I tend not to use third-party gems (at least when it is possible and reasonable) because they might be abandoned at any time. Also, if the Ruby on Rails framework changes, I might have to wait for those gems to be updated before updating my application to the latest RoR version.
In a perfect world where you're immortal, have infinite time, are the world's best programmer, and it makes no difference whether you launch your product in 1 hour or 10 years so you can code everything yourself from scratch, then maybe it makes more sense to avoid the code of other people.
But in the real world, people have solved problems for you already. And gems allow you to plug in those solutions.
A good rule of thumb is to prefer the community's favorite/popular gems for a given solution when you can. The more people using a gem, the more people have an interest in keeping it updated and the more eyeballs are scrutinizing it and sending pull requests. Chances are that a gem that's battle-tested by a bunch of people in production environments is better than what you'd come up with on your first attempt, right?
Gems that depend on one hobbyist maintainer that's not even using it in a self-itch-scratching production application are where you tend to find some risk. But even in those situations, if you end up having to fork his gem one day, you're still ahead of where you'd be if you started from scratch.
A better tendency is to avoid writing everything from scratch and instead leverage the brains of other people with similar needs.
Well, it's the risk you're taking in exchange for saved development time/budget and shorter time-to-market.
If you find yourself in a situation when one of your gems is abandoned, then you could look at similar gems or fork and improve that one. Either way, it's still better than developing everything by yourself.
Abandon this philosophy! A lot of websites/companies tend to do this -- it's better known at the "Not invented here" syndrome. Twitter, in particular, is a big offender. Open source is great -- use it.
A work is perfectly finished only when nothing can be added to it and nothing taken away.
– Joseph Joubert
That's the same philosophy we follow with our Rails projects at work. If you want to add a gem to the Gemfile, you basically have to argue it in. If you can't find a good reason why it should be in there, it will get kicked out. We are definitely not trying to reinvent the wheel, but keeping the number of potentially badly maintained moveable parts down is IMHO a good choice. So say yes to well-maintained and established Rails plugins, but be wary about stuff that looks like a quickly thrown together solution. More often that not you end up maintaining your own patches of it or forking the repo, so you might have as well reinvented that particular wheel...
Try, for example,devise gem, and then implement its functionality yourself. So you can define your own vision of the problem.
I prefer to use all gems, which I found useful. If after update any gem doesn't work, you can always fork and upgrade it.
I have some (I think) really great ideas for an online strategy game similar to Travian. There's some content that I haven't yet figured out and some other challenges that I don't know of yet.
This is quite a big project and perhaps too heavy for one person that isn't a skilled web developer (yet). I'd still like to give it a shot, but I'm having trouble choosing a platform. The world "scales" has been thrown around a lot lately and I've seen Ruby on Rails being bashed because it doesn't scale well, so I've come here to get some answers.
I like Ruby on Rails, both Ruby and Rails. I'm certainly no expert at it but I love working with it. I have also worked with Python + Django before and also with PHP (which I am not fond of.)
Ideally the game would have, let's say, 7000 players per server, presumably a lot of data to be processed per second. Would RoR still be a viable platform?
I'm sorry if this question is vague, I guess I'm looking for a "RoR is fine, go at it!" kind of answer. Anything you might want to add is fine.
Thanks!
So if I were you, I would be looking into non-blocking servers like node.js, just because they are MUCH more suited to keeping many connections open for long periods of time, which is what games need to do, compared to traditional web servers.
That being said
There are 3 main things to worry about when you are scaling a web app; memory, execution speed, and io (hd and network) in that order.
For memory, things are much better then they used to be. Phusion Passenger uses copy on write to fork its workers, so the rails environment will get shared among all the workers on a given slice, which is pretty significant. There have also been huge improvements to the way ruby manages memory compared to "the dark times", if you are using 1.8.7 then you want to be using the patches that make up Ruby Enterprise Edition (the difference is like night and day). 1.9.x was pretty much a total rewrite of the runtime, so if you are using that the memory issues ruby had have already been addressed.
For execution speed, 1.8.7 is typically "fast enough" (at least after tuning garbage collection settings). 1.9.2 is actually around the same speed as python, which puts it on the faster side of interpreted languages. How important this point is completely depends on the nature of your application.
Last point is IO, which isn't really a concern of rails, but more your persistence strategy. Rubyists tend to love new things, so you will find first class support for things like redis and mongodb, with loads of people talking about using them and their wins/gotchas. I would look into mongo if I were you and see if the durability trade-offs are acceptable.
I was in java/.net before going to rails, and at the end of the day you are going to pay more for infrastructure, but the amount will be completely dwarfed by what you save in development time.
build it in Rails, host it on Heroku.com - job done. Almost infinite scaling that you don't have to worry about how it works (it just does) and it hosts a lot of highly trafficked Facebook apps so can more than handle it.
I think Ruby on Rails is a good choice for what you need. Actually, we recently created a platform for an online gaming tournament, where players and their gaming bots were playing a logic game.
We used Ruby on Rails and Sidekiq on the backend, ReactJS and WebSocket on the frontend. And it worked well for a quite massive number of players. Here is the tutorial based on what we learnt while building it: How to Write a Game Engine Using Ruby on Rails
As you said yourself, you already have the answer and you are only looking for encouraging words:). I am not RoR expert myself, but I don't think scalability is still such a great issue on this platform. I would advice you to do an architecture spike (XP terminology). Write a test with 7000 clients and method which would perform similar operations to what you intend to create. For example you might load files, render views or even just wait... The point is to test only the thing you are worried about. Good luck!
This is a bit of an impossible question to answer because in order to know whether rails is suitable for what you want to do we would need to a lot more about what you are trying to do. The best advice I can give in the absence of information is for you to check out the railslab scaling videos in order to work it out for yourself.
I am experimenting with Ruby and Rails. I like Ruby, but not Rails. I have Java/PHP background, I have used some frameworks, but never totally liked any of them.
Anybody using Ruby to build web apps, but not any of the frameworks? (rails, merb etc). If yes, can you point me to some resources to learn it?
It might be wise to at least leverage Rack, and maybe Sinatra gets enough out of your way for you to feel comfortable. Sinatra isn't much more than a very small, simple wrapper around the rack handler afaik.
Doing Web apps in Ruby without using a framework is like cooking without heat. I question the sense of trying this.
That said, you could probably hook up Apache to call a Ruby program as a CGI. I suspect it would be dog slow, though.
Edit: Apparently you're not the only one crazy enough to attempt this, though: I found a tutorial on Simple Ruby CGI. The author claims as his rationale that "he has nothing better to do."
Your question looks quite similar to Ruby off the rails, so you probably should read the answers of that one.
I'm using ruby instead of bash and python on my ubuntu server for sysadmin tasks.
I've found Ruby a particularly clean and powerful replacement for the places I'd have previously used bash and python in sysadmin.
I personally found one thing for myself. When I prepare some interview or I am gonna build something from scratch, in most cases I need some basic features included in Rails. That usually happens when you build something not complex, but still non trivial. In 99% of cases I need ActiveRecord, ActiveSupport etc. It is easy to get those things in your Gemfile and play with them. One thing you will always repeat is basic application structure, easy console access with preloaded libs, rakes... I created minimal gem for myself to organise this process though.. :) If you read sources you'll see what I constantly do to start something new. It is not a big deal not to use any framework at all.
https://github.com/einzige/framework
What message queues are people using for their Rails apps and what was the driving force behind the decision to choose it. Does the latest Twitter publicity over their in house queue Starling falling down affect any existing design decisions.
I am working on an app that will need a message queue to process some background tasks, I haven't done much of this, and most of the stuff I have seen in the past has been about Starling and Workling, and to be honest the application is not very big and this solution would probably suffice, but I'd love to get experience integrating the best solution possible as I'm sure I will integrate one into a bigger app at some point.
What message queues would you suggest for a Rails app???
EDIT: Thanks for the suggestions, I'm going to look at a few of them this weekend.
EDIT Again: I've had a look around and a little overwhelmed for choice. I am however going to go about integrating RabbitMQ with Workling into the app I am building, then if I ever need some knowledge about a fast queue then I will have this and know whether or not it fits my needs.
EDIT: Finding more and more that DJ suits me just fine, if I ever "outgrow" it on a site I'd say that Resque is where I would head.
EDIT: (Dec 2014) So it's been a long time since I asked this, but I see it still gets some views or some votes, so I figured I'd update it on my approach now when it comes to my choice of background workers.
In my opinion currently the best way to run background jobs in Ruby is using Sidekiq. A lot of people have really lauded Sidekiq for it's threaded workers rather than process per worker which can use significantly less memory than the likes of Resque, which I was using before Sidekiq. This is good but for me this wasn't the killer feature. By using Sidetiq with Sidekiq, the scheduling of jobs is so trivial that I switched over and have never looked back from it, by far the easiest scheduling of jobs that I have used and has made Sidekiq a breeze to use.
As an update -- GitHub have moved to Resque on Redis instead of Delayed job. However they still recommend delayed_job for smaller setups:
https://github.com/resque/resque
Chris Wanstrath from github was at the SF Ruby meetup recently, talking about their queue. They tried Starling, beanstalk, and some other variants before settling on Shopify's delayed_job. They are pretty aggressive with their use of backgrounding.
Here's a blog post from last year that talks about their move to DJ.
Where I am now we rolled our own several years ago, but I'm taking some ideas from DJ to improve the handling.
I would recommend delayed-job as a dead simple solution if you don't expect any heavy load. Pros: easy to setup, easy to monitor, simple code, doesn't have any external dependencies. Previously we used ActiveMessaging (with ActiveMQ and stomp), but it was an overkill for our project, so we switched to delayed_job for its simplicity.
Anyway, if you need very mature and fast solution, ActiveMQ is a very good choice. If you don't want to spend too much time on maintaining full-scale message queueing solution you don't really need, delayed_job is a way to go. Here's a good article about Scribd experience with ActiveMQ.
Here are a few Ruby/Rails solutions, one or more of these may be a good fit depending on your needs:
http://xph.us/software/beanstalkd
http://rubyforge.org/forum/forum.php?forum_id=19781
http://backgroundrb.rubyforge.org
And, a hosted solution from Amazon which would make a great queue for sharing between Ruby/Rails and other components of a larger system:
http://aws.amazon.com/sqs
Hope this helps!
The Messaging Server you might want to go for is RabbitMQ. Erlang coolness, AMQP, good Ruby libs.
http://www.bestechvideos.com/2008/12/09/rabbitmq-an-open-source-messaging-broker-that-just-works
Rany Keddo gave a useful presentation about Starling + Workling at RailsConf Europe last year. He compared the different solutions available at the time.
Twitter's latest move away from Starling + Workling probably doesn't mean much to the regular rails app. They have a lot more issues of scale and probably have legacy issues with their datastore that prevents them from scaling past their current implementation.
Beanstalkd is a good alternative, simply because it runs as a daemon and has wrappers in other scripting languages (if you happen to change direction in the future or have different components written in other languages).
This link also has a good comparison of pros-cons of the various rails solutions available.
I use background_job which like delayed_job is a database-based queue.
A database makes an OK queue as long as you're not doing too much traffic in and out.
The reason I like background_job (and delayed_job) is that they do not require a separate process. They can run through cron. For me, this is of key importance because my messaging needs are even simpler than my meager sysadmin skills.
I'd like to know situations in which I should consider using a framework other than Rails.
Two things. First, Ruby is a relatively young language, and you may run into brick walls when trying to do slightly more esoteric things (like connect to non mainstream or older types of datasources). It also has poor GC, and no kernel threads, both of which are very important for a high performance platform. The main codebase (MRI) is quite hacky (lots of clever obfuscating programmer tricks like macros) and there are parts that are poorly written (gc and thread scheduling leap to mind). Again, it is a very young platform that got very popular very fast.
Secondly, while ruby the language and rails the ideas/paradigm are both phenomenal, ruby and rails the platforms are not. There is a hell of alot in both ruby and rails that is downright ugly, and deployment solutions are in the dark ages compared to what is considered normal for other platforms (php/asp/jsp).
Being accused of trolling here, so I will expound a bit. Due to the threading model, Rails cannot process requests concurrently unless you launch multiple full instances of your rails app. To do that you have two options, the relatively young and still under development passenger (mod_rails), or the tried and tested apache load balancer with multiple mongrel instances behind it.
Either way, the lack of the ability to just just spawn workers means you will want 5-10 full instances of your application running, which incurs a very large overhead (can easily be 300-500megs per app depending on your gems and how big your app is). Because of that, the infrastructure needed to serve rails is a hell of alot more complecated then most other things.
Now, that being said, the situation has been continuously getting better (I mean, passenger is usable now, it wasn't the last time I had to deal with deploying a rails app). I would be very surprised if rails doesn't catch up in the next few years.
Also, rubinius/jruby are doing things the right way, and are moving along at a great pace. I wouldn't be suprised if MRI gets dropped in the next few years in favor of one of those implementations for mainstream rails work.
Ruby on Rails isn't trying to be an end-all be-all web development framework. If you're going to build an application that is predominantly built using CRUD operations, you want to use a lot of AJAX, and you have total control over the database, then Ruby on Rails is one of a few excellent options. If you're doing something else, then there is a probably another framework that is a better match for your requirements.
edit: Matt amended his answer tastefully :) I've removed my own comments pointing out the things he's fixed.
Yes, Ruby definitely has some shortcomings. Green threads being a huge one. But as Matt has said, things are moving along in better directions.
The other posts are pretty much on the money. Decently simple CRUD apps are best suited for rails, though there are other frameworks you can try in Ruby that offer more flexibility.
Here's a great (and might I add objective) example of where not to use rails: Does the Rails ORM limit the ability to perform aggregations?
Wow, way to start a flamewar!
I'm going to start it off by saying that Rails will work for most apps. However, if you need to do a lot of async type work (like messaging between systems, such as getting a request, placing it in a queue and processing it in a different thread, or even on another machine), Rails is probably not your best choice. Ruby, at least at the current time, is not really strong on multithreaded code.
Let the insults fly!