What message queues are people using for their Rails apps and what was the driving force behind the decision to choose it. Does the latest Twitter publicity over their in house queue Starling falling down affect any existing design decisions.
I am working on an app that will need a message queue to process some background tasks, I haven't done much of this, and most of the stuff I have seen in the past has been about Starling and Workling, and to be honest the application is not very big and this solution would probably suffice, but I'd love to get experience integrating the best solution possible as I'm sure I will integrate one into a bigger app at some point.
What message queues would you suggest for a Rails app???
EDIT: Thanks for the suggestions, I'm going to look at a few of them this weekend.
EDIT Again: I've had a look around and a little overwhelmed for choice. I am however going to go about integrating RabbitMQ with Workling into the app I am building, then if I ever need some knowledge about a fast queue then I will have this and know whether or not it fits my needs.
EDIT: Finding more and more that DJ suits me just fine, if I ever "outgrow" it on a site I'd say that Resque is where I would head.
EDIT: (Dec 2014) So it's been a long time since I asked this, but I see it still gets some views or some votes, so I figured I'd update it on my approach now when it comes to my choice of background workers.
In my opinion currently the best way to run background jobs in Ruby is using Sidekiq. A lot of people have really lauded Sidekiq for it's threaded workers rather than process per worker which can use significantly less memory than the likes of Resque, which I was using before Sidekiq. This is good but for me this wasn't the killer feature. By using Sidetiq with Sidekiq, the scheduling of jobs is so trivial that I switched over and have never looked back from it, by far the easiest scheduling of jobs that I have used and has made Sidekiq a breeze to use.
As an update -- GitHub have moved to Resque on Redis instead of Delayed job. However they still recommend delayed_job for smaller setups:
https://github.com/resque/resque
Chris Wanstrath from github was at the SF Ruby meetup recently, talking about their queue. They tried Starling, beanstalk, and some other variants before settling on Shopify's delayed_job. They are pretty aggressive with their use of backgrounding.
Here's a blog post from last year that talks about their move to DJ.
Where I am now we rolled our own several years ago, but I'm taking some ideas from DJ to improve the handling.
I would recommend delayed-job as a dead simple solution if you don't expect any heavy load. Pros: easy to setup, easy to monitor, simple code, doesn't have any external dependencies. Previously we used ActiveMessaging (with ActiveMQ and stomp), but it was an overkill for our project, so we switched to delayed_job for its simplicity.
Anyway, if you need very mature and fast solution, ActiveMQ is a very good choice. If you don't want to spend too much time on maintaining full-scale message queueing solution you don't really need, delayed_job is a way to go. Here's a good article about Scribd experience with ActiveMQ.
Here are a few Ruby/Rails solutions, one or more of these may be a good fit depending on your needs:
http://xph.us/software/beanstalkd
http://rubyforge.org/forum/forum.php?forum_id=19781
http://backgroundrb.rubyforge.org
And, a hosted solution from Amazon which would make a great queue for sharing between Ruby/Rails and other components of a larger system:
http://aws.amazon.com/sqs
Hope this helps!
The Messaging Server you might want to go for is RabbitMQ. Erlang coolness, AMQP, good Ruby libs.
http://www.bestechvideos.com/2008/12/09/rabbitmq-an-open-source-messaging-broker-that-just-works
Rany Keddo gave a useful presentation about Starling + Workling at RailsConf Europe last year. He compared the different solutions available at the time.
Twitter's latest move away from Starling + Workling probably doesn't mean much to the regular rails app. They have a lot more issues of scale and probably have legacy issues with their datastore that prevents them from scaling past their current implementation.
Beanstalkd is a good alternative, simply because it runs as a daemon and has wrappers in other scripting languages (if you happen to change direction in the future or have different components written in other languages).
This link also has a good comparison of pros-cons of the various rails solutions available.
I use background_job which like delayed_job is a database-based queue.
A database makes an OK queue as long as you're not doing too much traffic in and out.
The reason I like background_job (and delayed_job) is that they do not require a separate process. They can run through cron. For me, this is of key importance because my messaging needs are even simpler than my meager sysadmin skills.
Related
For a mostly static website without a lot of stuff going on in the back end, how many page views can the free version of heroku handle (ruby on rails). Can it handle 100 a month, 1,000, 10,000, etc. When shoudl I consider upgrading it and are there better services? Thanks in advance!
Unfortunately, the only real answer to this question is "It depends". It depends on how many database calls you are making, whether or not those database calls are optimized, whether you have any form of caching in place, etc.
However, I am currently in a similar situation as you and just launched an early stage startup on Heroku. I highly recommend installing New Relic (don't worry, there's a free tier) to keep an eye on site usage/statistics.
Another (free!) addon I installed was Papertrail, which allows me to keep a close eye on the Heroku logs, and trigger alerts if there are too many errors (which could mean it's time to scale-up).
On the other side, you can test your Heroku server by sending load to it using the Apache Bend tool. See this answer for a good explanation.
Lastly, I would take a look at this question, which provides many good references on Heroku performance.
Good luck!
Heroku is free for up to 5 apps. Each app has access to 750 dyno-hours per month. That should be more then enough for a mostly static site.
I get about 4k-5k visitors per month on a Ruby app, but with a lot of caching.
We need CI on Heroku for our RoR app.
We want to use an add-on. I don't mind paying a few bucks to avoid complexity / distraction.
I see its basically Railsonfire vs. Tddium
What are the factors that we should consider in choosing to use one vs. the other?
A similar question was asked at the link below, but it appears that this was before the add-ons above were created.
Proper continuous integration and continuous deployment with Git and Heroku
Another option is CircleCi, which makes it incredibly easy, though it isn't yet in the Heroku add-on marketplace.
I personally believe speed is the major consideration. The speed at which your team moves is tied to how quickly you can get notifications, and a platform optimized for speed is going to make a major difference.
In addition, on slow platforms, tests may fail randomly (for example, Cucumber might fail if a button doesn't appear in 3 seconds, which may not happen soon enough on a slow platform).
I'm a founder of CircleCi, and we make an incredibly fast CI solution (faster than both services you mentioned). It's really easy to set up (one-click), though it's not in the Heroku marketplace (yet).
One consideration I'd give you to… uh… consider is support. I don't know anything about the Tddium team, but I do know that a couple months ago I tried Rails On Fire was blown away by the support. I asked some crazy questions (I do crazy things) and Florian responded in email and on Twitter right away with both help and suggestions.
Then, when I decided not to use the service because it couldn't support one edge-case feature, he implemented the feature! I hadn't been using it for a couple weeks and he wrote back to me personally to say "Hey, we finally got that feature you thought would be helpful."
I've used Hudson/Jenkins for years and know what you mean about complexity. But I also think that having someone on the other end is pretty key, and the RailsOnFire team, for now at least, do that quite well.
I have some (I think) really great ideas for an online strategy game similar to Travian. There's some content that I haven't yet figured out and some other challenges that I don't know of yet.
This is quite a big project and perhaps too heavy for one person that isn't a skilled web developer (yet). I'd still like to give it a shot, but I'm having trouble choosing a platform. The world "scales" has been thrown around a lot lately and I've seen Ruby on Rails being bashed because it doesn't scale well, so I've come here to get some answers.
I like Ruby on Rails, both Ruby and Rails. I'm certainly no expert at it but I love working with it. I have also worked with Python + Django before and also with PHP (which I am not fond of.)
Ideally the game would have, let's say, 7000 players per server, presumably a lot of data to be processed per second. Would RoR still be a viable platform?
I'm sorry if this question is vague, I guess I'm looking for a "RoR is fine, go at it!" kind of answer. Anything you might want to add is fine.
Thanks!
So if I were you, I would be looking into non-blocking servers like node.js, just because they are MUCH more suited to keeping many connections open for long periods of time, which is what games need to do, compared to traditional web servers.
That being said
There are 3 main things to worry about when you are scaling a web app; memory, execution speed, and io (hd and network) in that order.
For memory, things are much better then they used to be. Phusion Passenger uses copy on write to fork its workers, so the rails environment will get shared among all the workers on a given slice, which is pretty significant. There have also been huge improvements to the way ruby manages memory compared to "the dark times", if you are using 1.8.7 then you want to be using the patches that make up Ruby Enterprise Edition (the difference is like night and day). 1.9.x was pretty much a total rewrite of the runtime, so if you are using that the memory issues ruby had have already been addressed.
For execution speed, 1.8.7 is typically "fast enough" (at least after tuning garbage collection settings). 1.9.2 is actually around the same speed as python, which puts it on the faster side of interpreted languages. How important this point is completely depends on the nature of your application.
Last point is IO, which isn't really a concern of rails, but more your persistence strategy. Rubyists tend to love new things, so you will find first class support for things like redis and mongodb, with loads of people talking about using them and their wins/gotchas. I would look into mongo if I were you and see if the durability trade-offs are acceptable.
I was in java/.net before going to rails, and at the end of the day you are going to pay more for infrastructure, but the amount will be completely dwarfed by what you save in development time.
build it in Rails, host it on Heroku.com - job done. Almost infinite scaling that you don't have to worry about how it works (it just does) and it hosts a lot of highly trafficked Facebook apps so can more than handle it.
I think Ruby on Rails is a good choice for what you need. Actually, we recently created a platform for an online gaming tournament, where players and their gaming bots were playing a logic game.
We used Ruby on Rails and Sidekiq on the backend, ReactJS and WebSocket on the frontend. And it worked well for a quite massive number of players. Here is the tutorial based on what we learnt while building it: How to Write a Game Engine Using Ruby on Rails
As you said yourself, you already have the answer and you are only looking for encouraging words:). I am not RoR expert myself, but I don't think scalability is still such a great issue on this platform. I would advice you to do an architecture spike (XP terminology). Write a test with 7000 clients and method which would perform similar operations to what you intend to create. For example you might load files, render views or even just wait... The point is to test only the thing you are worried about. Good luck!
This is a bit of an impossible question to answer because in order to know whether rails is suitable for what you want to do we would need to a lot more about what you are trying to do. The best advice I can give in the absence of information is for you to check out the railslab scaling videos in order to work it out for yourself.
I'm working on a Rails 2.3.x app that needs to run a long, slow task asynchronously.
Delaying the request until processing has completed is a bit of a pain. We're happy to just send a "Okay, we've received that and we're working on it"-style response back to the client, and then just crunch on the work once the client's gone away.
So I've seen a few methods for doing this (such as forking or using cron), but it seems that task queuing systems might be a better idea.
It seems like the three main ones are:
BackgroundFu,
BeanstalkD, and
Sparrow
Sparrow looks particularly interesting due to it's support for Memcached, but I don't want to just pick one without making an informed decision. Does anyone have any insights regarding best practices? I'd love to hear about your experiences with the above, especially any major gotchas with this approach or the libraries mentioned. Perhaps there's an even better way to do this that I've missed completely?
Just so it's out there: Rails 3.x isn't an option right now, this needs to work for Rails 2.3.x.
Thanks!
I recommend delayed_job, specifically this fork and branch. I'm using it myself on rails 2.3.5 and it works well.
It has a rails 3 branch for when you do want to upgrade, it works with multiple backends, and it has an optional pretty syntax:
Notifier.delay.deliver_signup #user
By the way, just forking is a mistake as you'll find the fork doesn't have a good database connection.
I'd like to know situations in which I should consider using a framework other than Rails.
Two things. First, Ruby is a relatively young language, and you may run into brick walls when trying to do slightly more esoteric things (like connect to non mainstream or older types of datasources). It also has poor GC, and no kernel threads, both of which are very important for a high performance platform. The main codebase (MRI) is quite hacky (lots of clever obfuscating programmer tricks like macros) and there are parts that are poorly written (gc and thread scheduling leap to mind). Again, it is a very young platform that got very popular very fast.
Secondly, while ruby the language and rails the ideas/paradigm are both phenomenal, ruby and rails the platforms are not. There is a hell of alot in both ruby and rails that is downright ugly, and deployment solutions are in the dark ages compared to what is considered normal for other platforms (php/asp/jsp).
Being accused of trolling here, so I will expound a bit. Due to the threading model, Rails cannot process requests concurrently unless you launch multiple full instances of your rails app. To do that you have two options, the relatively young and still under development passenger (mod_rails), or the tried and tested apache load balancer with multiple mongrel instances behind it.
Either way, the lack of the ability to just just spawn workers means you will want 5-10 full instances of your application running, which incurs a very large overhead (can easily be 300-500megs per app depending on your gems and how big your app is). Because of that, the infrastructure needed to serve rails is a hell of alot more complecated then most other things.
Now, that being said, the situation has been continuously getting better (I mean, passenger is usable now, it wasn't the last time I had to deal with deploying a rails app). I would be very surprised if rails doesn't catch up in the next few years.
Also, rubinius/jruby are doing things the right way, and are moving along at a great pace. I wouldn't be suprised if MRI gets dropped in the next few years in favor of one of those implementations for mainstream rails work.
Ruby on Rails isn't trying to be an end-all be-all web development framework. If you're going to build an application that is predominantly built using CRUD operations, you want to use a lot of AJAX, and you have total control over the database, then Ruby on Rails is one of a few excellent options. If you're doing something else, then there is a probably another framework that is a better match for your requirements.
edit: Matt amended his answer tastefully :) I've removed my own comments pointing out the things he's fixed.
Yes, Ruby definitely has some shortcomings. Green threads being a huge one. But as Matt has said, things are moving along in better directions.
The other posts are pretty much on the money. Decently simple CRUD apps are best suited for rails, though there are other frameworks you can try in Ruby that offer more flexibility.
Here's a great (and might I add objective) example of where not to use rails: Does the Rails ORM limit the ability to perform aggregations?
Wow, way to start a flamewar!
I'm going to start it off by saying that Rails will work for most apps. However, if you need to do a lot of async type work (like messaging between systems, such as getting a request, placing it in a queue and processing it in a different thread, or even on another machine), Rails is probably not your best choice. Ruby, at least at the current time, is not really strong on multithreaded code.
Let the insults fly!