I am running a Ruby on Rails application, presently using delayed_job to send emails. But I want to switch to RabbitMQ messaging queue. I am not able to find any useful resource to get started with it. I read many RabbitMQ docs and what not. Please get me some heads up to accomplish this task.
RabbitMQ integrations for Rails heavily depend on eventmachine and callback-based code. This will make your code really complex, and you will need to make all your code aware of the event loop.
For instance, you can't use the same code if you deploy behind thin (which already has a running event loop) or behind unicorn (in which you have to instantiate a reactor and manage its lifecycle).
On the other side, you can abstract this to an actual job queue (Resque is already much faster than DJ, Sidekiq smokes its pants off, and Beanstalkd/Stalker is a very good contender) which is probably going to be compatible to the Rails.queue abstraction in rails 4.
In Rails 4, you have also the option to configure all ActionMailer to be async by default (thereby delegating to any configured job queue). http://reefpoints.dockyard.com/ruby/2012/06/26/rails-4-sneak-peek-async-actionmailer.html
If you are looking to kill 2 birds with one stone, and your sending an email needs are based on active record lifecycle events (which they almost always are), you should check out my gem.
https://github.com/jasonayre/active_pubsub
It was built on the bunny gem and works much like active record observers, except you run the subscribers in a separate process. (this is actually just a bonus functionality, its intended largely to allow multiple rails apps to subscribe to each others events) -- I.E.
#user model
class User < ::ActiveRecord::Base
include ::ActivePubsub::Publishable
publish_as "myapp"
end
#subscriber
class PostSubscriber < ::ActivePubsub::Subscriber
observes "cms"
as "aggregator"
on :created do |user_hash|
user = User.find(user_hash[:id])
#or whatever
WelcomeMailer.deliver(user)
end
end
== async delivery + keeps your controllers slimmer.
If you are looking for a great Ruby background jobs processing library to replace delayed_job you should give Sidekiq a try. There even is a recent Railscast about it here.
You can get an overview on Background Jobs Gems on Knight.io (I started this site).
Related
Given an existing rails app with background processes managed by Sidekiq.
How can I migrate calls such as:
Model.delay.some_action()
From Sidekiq's syntax to Active Job's back-end agnostic syntax?
Update:
#craig.karminsky has pointed out the well-written Sidekiq to Active Job Wiki Page. This page addresses mailers.
Old Sidekiq syntax:
MyMailer.delay.send_message()
Active Job syntax:
MyMailer.send_message().deliver_later
That's a good solution for mailers, but how can I migrate non-mailer calls to .delay such as:
NotAMailer.delay.do_something()
Mike Perham, the creator of Sidekiq, has a very good Wiki page up for just such a situation.
I've made a gem for that on top of activejob: https://github.com/cristianbica/activejob-perform_later/. However the gem isn't very tested and this pattern is not a good one. Having code through your application that gets executed delay it will make your app hard to maintain and trigger all kind of bugs
I'm creating a new rails APP that's works with the same postgres DB as my API.
The API and the APP has a couples of common models.
Exemple :
The API and the APP has a Connection model
The APP will create a simple connection in the DB and the API will do the rest (like handler using the connection for many things )
So I wanted to use Hair Trigger that allowed to trigger a DB insert, or update and like it to a DB behavior that's will be execute.
But I would like to execute a method from one of my helpers.
like that :
class Connection < ActiveRecord::Base
trigger.after(:insert) do
SomeObject.method()
end
end
I can't use after_create because the creation action is not done on the same server (but same DB)
any ideas ?
Thanks !
The reason why Hair Trigger only supports SQL is that it's purpose is to help you maintain database triggers (https://www.postgresql.org/docs/9.6/static/sql-createtrigger.html).
Database triggers are decoupled from your application and will only be able to trigger SQL inside your database. It won't actually be able to trigger any code inside of your application.
A solution to your problem would be to use events. If you ever worked with Javascript you might have come across Event and EventListener before. The purpose of events is to decouple your code. One part of your application sends an event when an action has been taken e.g. Something was inserted into the database. Other parts, as many as you like, of your application can then subscribe to these events and take the actions necessary.
Since you are working with Rails I would suggest you to take a closer look at the Wisper gem (github.com/krisleech/wisper). If your App and API; like you are describing, lives in different processes (or servers), you should look into using a background worker to handle the events asynchronously. A common approach to this is to use Sidekiq Wisper (github.com/krisleech/wisper-sidekiq) that enables you to trigger the events async using Sidekiq as your background worker.
For a good introduction to the subject, here's a guide to get you started (I'm not the author): http://www.g9labs.com/2016/06/23/rails-pub-slash-sub-with-wisper-and-sidekiq/
There might be easier ways to solve the issue you are facing but you will need to post some more details on the use case for us to give you some better pointers.
I need to generate xml from a model and send it to a web service on model save.
I'm sure this is a common case and should be straight forward. Just create a job on after_save callback that generates the xml and sends it to the endpoint.
Since I'm new to Ruby on Rails I'm not to sure how to handle this though. My questions are more about code organization. It's not unlikely that this api connection will be discontinued in the future so I need a clean modular way to get rid of it. Would it be best practice/convention to put this in a separate gem? Can gems actually add jobs to an existing rails queue? Can gems create migrations on install? I'll probably need to add a model to keep track of the api sync. How about dropping a table on gem uninstall? Or should I not use a gem for this at all?
I realize these are broad and basic Ruby on Rails questions but I'm kind of drowning in documentation. I'm just hoping for some examples and/or advice and maybe some pointers to relevant documentation. Thanks.
Gem installs/uninstalls are unrelated to apps, they live on different level and do not khow anything about your app code, db and so on unless they are loaded.
Gems for rails can provide rake tasks and/or generators, for example you can look into devise gem structure on how it does this.
But i'd advise against moving code to a gem before you know you have to, like for example when you need to reuse it in different project.
To reuse code inside single project - use mixins/concerns
In general:
don't make it a gem
it's an unnecessary world of pain, pretty much always,
never make anything a gem unless you intend to use it in the same way in 3+ applications
don't extract it into a concern either,
it doesn't seem very likely that you'll do the same operation on multiple models, code reuse seems to not be an issue here and you can actually reuse code more efficiently using service classes too
a lot of experienced Rails programmers regard this practice as concerning, forgive the pun. It seems this view is not shared by the Rails development team, but at least from my experience writing service classes seems like unnecessary complexity until your project grows enough and then you need to refactor a BUNCH of stuff and you realize you would have been better off ditching concerns from the start
use a service class instead and delegate the necessary methods to it from the model
this will leave you with a clean interface to extract later and will also allow you to use dependency injection if you need to mock your XML service for tests
don't tie API requests to model callbacks, there's usually just 2-3 places where you need to do something with the API and a bunch of other cases where that may not be the case, imagine:
tests,
or if you get a requirement to implement cache column,
or a "number of visits" column
or a gem like Paperclip that thought that it wanted to add something to the model but changed his mind and instead of that just touched updated_at
or any such trickery which will make you a grandiose API spammer and a sufferer of VERRRRY slow database updates
if you DO tie API requests to model callbacks,
then you better make sure that error handling is done properly and that timeouts etc don't rollback or delay your DB operation,
best way from my experience is to run these things through ActiveJob + one of the backends (though obviously not the :inline backend and ideally one of the backends which don't use your main database and allow asynchronous job submission - sidekiq comes to mind as a candidate)
In a rails web app, if I write messages to a queue like rabbitmq, how will clients be notified when a producer sends a message to the queue?
I'm guessing I have to create a seperate process that runs in the background to respond to messages correct? i.e. this code is outside of the scope of a web application.
If this is the case, is it possible to re-use the models/libs that are in the rails application already? do I have to copy this code in 2 places then?
It looks like your application requires what's usually called a background or worker process. This is a fairly common requirement for any moderately complex web application.
I'm guessing I have to create a seperate process that runs in the background to respond to messages correct?
Yes - you're right about this. Whilst it's perfectly possible to use threads to handle the background tasks (in your case, reading and processing messages from RabbitMQ), the standard and recommended route for a Rails application is to run a separate background process.
If this is the case, is it possible to re-use the models/libs that are in the rails application already?
Absolutely. The simplest possible way to get this working is by using Rails' built in runner command.
Another option is to create a ruby script which loads up your Rails application. For example, you could create the file my_script.rb in the root of your project, which might look something like this:
# Load my application:
require File.join(File.dirname(__FILE__), 'config/environment.rb')
# Now you can access your Rails environment as normal:
MyModel.all.each { |x| x.do_something }
If your needs become more complex, or you find that you need to run more than one background process to keep up with the volume of data you need to process, you might want to look at one of the many available libraries and frameworks which can help with this.
Once you've created your background process, you'll need a way to run it continuously when you deploy it to your production server. Whilst it's possible to use libraries like daemons, as suggested by ctcherry, I would recommend using a dedicated tool like upstart (if deploying to ubuntu) or runit. A good summary of the most popular options is available here.
You are correct, you do need a background process. And you can keep the code for that process in the lib folder of the Rails project if you like, I have done that before without issue, and it keeps related code together which is nice.
I used this library to create my long running process, it was quite simple:
http://daemons.rubyforge.org/
In order to re-use models from your rails application you can run a require on the config/environment.rb file to get everything loaded. (Set RAILS_ENV as an environment variable first to select the correct envrionement) From that point the script behaves as though you are inside a rails console session.
Firstly let me mention that I'm new to web-frameworks.
I have to write my first web-app for a Uni project. I spent two weeks learning Grails and Django. Started working with Rails yesterday and loved it. So I've decided to go with it and discard my work in the other frameworks.
About the app
It's supposed to be a Twitter app that utilizes Twitter's Streaming API to record tweets which match a set of specified filters. (I'm going to use the Tweetstream gem which takes care of connecting to Twitter and capturing matching tweets).
The app's web interface should have the following functionality -
Creating new requests The user inputs a set of filter parameters (keywords to track) & URL/username/password of an existing PostgreSQL or MySQL database.
When a request is created, the web-app spawns a background ruby process. This process connects to Twitter via the Tweetstream gem. It also connects to the database specified by the user to stores received tweets.
View/terminate of existing requests
The user should be able to see a list of requests that are running as background processes by visiting a URL such as /listRequests.
See further details about a process/terminate the process
The user should be able to go to URL such as /requests/1/detail to view some details (e.g how long request has been running, number of tweets captured, etc). The user should also be able to terminate the process.
My inexperience is showing as I'm unable to comprehend -
what my models should be (maybe Request should be a model. Tweet doesn't need to be a model as it's not being stored locally)
how I'm going to connect to remote databases.
how I can create background processes (backgroundrb??) and associate them with request objects so that I can terminate then when the user asks.
At the end of the day, I've got to build this myself, so I'm not asking for you to design this for me. But some pointers in the right direction would be extremely helpful and appreciated!
Thanks!
Hmm.
Since the web app is just a thin wrapper around the heavy-lifting processes, it might be more appropriate to just use something like Sinatra here. Rails is a big framework that pulls in lots of stuff that you won't need for this project, even though it will work.
Does the "background process" requirement here strictly mean a separate process, or does it just mean concurrency? TweetStream uses the EventMachine gem to handle updates as they come, which uses a separate thread for each connection. It would be quite possible to spawn the TweetStream clients from a simple Sinatra web app, keep them in a big array, have them all run concurrently with no trouble, and simply run stop on a given client when you want it to stop. No need for a database or anything.
I'm not sure exactly what your prof is looking for you to do here, but MVC doesn't really fit. It's better to work with the requirements than to mush it into a design pattern that doesn't match it :/
Even so, I <3 Rails! Definitely get on that when you're working primarily with objects being represented in a database :)
Quite a project. Most of what will be challenging is not related to rails itself, but rather the integration with background processes. backgroundrb is a bit out of fashion. The last commit on the main github project is over a year ago, so it's likely not up to snuff for Rails 3. Search around and evaluate your options. Resque is popular, but I'm not sure if your real-time needs match with its queue-based structure.
As for your app, I see only a single model, but don't call it request. That's a reserved name in rails. Perhaps a Search model, or something along that line.
Connecting to different databases is straight forward but will require direct configuration of your ActiveRecord class during operation rather than using database.yml.