What is the best way to get alerts of some kind (email, HTTP callback, etc) about failed jobs in DelayedJob?
I'm running in Heroku, so keeping the workers alive (god, monit, etc) is taken care of.
NewRelic's error tracking functionality apparently doesn't register fatal errors in Delayed::PerformableMethod, but I do get performance metrics about stuff that doesn't fail.
I use Exceptional (http://getexceptional.com).
I had to do a little custom integration, but it was really simple.
http://pinds.com/2010/11/24/tracking-errors-in-collectiveideas-delayedjob-using-exceptional/
Related
That, basically. I have a Rails 5 application but really more on a general level Im trying to find a way to get email alerts when my dynos reach certain memory usage threshold. Im using one web and one worker. Cant find anything.
Heroku has an 'Application metrics' page which can also alert you on some conditions. However there is no option for alerting for memory usage. We have asked Heroku support and this is their answer:
Yes, unfortunately there's not a built-in way to receive alerts about
memory usage. However, you could potentially set up your Papertrail
instance to alert you if your app emits an R14 or an R15. Keep in mind
that those are cases where it might be too late to take any corrective
action on memory as it is already alerting a disruption to the app.
For more granularity, you'll need to enable the log-runtime-metrics
lab so that your memory usage is printed to your application logs:
https://devcenter.heroku.com/articles/log-runtime-metrics.
You can also use a log drain to parse, chart, and setup alerts for
those errors. Librato is the most common tool I see customers using
for that kind of workflow: https://elements.heroku.com/addons/librato.
Relying on Papertrail is good enough for us for now, but we are still exploring other options. One option is to simply use the Rails app to report its own memory usage, for example through a system call like this pmap #{Process.pid} | tail -1. Another option is to use Monit (https://mmonit.com/) but it's not very easy to set it up and configure. (You will need a custom build pack to run this on Heroku.) Also Heroku supports some 3rd party addons for monitoring which can send alerts, like New Relic.
You can't in Heroku, but the best solution I've found so far is to use App Signal
They allow you to set up custom alerts for any metric they track, including host memory usage.
App Signal is a pretty solid APM in general too, so you can ditch New Relic, Scout, or whatever other tool you might be using as well.
I'm coming from a PHP environment (at least in terms of web dev) and into the beautiful world of Ruby, so I may have some dumb questions. I imagine there are some fundamentally different options available when not using PHP.
In PHP, we use memcache to store alerts we want to display in a bar along the top of the page. When something happens that generates an alert (such as a new blog post being made), a cron script that runs once every 5 minutes or so puts that information into memcache.
Now when a user visits the site, we look in memcache to find any alerts that they haven't already dismissed and we display them.
What I'm guessing I can do differently in Rails, is to by-pass the need for a cron script, and also the need to look in memcache on every request, by using a Singleton and a polling process running in a separate thread to copy from memcache to this singleton. This would, in theory, be more optimized than checking memcache once-per-request and also encapsulate the polling logic into one place, rather than being split between a cron task and the lookup logic.
My question is: are there any caveats to having some sort of runloop in the background while a Rails app is running? I understand the implications of multithreading, from Objective-C/Java, but I'm asking specifically about the Rails (3) environment.
Basically something like:
class SiteAlertsMap < Hash
include Singleton
def initialize
super
begin_polling
end
# ... SNIP, any specific methods etc ...
private
def begin_polling
# Create some other Thread here, which polls at set intervals
end
end
This leads me into a similar question. We push (encrypted) tasks onto an SQS queue, for things related to e-commerce and for long-running background tasks. We don't use cron for this, but rather we have a worker daemon written in PHP, which runs in the background. Right now when we deploy, we have to shut down this worker and start it again from the new code-base. In Rails, could I somehow have this process start and stop with the rails server (unicorn) itself? I don't think that's something I'd running on the main process in a separate thread, since we often want to control it as a process by itself, but it would be nice if it just conveniently ran when the web application was running.
Threading for background processes in ruby would be a terrible mistake, especially since you're using a multi-process server. Using unicorn with say 4 worker processes would mean that you'd be polling from each of them, which is not what you want. Ruby doesn't really have real threads, it has green threads in 1.8 and a global interpreter lock in 1.9 IIRC. Many gems and libraries are also obnoxiously unthreadsafe.
Using memcache is still your best option and, if you have it set up correctly, you should only see it adding a millisecond or two to the request time. Another option which would give you the benefit of persisting these alerts while incurring minimal additional overhead would be to store these alerts in redis. This would better protect you against things like memcache crashing or server reboots.
For the background jobs you should use a similar approach to what you have now, but there are several off the shelf handlers for this like resque, delayed_job, and a few others. If you absolutely have to use SQS as the backend queue, you might be able to find some code to help you, but otherwise you could write it yourself. This still requires the other daemon to be rebooted whenever there is a code change. In practice this isn't a huge concern as best practices dictate using a deployment system like capistrano where a rule can easily be added to bounce the daemon on deploy. I use monit to watch the daemon process, so restarting it is as easy as telling monit to restart it.
In general, Ruby is not like Java/Objective-C when it comes to threads. It follows the more Unix-like model of process based isolation, but the community has come up with best practices and ways to make this less painful than in other languages. Ruby does require a bit more attention to setting up its stack as it is not as simple as enabling mod_php and copying some files around, but once the choices and architecture is understood, it is easier to reason about how your application works. The process model, in my opinion, is much better for web apps as it isolates code and state from the effects of other running operations. The isolation also makes the app easier to work with in a distributed system.
I am using rakismet to check for spam in comments.
Right now, I do it in a before_create callback and I am wondering in a production site, if this is the most efficient way of doing it or should this be done by a background job.
Can you share your experience in terms of how much delay does this add to the responsiveness of your production apps?
I have not used rakisment, but doing any pre-processing on an action will slow it down, and in the situation of your spam filter, it will slow down more and more as more spam indicators are included in the rakismet dictionary.
I would recommend a two step process:
In the before_create, do a minimal spam check to catch very obvious ones. You can search for words ("viagra", "cialis", "debt", etc), as well as check that the submitter isn't submitting many comments very fast. This will be fairly quick, and not slow your app down too much.
In a Delayed Job (One of the more well known background processing libraries for Ruby), run your rakismet checks. These can delete/flag the comments after the fact.
This solution limits blatant spam immediately, and will eventually leverage the capabilities of rakismet to clean up the comments entirely, without causing too much strain or slow down to the system.
One benefit of this approach is that it is extremely easy to scale your Delayed Job processes, but just starting more workers on the same (or different) server(s). This means that your main app won't crawl, as the heavy lifting has been offloaded to multiple instances of the worker process.
Curious, what are you folks doing in as far as automating your unit tests with ruby on rails? Do you create a script that run a rake job in cron and have it mail you results? a pre-commit hook in git? just manual invokation? I understand tests completely, but wondering what are best practices to catch errors before they happen. Let's take for granted that the tests themselves are flawless and work as they should. What's the next step to make sure they reach you with the potentially detrimental results at the right time?
Not sure about what exactly do you want to hear, but there are couple of levels of automated codebase control:
While working on a feature, you can use something like autotest to get an instant feedback on what's working and what's not.
To make sure that your commits really don't break anything use continuous integration server like cruisecontrolrb or Integrity (you can bind these to post-commit hooks in your SCM system).
Use some kind of exception notification system to catch all the unexpected errors that might pop up in production.
To get some more general view of what happened (what was user doing when the exception occured) you can use something like Rackamole.
Hope that helps.
If you are developing with a team, the best practice is to set up a continuous integration server. To start, you can run this on any developers machine. But in general its nice to have a dedicated box so that its always up, is fast, and doesn't disturb a developer. You can usually start out with someone's old desktop, but at some point you may want it to be one of the faster machines so that you get immediate response from tests.
I've used cruise control, bamboo and teamcity and they all work fine. In general the less you pay, the more time you'll spend setting it up. I got lucky and did a full bamboo set up in less than an hour (once)-- expect to spend at least a couple hours the first time through.
Most of these tools will notify you in some way. The baseline is an email, but many offer IM, IRC, RSS, SMS (among others).
One of the articles I really enjoyed reading recently was Quality Control by Last.FM. In the spirit of this article, I was wondering if anyone else had favorite monitoring setups for web type applications. Or maybe if you don't believe in Log Monitoring, why?
I'm looking for a mix of opinion slash experience here I guess.
We get a bunch of email/pager alerts from an older host/app/network monitoring environment that get gradually more abusive depending on severity of the problem/time taken to respond. Fortunately we all have thick skins and very broad senses of humour. :)
We use log4net, and normally write both to log files and the database. However, when we've been tracking down a particularly difficult problem, we've enabled the email appender, so that critical log messages went straight to a developer's email account. This allowed us to figure out what was happening more immediately.
In addition, our infrastructure team has several tools they use to monitor system uptime, event logs, etc., to give them early warning when something is about to go down. We've also helped them implement custom monitoring scripts that test specific functionality of our code.