I have a task that creates new activerecord records which I have recently moved to a background task using delayed_job and foreman, as recommended by Heroku
Sometimes this works fine, but sometimes it causes the Rails app in the browser to stop responding.
At this point I can see from the database that all the delayed jobs have completed, and that all the new records have been created.
However when I kill the processes I get a futher 11,200 lines of terminal output. This mostly consists of the execution by the web process of two methods on the model, both of which involve calls to the database:
validate :hit_database_to_see_if_model_exists?
before_save :get_rows_from_database_and_perform_calculation
There are also a number of INSERT statements which I'm sure have already hit the database because the number of records does not change before/after killing the processes
Here is my Procfile:
web: bundle exec rails server thin -p $PORT -e $RACK_ENV
worker: bundle exec rake jobs:work
So it feels like I am getting a 'stack overflow' (woop). Can you shed any light on:
What generally is going on?
Where in Rails this 'stack overflow' is taking place
Whether these things are actually happening after I hit 'Ctrl + C' or just being printed to the terminal at that point?
What might be causing this?
How I could debug / fix it?
UPDATE
It looks like there are certain tasks that are being assigned to the web process by the background task, but not being executed until the browser is 'prodded.' In some circumstances they all execute, but if there are too many the app falls over. Any idea on what might be causing this?
UPDATE
I tried running the web and worker processes in two separate windows.
In this scenario I have been unable to replicate the problem of the browser hanging, and in each case the worker process completed properly.
However I did make the interesting observation that if I don't touch the browser then no output appears in the web window. However if I do touch the browser then thousands of lines of what the worker process is doing at that moment appear in the web window.
Is this normal? Does this shed any light on what the problem might be?
Update
At the bottom of the terminal output after I kill the processes it says "Killed: 9"
07:45:21 system | sending SIGKILL to all processes
Killed: 9
What exactly does this 9 refer to? Is this unusual?
Update
I am using:
delayed_job 3.0.4
delayed_job_active_record 0.3.3
delayed_job_web 1.1.2
foreman 0.60.2
RESOLUTION
Thanks to #Justin's answer below (and this related question). It seems that Ruby buffers stdout by default, and that this buffer was overflowing, causing the app to stop responding. I added $stdout.sync = true at the top of config/environments/development.rb, and the problem appears to have gone away.
This is only a partial answer, but it might help you with debugging.
Rails buffers logging by default, and flushes it after every web request. One option is to simply replace the logger with a simpler logger
Rails.logger = Logger.new(STDOUT)
You can also configure the buffered logger to flush more often
Rails.logger.auto_flushing = (Rails.env.development? || Rails.env.test?)
You also have to be careful about STDOUT. In my current project I have stdout flushing enabled in both config.ru and my background bootup (I'm using sidekiq, so the boot process may be a little different)
STDOUT.sync = true
As to your larger problem,
I'm surprised that the rails process is running background tasks. Is there an option to disable that, at least for experimentation?
Then there are the standard debugging tools - all the database calls on save worry me, so I'd try various combinations of disabling them to see if anything improves. Especially with two of them - for instance, if your before_save hook changes a value in the model, it might be trigger a validation; if that resets the before_save hooks, you'd have a loop.
Related
I am currently using the delayed_job_active_record gem to run some scheduled tasks on a long run basis. The processes run in the background on a separate worker dyno on heroku and rarely go wrong but in some cases I would like to be able to stop a process mid run. I have been running the processes locally and because of the setup I have, the scheduled tasks only kick off the process which is essentially a very long loop.
Using
bin/delayed_job stop
only stops the jobs but since the process has started, it doesn't top this.
Because of this, I can't seem to stop the process once it has got going without restarting the entire dyno. This seems a bit excessive but is my only option at the moment.
Any help is greatly appreciated
I don't think there's anyway to interrupt it without essentially killing the process like you are doing. I would usually delete the job record in the database and then terminate the worker running it so it doesn't just retry the job (if you've got retries enabled for that job).
Another option... Since you know it's long running and, I imagine, has multiple steps... Modularize the operation and/or add periodic checks for a 'cancelled' flag you put somewhere in the model(s). If you detect the cancelled request, you can then give up and do any cleanup needed. This is probably preferred anyway so you can manage what happens when it's aborted more explicitly.
Can i run delayed_job or similar schedule frameworks inside of the web server eg. thin or unicorn?
If yes how do i start it? (code example would be very cool!)
The reason is that i want to save money during my application is just in a build-up phase and it is hosted on heroku.
Officially
No, there is no supported way to run delayed_jobs asynchronously within the web framework. From the documentation on running jobs, it looks like the only supported way to run a job is to run a rake task or the delayed job script. Also, it seems conceptually wrong to bend a Rack server, which was designed to handle incoming client requests, to support pulling tasks off of some queue somewhere.
The Kludge
That said, I understand that saving money sometimes trumps being conceptually perfect. Take a look at these rake tasks. My kludge is to create a special endpoint in your Rails server that you hit periodically from some remote location. Inside this endpoint, instantiate a Delayed::Worker and call .start on it with the exit_on_complete option. This way, you won't need a new dyno or command.
Be warned, it's kind of a kludgy solution and it will tie up one of your rails processes until all delayed jobs are complete. That means unless you have other rails processes, all incoming requests will block until this queue request is finished. Unicorn provides facilities to spawn worker processes. Whether or not this solution will work will also depend on your jobs and how long they take to run and your application's delay tolerances.
Edit
With the spawn gem, you can wrap your instantiation of the Delayed::Worker with a spawn block, which will cause your jobs to be run in a separate process. This means your rails process will be available to serve web requests immediately instead of blocking while delayed jobs are run. However, the spawn gem has some dependencies on ActiveRecord and I do not know what DB/ORM you are using.
Here is some example code, because it's becoming a bit hazy:
class JobsController < ApplicationController
def run
spawn do
#options = {} # youll have to get these from that rake file
Delayed::Worker.new(#options.merge(exit_on_complete: true)).start
end
end
end
Here's a link to a similar question:
Is it feasible to run multiple processeses on a Heroku dyno?
Bear in mind, as the post says, if you're only using one web dyno, it will be shut down if there's no traffic going to it.
In a similar vein, you might look into:
http://blog.codeship.io/2012/05/06/Unicorn-on-Heroku.html
To save on the need for multiple web dynos whilst you're building your app (although it's still subject to the above shutdown issue).
I would suggest you might look at running on a VPS directly, rather than Heroku (check out the railscast):
http://railscasts.com/episodes/337-capistrano-recipes
Once set up, it's pretty easy to deploy to. Heroku cuts out the devops part for you.
You can run it inside a separate worker of Unicorn, so it shares memory with the master process and get restarted together with the app.
See https://gist.github.com/brauliobo/11298486
With a Rails stack, how can I create a background process that handles events by spawning threads that are worked in real time?
The workers on Heroku pick up jobs every 5 seconds. I need real time. Ideally I'd like to get this working on Heroku, but if I need to, I will move away from it.
This has a long list of background workers: Background Job Manager for Rails 3 but it is not clear if your question heroku specific or not
I think you are looking for something like "run_later" which instead of queueing a job actually returns the request and runs a block in a separate process.
Here is a link to the Rails 3+ version, you can follow the fork network to find many other implementations:
https://github.com/Zelnox/run_later
(I don't use Heroku so I don't know if it runs on it)
Heroku runs rake jobs:work, so you can replace that with your own rake task, either running delayed_job with a shorter than 5 second timeout, or just performing your own task. Probably a good idea to keep a sleep statement in there.
The new cedar stack will run anything you want, so it might be worth checking that out.
With regards to run_later, or spawning from the current request, this does work but if the background process doesn't complete within the 30 second request timeout then heroku will kill it.
I think you need delay_job. please checkout this gem
We have a rails app running on passenger and we background process some tasks using a combination of RabbitMQ and Workling. The workling's worker process is started using the script/workling_client command. There is always only one worker process started, and the script/workling_client has a :multiple => false options, thus allowing only one instance. But sometimes, under mysterious circumstances which I haven't been able to track down, more worklings spawn up. If I let the system run for some time, more and more worklings appear. I'm not sure if these rogue worklings cause any problems, but it is still unsettling not to know why is it happening. We are using Monit to monitor the workling process. So if it dies, it will spawn it up again. But this still does not explain how come there are suddenly more than one of them.
So my question is: does anyone know what can be cause of this and how to make it stop? Is it possible that workling sometimes dies by itself, without deleting it's pid file? Could there be something wrong with the Daemons gem workling_client is build upon?
Not an answer - I have the same problems running RabbitMQ + Workling.
I'm using God to monitor the single workling process as well (:multiple => false)...
I found that the multiple worklings were eating up huge amounts of memory & causing serious resource usage, so it's important that I find a solution for this.
You might find this message thread helpful: http://groups.google.com/group/rubyonrails-talk/browse_thread/thread/ed8edd0368066292/5b17d91cc85c3ada?show_docid=5b17d91cc85c3ada&pli=1
I'm running an application that kicks off a Rufus Scheduler process in an initializer. The application is running with Passenger in production and I've noticed a couple weird behaviors:
First, in order to restart the server and make sure the initializer gets run, you have to both touch tmp/restart.txt and load the app in a browser. At that point, the initializer fires. The horrible thing is that if you only do the touch, the processes scheduled by Rufus get reset and aren't rescheduled until you load the app in a browser.
This alone I can deal with. But this leads to the second problem: I'll notice that the scheduled process hasn't run, so I load a page and suddenly the log file is telling me that it's running the initializers as if I'd rebooted. So, at some point, Passenger is randomly rebooting as if I'd touched tmp/restart.txt and wiping out my scheduled processes.
I have an incredibly poor understanding of Passenger and Rails's integration, so I don't know whether this occasional rebooting is aberrant or all part of the architecture. Can anyone offer any wisdom on this situation?
What you describe is the way Passenger works. It spawns new instances of the application when traffic warrants them, and shuts them down after periods of inactivity to free resources.
You should read the Passenger documentation, particularly the Resource Control and Optimization section. There are settings which can prevent the application from being shut down by Passenger, if that is what you want.
Using the PassengerPoolIdleTime setting, you could keep at least one process running, but you'll almost certainly want Passenger to start up other instances of the app as necessary. This thread on the Rufus Scheduler Google Group mentions using lock files to prevent more than one process from starting the scheduler, that may be useful to you.