As described here, I'm detecting that I've been forked by Phusion Passenger, and revive a background thread that is aggregating some data that will eventually get packaged and sent to a remote server after a set amount of time. But sometimes, before the thread wakes up from the sleep, the process disappears, and (according to my log messages, that report the PID when the thread wakes up), I never hear from it again. Any way to control or prevent this?
You shouldn't be creating threads within a Passenger hosted process. If Passenger doesn't think your process is busy servicing requests, it is free to shut it down without warning. Those background threads should be used only in the course of your request processing.
What you want is a background job processing facility like delayed_job to offload this.
Related
I have modified the SimpleChatServer example of EventMachine to work with a Rails App as a chat server. I initialize the chat server in a seperate thread as follows.
Thread.new {
puts "I entered a new thread"
EventMachine.run do
puts "I entered a new thread"
EventMachine.start_server("0.0.0.0", 3100, SimpleChatServer)
end
}
I have hosted my app on a VPS running apache and am using Phusion Passenger to serve the rails app. The chat server works perfect except for one problem: The server stops automatically after a few minutes. When I check the error log i find nothing related to the shutdown. An interesting thing that I have observed however is a strange behavior of the shutdown: During day time in my location (11 am - 5 pm) the chat server stops after a few minutes of starting (My timezone is 10 hours ahead of my server's timezone). However during nighttime at my side the server keeps on running without a shut down. This strange behavior is bogging down my mind.
My own assumption is that during my day time, the VPS has more load to handle and thus it kills the Chat Server thread. Can I somehow avoid that? Also I would love to know if there is any other reason for this strange behavior. Please help me with this
Addition: When I check my error log I see this.
"[ 2015-03-06 08:00:20.5859 25041/7f20f1439700 agents/HelperAgent/Main.cpp:722 ]: Disconnecting long-running connections for process 25069"
Here 25069 is the PID of my chat server. How can i avoid this? How can I instruct linux not to kill my process ever?
A long while back I found the solution to this in a thread on github. The process gets killed because passenger kills idle applications to save memory. In order to disable this and keep my process running, I needed to set max_pool_idle_time to 0 in my passenger configuration. Here is a link to the original thread: https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html#_configuring_phusion_passenger
I am trying to understand exactly how requests to a rails application get processed with Phusion Passenger. I have read through the Passenger docs (found here: http://www.modrails.com/documentation/Architectural%20overview.html#_phusion_passenger_architecture) and I understand how they maintain copies of the rails framework and your application code in memory so that every request to the application doesn't get bogged down by spinning up another instance of your application. What I don't understand is how these separate application instances share the native ruby process on my linux machine. I have been doing some research and here is what I think is happening:
One request hits the web server which dispatches Passenger to fulfill the request on one of Passenger's idle worker processes. Another request comes in almost simultaneously and is handled by yet another idle Passenger worker process.
At this point there are two requests being executed which are being managed by two different Passenger worker processes. Passenger creates a green thread on Linux's native Ruby thread for each worker process. Each green thread is executed using context-switching so that blocking operations on one Passenger worker process do not prevent that other worker process from being executed.
Am I on the right track?
Thanks for your help!
The application instances don't "share the native Ruby process". An application instance is a Ruby process (or a Node.js process, or a Python process, depending on what language your app is written in), and is also the same as a "Passenger worker process". It also has got nothing to do with threading.
Im confusing about where should I have a script polling an Aws Sqs inside a Rails application.
If I use a thread inside the web app probably it will use cpu cycles to listen this queue forever and then affecting performance.
And if I reserve a single heroku worker dyno it costs $34.50 per month. It makes sense to pay this price for it for a single queue poll? Or it's not the case to use a worker for it?
The script code:
What it does: Listen to converted pdfs. Gets the responde and creates the object into a postgres database.
queue = AWS::SQS::Queue.new(SQSADDR['my_queue'])
queue.poll do |msg|
...
id = received_message['document_id']
#document = Document.find(id)
#document.converted_at = Time.now
...
end
I need help!! Thanks
You have three basic options:
Do background work as part of a worker dyno. This is the easiest, most straightforward option because it's the thing that's most appropriate. Your web processes handle incoming HTTP requests, and your worker process handles the SQS messages. Done.
Do background work as part of your web dyno. This might mean spinning up another thread (and dealing with the issues that can cause in Rails), or it might mean forking a subprocess to do background processing. Whatever happens, bear in mind the 512 MB limit of RAM consumed by a dyno, and since I'm assuming you have only one web dyno, be aware that dyno idling means your app likely isn't running 24x7. Also, this option smells bad because it's generally against the spirit of the 12-factor app.
Do background work as one-off processes. Make e.g. a rake handle_sqs task that processes the queue and exits once it's empty. Heroku Scheduler is ideal: have it run once every 20 minutes or something. You'll pay for the one-off dyno for as long as it runs, but since that's only a few seconds if the queue is empty, it costs less than an always-on worker. Alternately, your web app could use the Heroku API to launch a one-off process, programmatically running the equivalent heroku run rake handle_sqs.
We have background (resque) jobs on heroku for our Ruby on Rails application.
When Heroku bounce the box as they did yesterday, our background jobs are lost.
Our background jobs run for about 2 - 6 hours.
Is there anyway to keep them running or restart them automatically after failure?
In a nut shell, no, there's no easy way (if by easy you mean transparent to you, the app developer).
The best way to handle such interruptions is to properly save the job state on receipt of the SIGTERM signal from the dyno manifold. Your worker dyno will be reconstituted in a different physical location where it can resume processing interrupted jobs.
While this involves extra development effort, the result is a more robust and resilient app (no matter what the underlying platform).
I'm running an application that kicks off a Rufus Scheduler process in an initializer. The application is running with Passenger in production and I've noticed a couple weird behaviors:
First, in order to restart the server and make sure the initializer gets run, you have to both touch tmp/restart.txt and load the app in a browser. At that point, the initializer fires. The horrible thing is that if you only do the touch, the processes scheduled by Rufus get reset and aren't rescheduled until you load the app in a browser.
This alone I can deal with. But this leads to the second problem: I'll notice that the scheduled process hasn't run, so I load a page and suddenly the log file is telling me that it's running the initializers as if I'd rebooted. So, at some point, Passenger is randomly rebooting as if I'd touched tmp/restart.txt and wiping out my scheduled processes.
I have an incredibly poor understanding of Passenger and Rails's integration, so I don't know whether this occasional rebooting is aberrant or all part of the architecture. Can anyone offer any wisdom on this situation?
What you describe is the way Passenger works. It spawns new instances of the application when traffic warrants them, and shuts them down after periods of inactivity to free resources.
You should read the Passenger documentation, particularly the Resource Control and Optimization section. There are settings which can prevent the application from being shut down by Passenger, if that is what you want.
Using the PassengerPoolIdleTime setting, you could keep at least one process running, but you'll almost certainly want Passenger to start up other instances of the app as necessary. This thread on the Rufus Scheduler Google Group mentions using lock files to prevent more than one process from starting the scheduler, that may be useful to you.