Client's service rulebook states we should log system state on startup and process termination. When in the boot process do I have access to the logger? How do I know when the app is loaded? We're using Puma if that helps.
Related
Phusion Passenger version: 5.0.29
I've read the documentation about Passenger's OOB (Out-of-Band) feature and would like to use it to make a decision out-of-band on whether a process should exit or not. If the process has to exit, then at the end of the OOB work, the process calls raise SystemExit
We've managed to get this working where the process exits and then Passenger spins up a new process later to handle the new incoming requests. But, we're seeing occasional 502s with the following lines in the passenger log.
[ 2019-03-27 22:25:13.3855 31726/7f78b6c02700 age/Cor/Con/InternalUtils.cpp:112 ]: [Client 1-10] Sending
502 response: application did not send a complete response
[ 2019-03-27 22:25:13.3859 31726/7f78b6201700 age/Cor/CoreMain.cpp:819 ]: Checking whether to disconnect
long-running connections for process 10334, application agent
App 16402 stdout:
App 16441 stdout:
App 16464 stdout:
[ 2019-03-27 22:28:05.0320 31726/7f78ba9ff700 age/Cor/App/Poo/AnalyticsCollection.cpp:102 ]: Process (pid=16365, group=agent) no longer exists! Detaching it from the pool.
Is the above behavior due to a race condition between the request handler forwarding the request to the process and the process exiting? Is Passenger designed to handle this scenario? Any workaround / solution for this problem?
Thank you!
Looks like we need to run "passenger-config detach-process [PID]" for graceful termination.
Improved process termination mechanism
If you wish to terminate a specific application process -- maybe because it's misbehaving -- then you can do that simply by killing it with the kill command. Unfortunately this method has a few drawbacks:
Any requests which the process was handling will be forcefully aborted, causing error responses to be returned to the clients.
For a short amount of time, new requests may be routed to that process. These requests will receive error responses.
In Passenger 5 we've introduced a new, graceful mechanism for terminating application processes: passenger-config detach-process. This command removes the process from the load balancing list, drains all existing requests, then terminates it cleanly.
You can learn more about this tool by running:
passenger-config detach-process --help
Source: https://blog.phusion.nl/2015/03/04/whats-new-in-passenger-5-part-2-better-logging-better-restarting-better-websockets-and-more/
I'd like to have two separate instances of PhantomJS constantly waiting on an "Ask" from the main application.
I'm confused how a worker difference from a web process on heroku.
In the Procfile, if I were to define web as NOT my application and started it from the worker process, what would happen?
Or, for example, puma web server starts out multiple threads. Does the web process acount for multiple threads or do you need a web or worker per every thread puma creates?
Thanks
As the title states, I'm trying to start a Ruby on Rails app through systemd socket activation. I've no prior experience with Ruby and the lack of quality documentation is frustrating.
Obtaining the listen socket from systemd is trivial enough, though I'm struggling to put that socket to use. Is there an app server, that accepts a pre-existing listen socket? Or do I have to write my own, possibly by implementing a Rack::Handler (going by what limited information I've been able to gather from the documentation)? In the latter case a minimal example implementation would be helpful.
I'm looking for a production-suitable, lightweight, least-effort solution, capable of serving an average of 2 concurrent users on embedded hardware.
Starting with Puma 3.0.0, socket activation is supported by its Rack module, which is used by rails server. This means that running bin/rails server puma from systemd with socket activation will now work out of the box (tested with Puma 3.9.1). One caveat is that you must start bin/rails inside your application root, not the globally installed version.
For more details on this setup and some examples, see my blogpost.
In absence of any alternatives, I'll likely end up using the Puma web server, which implements the systemd socket activation protocol.
EDIT: Puma does implement socket activation, but not in the Rack handler.
I arrived at my initial conclusion via grep -r LISTEN_FDS and only discovered my mistake once I actually tried it.
The application servers used by Ruby web applications that I know have the concept of worker processes. For example, Unicorn has this on the unicorn.rb configuration file, and for mongrel it is called servers, set usually on your mongrel_cluster.yml file.
My two questions about it:
1) Does every worker/server works as a web server and spam a processes/threads/fiber each time it receives a request, or it blocks when a new request is done if there is already other running?
2) Is this different from application server to application server? (Like unicorn, mongrel, thin, webrick...)
This is different from app server to app server.
Mongrel (at least as of a few years ago) would have several worker processes, and you would use something like Apache to load balance between the worker processes; each would listen on a different port. And each mongrel worker had its own queue of requests, so if it was busy when apache gave it a new request, the new request would go in the queue until that worker finished its request. Occasionally, we would see problems where a very long request (generating a report) would have other requests pile up behind it, even if other mongrel workers were much less busy.
Unicorn has a master process and just needs to listen on one port, or a unix socket, and uses only one request queue. That master process only assigns requests to worker processes as they become available, so the problem we had with Mongrel is much less of an issue. If one worker takes a really long time, it won't have requests backing up behind it specifically, it just won't be available to help with the master queue of requests until it finishes its report or whatever the big request is.
Webrick shouldn't even be considered, it's designed to run as just one worker in development, reloading everything all the time.
off the top of my head, so don't take this as "truth"
ruby (MRI) servers:
unicorn, passenger and mongrel all use 'workers' which are separate processes, all of these workers are started when you launch the master process and they persist until the master process exits. If you have 10 workers and they are all handling requests, then request 11 will be blocked waiting for one of them to complete.
webrick only runs a single process as far as I know, so request 2 would be blocked until request 1 finishes
thin: I believe it uses 'event I/O' to handle http, but is still a single process server
jruby servers:
trinidad, torquebox are multi-threaded and run on the JVM
see also puma: multi-threaded for use with jruby or rubinious
I think GitHub best explains unicorn in their (old, but valid) blog post https://github.com/blog/517-unicorn.
I think it puts backlog requests in a queue.
Sometimes the erlang application will coredown by out of memery or other reasons.
Can I config something to let it restart when it coredown?
erlang support option --heart for monitoring if erlang process still alive. In case of crashing or hanging erlang process, heartbeat part reboots the whole server (it assumes your erlang application starts at booting of your server). You can read all details here:
http://www.erlang.org/doc/man/heart.html