I have been working on creating a minimal Rails GraphQL docker container based on Alpine 3.7 running Passenger 5.2.0. Unfortunately, Passenger keeps crashing and I am unsure what the cause is.
This is the head of the Passenger log file
[ N 2018-02-06 16:40:25.3740 14/T1 age/Wat/WatchdogMain.cpp:1252 ]: Starting Passenger watchdog...
[ N 2018-02-06 16:40:25.4056 17/T1 age/Cor/CoreMain.cpp:1242 ]: Starting Passenger core...
[ N 2018-02-06 16:40:25.4059 17/T1 age/Cor/CoreMain.cpp:252 ]: Passenger core running in multi-application mode.
[ N 2018-02-06 16:40:25.4244 17/T1 age/Cor/CoreMain.cpp:977 ]: Passenger core online, PID 17
App 55 stdout:
[ pid=17, timestamp=1517935227 ] Process aborted! signo=SIGSEGV(11), reason=SI_KERNEL, si_addr=0x0, randomSeed=1517935225
And this is the end of the Passenger log file
[ W 2018-02-06 16:27:28.8642 14/T4 age/Wat/AgentWatcher.cpp:97 ]: Passenger core (pid=17) crashed with signal SIGSEGV, restarting it...
2018/02/06 16:27:28 [error] 38#0: *4 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: _, request: "HEAD / HTTP/1.1", upstream: "passenger:unix:/app/current/tmp/passenger.pnTbJGQ/agents.s/core:", host: "0.0.0.0"
[ N 2018-02-06 16:27:28.8808 86/T1 age/Cor/CoreMain.cpp:1242 ]: Starting Passenger core...
[ N 2018-02-06 16:27:28.8812 86/T1 age/Cor/CoreMain.cpp:252 ]: Passenger core running in multi-application mode.
[ N 2018-02-06 16:27:28.9032 86/T1 age/Cor/CoreMain.cpp:977 ]: Passenger core online, PID 86
Lastly, here are the relevant files (including the Passenger log with backtrace, Dockerfile, etc)
https://gist.github.com/nater540/c22414d21bf22a990a474c93936b7c06
If anyone has any suggestions or insight that would be absolutely fantastic!
Related
Hi I'm using Nginx + Passenger
I'm stuck in the issue.
[ E 2020-09-20 09:32:51.5408 14555/To age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /home/xxxxxxx: A timeout occurred while starting a preloader process.
Error ID: 55f4b7bd
Error details saved to: /tmp/passenger-error-6SSEIC.html
[ E 2020-09-20 09:32:51.5478 14555/T8 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-2] Cannot checkout session because a spawning error occurred. The identifier of the error is 55f4b7bd. Please see earlier logs for details about the error.
sudo nano /etc/nginx/passenger.conf
passenger_root /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini;
passenger_ruby /home/deploy/.rbenv/shims/ruby;
I am deploying a Ruby on Rails 5.2 app on Ubuntu 16.04 running Apache2 and Passenger 6.0.1 with rvm 1.29.7 in single user mode. I can't seem to get the app to start up with the following failures in the subprocess "Initialize Language Runtime". I've exhausted my search on anyone having a similar problem, so I'm thinking there is something unique to my installation. I'm providing as many details as I think are relevant below, but let me know if you need additional information to help me solve this one:
Here is /var/log/apache2/error.log after I attempt to restart apache:
[Thu Jan 31 03:55:14.310845 2019] [mpm_prefork:notice] [pid 12051] AH00173: SIGHUP received. Attempting to restart
[ N 2019-01-31 03:55:14.3335 14690/T9 age/Cor/CoreMain.cpp:671 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ N 2019-01-31 03:55:14.3337 14690/T1 age/Cor/CoreMain.cpp:1245 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ N 2019-01-31 03:55:14.3338 14690/Tb Ser/Server.h:902 ]: [ApiServer] Freed 0 spare client objects
[ N 2019-01-31 03:55:14.3338 14690/Tb Ser/Server.h:558 ]: [ApiServer] Shutdown finished
[ N 2019-01-31 03:55:14.3339 14690/T9 Ser/Server.h:902 ]: [ServerThr.1] Freed 0 spare client objects
[ N 2019-01-31 03:55:14.3339 14690/T9 Ser/Server.h:558 ]: [ServerThr.1] Shutdown finished
[ N 2019-01-31 03:55:14.3618 19760/T1 age/Wat/WatchdogMain.cpp:1366 ]: Starting Passenger watchdog...
[ N 2019-01-31 03:55:14.3875 19763/T1 age/Cor/CoreMain.cpp:1339 ]: Starting Passenger core...
[ N 2019-01-31 03:55:14.3878 19763/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2019-01-31 03:55:14.3949 19763/T1 age/Cor/CoreMain.cpp:1014 ]: Passenger core online, PID 19763
[Thu Jan 31 03:55:14.415668 2019] [mpm_prefork:notice] [pid 12051] AH00163: Apache/2.4.18 (Ubuntu) Phusion_Passenger/6.0.1 OpenSSL/1.0.2g configured -- resuming normal operations
[Thu Jan 31 03:55:14.415804 2019] [core:notice] [pid 12051] AH00094: Command line: '/usr/sbin/apache2'
[ N 2019-01-31 03:55:14.8130 14690/T1 age/Cor/CoreMain.cpp:1324 ]: Passenger core shutdown finished
[ N 2019-01-31 03:55:16.8669 19763/Tb age/Cor/SecurityUpdateChecker.h:519 ]: Security update check: no update found (next check in 24 hours)
Here is my environment:
ubuntu#ip-172-31-13-212:~$ env
XDG_SESSION_ID=20
rvm_bin_path=/home/ubuntu/.rvm/bin
TERM=xterm-256color
SHELL=/bin/bash
SSH_CLIENT=173.52.197.236 49586 22
SSH_TTY=/dev/pts/2
rvm_stored_umask=0002
USER=ubuntu
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
rvm_path=/home/ubuntu/.rvm
rvm_prefix=/home/ubuntu
MAIL=/var/mail/ubuntu
PATH=/home/ubuntu/bin:/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/ubuntu/.rvm/bin:/home/ubuntu/.rvm/bin
rvm_loaded_flag=1
PWD=/home/ubuntu
LANG=en_US.UTF-8
rvm_version=1.29.7-next (master)
SHLVL=1
HOME=/home/ubuntu
LOGNAME=ubuntu
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
SSH_CONNECTION=173.52.197.236 49586 172.31.13.212 22
LESSOPEN=| /usr/bin/lesspipe %s
XDG_RUNTIME_DIR=/run/user/1000
LESSCLOSE=/usr/bin/lesspipe %s %s
rvm_user_install_flag=1
_=/usr/bin/env
I found the answer to this issue thanks to the folks supporting the Passenger opensource project. It turns out that in my virtual host file for Apache, the path I provided for PassengerRuby was pointing to a nonexistent ruby. You can read about it here: https://github.com/phusion/passenger/issues/2179
I have been trying to setup a Rails application in RHEL 7 using Apache (v 2.4.6), Phusion Passenger (v 5.3.5).
Everything works fine without SSL configuration.
But if I set up virtualhost for 443 port, I get the following error after restarting (service httpd restart) the Apache service.
[Wed Nov 07 18:20:50.215257 2018] [mpm_prefork:notice] [pid 31701] AH00170: caught SIGWINCH, shutting down gracefully
[ N 2018-11-07 18:20:51.2344 31732/T7 age/Cor/CoreMain.cpp:641 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ N 2018-11-07 18:20:51.2344 31732/T1 age/Cor/CoreMain.cpp:1216 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ N 2018-11-07 18:20:51.2345 31732/T1 age/Cor/CoreMain.cpp:1117 ]: Checking whether to disconnect long-running connections for process 3793, application /opt/apps/my_rails_app (production)
[ N 2018-11-07 18:20:51.2347 31732/T9 Ser/Server.h:903 ]: [ServerThr.2] Freed 0 spare client objects
[ N 2018-11-07 18:20:51.2347 31732/T9 Ser/Server.h:559 ]: [ServerThr.2] Shutdown finished
[ N 2018-11-07 18:20:51.2348 31732/T7 Ser/Server.h:903 ]: [ServerThr.1] Freed 0 spare client objects
[ N 2018-11-07 18:20:51.2348 31732/T7 Ser/Server.h:559 ]: [ServerThr.1] Shutdown finished
[ N 2018-11-07 18:20:51.2349 31732/Tc Ser/Server.h:903 ]: [ApiServer] Freed 0 spare client objects
[ N 2018-11-07 18:20:51.2349 31732/Tc Ser/Server.h:559 ]: [ApiServer] Shutdown finished
[ N 2018-11-07 18:20:51.2349 31732/T1 age/Cor/CoreMain.cpp:1117 ]: Checking whether to disconnect long-running connections for process 3793, application /opt/apps/my_rails_app (production)
[ N 2018-11-07 18:20:51.4769 31732/T1 age/Cor/CoreMain.cpp:1295 ]: Passenger core shutdown finished
[ N 2018-11-07 18:20:51.5469 4216/T1 age/Wat/WatchdogMain.cpp:1300 ]: Starting Passenger watchdog...
[ N 2018-11-07 18:20:51.5688 4219/T1 age/Cor/CoreMain.cpp:1310 ]: Starting Passenger core...
[ N 2018-11-07 18:20:51.5689 4219/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2018-11-07 18:20:51.5754 4219/T1 age/Cor/CoreMain.cpp:985 ]: Passenger core online, PID 4219
[Wed Nov 07 18:20:51.576671 2018] [suexec:notice] [pid 4214] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[ N 2018-11-07 18:20:51.5800 4219/T7 age/Cor/CoreMain.cpp:641 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ N 2018-11-07 18:20:51.5800 4219/T1 age/Cor/CoreMain.cpp:1216 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ N 2018-11-07 18:20:51.5801 4219/T7 Ser/Server.h:903 ]: [ServerThr.1] Freed 0 spare client objects
[ N 2018-11-07 18:20:51.5801 4219/T7 Ser/Server.h:559 ]: [ServerThr.1] Shutdown finished
[ N 2018-11-07 18:20:51.5802 4219/Tb Ser/Server.h:903 ]: [ApiServer] Freed 0 spare client objects
[ N 2018-11-07 18:20:51.5802 4219/Tb Ser/Server.h:559 ]: [ApiServer] Shutdown finished
[ N 2018-11-07 18:20:51.5802 4219/Ta Ser/Server.h:903 ]: [ServerThr.2] Freed 0 spare client objects
[ N 2018-11-07 18:20:51.5802 4219/Ta Ser/Server.h:559 ]: [ServerThr.2] Shutdown finished
[Wed Nov 07 18:20:51.598254 2018] [auth_digest:notice] [pid 4214] AH01757: generating secret for digest authentication ...
[ N 2018-11-07 18:20:51.6198 4237/T1 age/Wat/WatchdogMain.cpp:1300 ]: Starting Passenger watchdog...
[ N 2018-11-07 18:20:51.6221 4219/T1 age/Cor/CoreMain.cpp:1295 ]: Passenger core shutdown finished
[ N 2018-11-07 18:20:51.6448 4240/T1 age/Cor/CoreMain.cpp:1310 ]: Starting Passenger core...
[ N 2018-11-07 18:20:51.6449 4240/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2018-11-07 18:20:51.6517 4240/T1 age/Cor/CoreMain.cpp:985 ]: Passenger core online, PID 4240
[Wed Nov 07 18:20:51.653813 2018] [lbmethod_heartbeat:notice] [pid 4214] AH02282: No slotmem from mod_heartmonitor
[Wed Nov 07 18:20:51.658584 2018] [mpm_prefork:notice] [pid 4214] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) Phusion_Passenger/5.3.5 OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Wed Nov 07 18:20:51.658612 2018] [core:notice] [pid 4214] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[ N 2018-11-07 18:20:54.1476 4240/T5 age/Cor/SecurityUpdateChecker.h:517 ]: Security update check: no update found (next check in 24 hours)
I did not find any particular error message from the logs as above.
Looking forward to your help.
Thanks
EDIT 1
On the web page, I get Forbidden You don't have permission to access / on this server.
ssl_error_log shows the following error:
[Wed Nov 07 19:02:46.529549 2018] [autoindex:error] [pid 14038] [client 138.35.142.252:35811] AH01276: Cannot serve directory /opt/apps/my_rails_app/public/: No matching DirectoryIndex (index.html) found, and server-generated directory index forbidden by Options directive
SSL conf file (ssl.conf) looks as below:
<Directory "/opt/apps/my_rails_app/public">
Options ExecCGI FollowSymLinks
Require all granted
Order allow,deny
Allow from all
</Directory>
PassengerAppEnv production
PassengerAppRoot "/opt/apps/my_rails_app/public"
PassengerHighPerformance on
I am developing android service with ROR+Nginx+passenger and Amazon Web service.
The server goes well, but yesterday the server suddenly got downed.
I've tried to solve the problem, but I couldn't.
Here's my problem.
First, the EC2 instance's log/production.log doesn't write any log.
I can get 502 bad gateway error message in my android app log.
Second, all requests including http, https are responded by 502 badgateway.
Third, my AWS load balancer is logging 50X error.
I guess this problem is about ELB, but I don't know how to solve.
I have ssl certificate, and I have only one EC2 instance.
Any help will be appreciated.
Edited
2017-02-22 00:22:30.9139 25313/7f222067b700 age/Cor/Con/CheckoutSession.cpp:269 ]: [Client 1-356202] Returning HTTP 503 due to: Request queue full (configured max. size: 100)
[ 2017-02-22 00:24:00.3704 25313/7f221ee09700 age/Cor/App/Implementation.cpp:304 ]: Could not spawn process for application /home/ec2-user/popcake: An error occurred while starting up the preloader: it did not write a startup response in time.
Error ID: 10045ede
Error details saved to: /tmp/passenger-error-JS1Bnt.html
Message from application: An error occurred while starting up the
preloader: it did not write a startup response in time. Please read this article for more information about this problem.<br>
<h2>Raw process output:</h2>
(empty)
[ 2017-02-22 00:24:00.4527 25313/7f222067b700 age/Cor/Con/CheckoutSession.cpp:285 ]: [Client 1-353433] Cannot checkout session because a spawning error occurred. The identifier of the error is 10045ede. Please see earlier logs for details about the error.
[ 2017-02-22 00:30:29.0872 25313/7f222067b700 age/Cor/CoreMain.cpp:532 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ 2017-02-22 00:30:29.1173 25313/7f2227bfc840 age/Cor/CoreMain.cpp:901 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ 2017-02-22 00:30:29.1174 25313/7f221fe7a700 Ser/Server.h:817 ]: [ApiServer] Freed 0 spare client objects
[ 2017-02-22 00:30:29.1174 25313/7f221fe7a700 Ser/Server.h:464 ]: [ApiServer] Shutdown finished
[ 2017-02-22 00:30:29.2400 25318/7f77237e2700 age/Ust/UstRouterMain.cpp:422 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ 2017-02-22 00:30:29.2741 25318/7f772a775840 age/Ust/UstRouterMain.cpp:492 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ 2017-02-22 00:30:29.3043 25318/7f77237e2700 Ser/Server.h:464 ]: [UstRouter] Shutdown finished
[ 2017-02-22 00:30:29.3042 25318/7f7722fe1700 Ser/Server.h:817 ]: [UstRouterApiServer] Freed 0 spare client objects
[ 2017-02-22 00:30:29.3048 25318/7f7722fe1700 Ser/Server.h:464 ]: [UstRouterApiServer] Shutdown finished
[ 2017-02-22 00:30:29.3720 25318/7f772a775840 age/Ust/UstRouterMain.cpp:523 ]: Passenger UstRouter shutdown finished
[ 2017-02-22 00:30:29.4023 25313/7f222067b700 age/Cor/CoreMain.cpp:532 ]: Signal received. Gracefully shutting down... (send signal 1 more time(s) to force shutdown)
If you see the log that you have added it says "Request queue full", basically you have some api's which are long running, because of which 50x is being thrown after the queue was full.
Restarting nginx might fix the issue for now but you need to look at the response times of your apis and see which are slow performing and fix them otherwise it will be a recurring issue and you will have to restart nginx everyday.
I have a rails application that I am deploying on an aws ec2 instance, this is my first time using the Rubber Gem, for which I have followed this tutorial.
After removing the default nginx configurations and after intial deployment (step 5 of the tutorial I have mentioned) when ever I try and access my app using the public IP or DNS I get a 403 Forbidden nginx screen.
On inspecting the error log of nginx on the server this is what I found. I have been stuck on this issue for quite sometime now, any help would be greatly appreciated. Thanks.
tail -f /var/log/nginx/error.log
[ 2015-07-10 05:49:13.7236 13522/7fc24be3a780 age/Wat/Main.cpp:1264 ]: Starting PassengerAgent watchdog...
[ 2015-07-10 05:49:13.7526 13525/7f2f7f588780 age/Hel/Main.cpp:944 ]: Starting PassengerAgent server...
[ 2015-07-10 05:49:13.7529 13525/7f2f7f588780 age/Hel/Main.cpp:237 ]: PassengerAgent server running in multi-application mode.
[ 2015-07-10 05:49:13.7550 13525/7f2f7f588780 age/Hel/Main.cpp:698 ]: PassengerAgent server online, PID 13525
[ 2015-07-10 05:49:13.7851 13530/7f39b592e780 age/Log/Main.cpp:442 ]: Starting PassengerAgent logger...
[ 2015-07-10 05:49:13.7863 13530/7f39b592e780 age/Log/Main.cpp:323 ]: PassengerAgent logger online, PID 13530
[ 2015-07-10 05:49:13.7876 13525/7f2f7f588780 age/Hel/Main.cpp:857 ]: Watchdog seems to be killed; forcing shutdown of all subprocesses
[ 2015-07-10 05:49:13.7877 13525/7f2f79042700 age/Hel/Main.cpp:508 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
2015/07/10 05:49:13 [alert] 9729#0: Required key 'core_address' is missing (-1: Unknown error)
Passenger author here. This error occurs when the Passenger module compiled inside Nginx, does not match the actual Passenger version that is installed. In other words, your installation is broken.
I recommend that you start over completely, on a new and empty server, using the Passenger official walkthrough documentation.