Could you tell me what happens with AWS Server now? From 3 weeks ago, util now, whenever I deploy my RoR app into AWS Server (using ElasticBeantalk tool), I meet a strange issue
Deployment time is quite good (just about 10-15 minutes), and the healthy of server is still green. But after that, the server is inaccessible. This status last about 3 - 4 hours !!! Then, everything is OK, server run fast and smoothly. I totally don't understand server healthy still un-change although this error happen. Everything I can do is "refresh browser periodically until it run"
I don't think my application is bigger enough with total deployment time like that. It just takes me about 20 minutes on local (production mode)
Here're some error I found out when server is hang:
"An error occured while starting up the preloader."
"Gateway timeout" when loading application.js (using chrome debug)
"Bad gateway" when loading application.js (using chrome debug)
Please give me some advise to solve that. I have been stucked on this issue for a long time
Thanks
Related
I have a rails app which when deployed to Heroku takes 3 to 4 minutes to go live even after successful deployment. During that that period it shows error as "request timeout". I have enabled preboot as mentioned here:
https://devcenter.heroku.com/articles/preboot but still same result. I am on Heroku enterprise and still facing this issue. Please suggest what i am missing, here is the dyno and Add-ons info that i am using
My problem is similar to AWS: None of the Instances are sending data but has a slightly different error message.
I have a Rails application running on ElasticBeanstalk, and it appears to be running correctly. Periodically, Enhanced Health Monitoring sends me error messages such as:
Environment health has transitioned from Ok to Degraded. 20.0 % of the
requests are failing with HTTP 5xx.
where the percentage varies up to 100%. Even though I've made no changes, a minute later I get a followup message telling me that everything is back to normal:
Environment health has transitioned from Degraded to Ok.
I've downloaded the full logs from ElasticBeanstalk but I don't know exactly where to look (there are around 20 different log files in various directories).
I'm currently using the free AWS tier with the smallest instances of database, server, etc. Could this be the cause? Which of the log files should I be looking in, and what should I be looking for?
I run rails apps on Elastic Beanstalk and have found it helpful to think about Beanstalk as a computer (in this case an Amazon EC2 instance) running your rails app and a web server (either Passenger or Puma). When you get a 500 error, it could be because your rails app didn't properly deploy–in which case Passenger or Puma will return an error—or your app is deployed properly but encountered an error just like it might on your local machine.
In either case, to diagnose an error, download the full logs from your AWS console (open the correct app environment and then choose Logs > Request Logs > Full logs > Download). Deployment errors are harder to diagnose, but I recommend starting by looking in var-XX/logs/log/eb-activity.log. I suspect your error is coming from your rails app itself, in which case I recommend looking in var-XX/app/support/logs/passenger.log and production.log. To find a 500 error, search for "500 Internal" and then treat the error like you would any other rails error.
You can go to the EC2 instance and run the application just like you would run on your local machine and see the logs.
You can ssh into your EC2 instance using the command eb ssh and go to /opt/python/ directory (It will be different for Ruby or other programming languages).
/opt/python/run is the directory where you will find the version of your application which is run from the EC2 instance. Look for the directory venv and app inside run directory.
Note: The above folder structure is for Python but a similar folder structure post deployment can be found for any other programming language. Just look for the standard directory structure for the deployment environment for your programming language.
For Python:
/opt/python: Root of where you application will end up.
/opt/python/current/app: The current application that is hosted in the environment.
/opt/python/on-deck/app: The app is initially put in on-deck and then, after all the deployment is complete, it will be moved to current. If you are getting failures in your container_commands, check out out the on-deck folder and not the current folder.
/opt/python/current/env: All the env variables that eb will set up for you. If you are trying to reproduce an error, you may first need to source /opt/python/current/env to get things set up as they would be when eb deploy is running.
/opt/python/run/venv: The virtual env used by your application; you will also need to run source /opt/python/run/venv/bin/activate if you are trying to reproduce an error.
I know it is a little late but I wanted to comment the trick I use to find the error, I use to connect via ssh and then, once in the app I try to enter "rails console" It uses to fail, but it shows normally the error you´re making. This little trick saved my life several times. Hope it helps!
I have a Rails 4.2 app running on Heroku. Occasionally there is an issue that causes most incoming requests to get a server error. For example, there could be a memory leak or a max database connection issue. How can I setup a script or service to automatically restart the server when it detects errors?
I think this service could ping the app every few minutes and if it detects an error, it should confirm there's really a problem and then run heroku restart. How could this be set up?
After Googling this topic, I came across Neptune.io, which seems to provide a useful service for this task.
I have a RoR app running on elastic beanstalk. I have occasionally seen 403 errors from Passenger for a while. Most of the time 1 server is running but this gets increased to 3 or 4 instances in busy periods during the day.
Session stickeyness is not turned on
I have noticed that when a new server is started the ELB is sending requests to it before bundle install has finished.
If I ssh to the newly started server I can see in /var/app/current/ that the app has not yet been installed and if I run top it looks like bundler is running and compiling things with cc1, etc.
/var/app/support/log/passenger.log shows that requests to valid urls within my rails app are being received and responded to with 404. Hardly surprising because the app isn't there yet
After 5-10 minutes all of the compiling is complete and the app files appear in /var/app/current and all is well.
This doesn't seem quite right to me. How do I set up the ELB / my rails app so that the ELB can tell when it is ready to receive requests?
I found the answer to this. There was no application health check url set. In this case the ELB pings the instance to see if it's healthy, i.e. it checks that it is booted rather than if rails is up and running. Setting the health check url to '/login/' fixed it for me because this gives a 404 until rails in running and a 200 afterwards.
Elastic beanstalk demands 2 correct responses before it deems an instance to be healthy. It checks the instance every 5 minutes. This means that an instance can take a while to start serving requests. i.e. it takes boot time + waiting for next poll from elb + 5 minutes before it sees any real traffic
Background
I'm running a Ruby on Rails application that has to serve a lot of static files as well.
My setup currently is:
Debian Linux Lenny 5.0
Apache 2.2.9
Passenger 2.2.10
The problem
Everything runs fine. I see apache process spinning up, passenger instances get created and everything works fast and snappy.
Then, after some time Apache does not respond to requests any more. Clients do get a connection and are "waiting for a response", but none comes.
I cannot manually reproduce this problem. Sometimes it occurs a few hours after a restart, other times it takes a few days to happen. Here's what I found:
Apache process are up; Passenger is there, but it does not have any instances spun up (probably because instances die after a period of inactivity)
No error messages or problems in /var/log/syslog, /var/log/messages, not in apache's access and errors logs, not in my Rails production log. Nothing.
When I stop and start apache everything is back to normal.
Does any one have any clues what's happening here? And how it can be resolved?
Due to an enormous load on static files we decided to host static files on separate server (later Amazon S3+CloudFront) for performance reasons.
My current guess is that Apache couldn't cope with the large number of requests on static files and also doing Passenger. The current setup is Nginx+Unicorn for the Rails application and S3+CloudFront for static files.