I needed to create an image of the current droplet on DigitalOcean, so I powered off the droplet from dashboard, created the new image of the current droplet and then wanted to turn on the droplet again. So I hit "Power Cycle".
The problem is that after 10 minutes, the app is still not up. What happened and, how can I fix it?
Thanks
It sounds like the application may not be configured to start on boot. The power cycle feature at DO is basically just a gentle reboot (it tries to wait for things to shut down instead of deploying more harsh methods to stop the processes). Does the application start up if you SSH in and start it manually? If so, next check on what starts it automatically at boot. It might have encountered an error (environment errors are common) and should have reported somewhere (syslog, application log, etc.).
Just to help you out moving forward, DO supports live snapshots now, so you should be able to snapshot your droplet without powering it off. :)
Related
I'm planning to have a Rails App that has a very content rich interactive page where many users will connect to.
Development has went well and small time testing on the Dev servers went without a hitch either.
Problems started when we started alpha testing with selected groups of people. the sever would grind to a halt suddenly. Nginx would stop because of queue being full. I was at a lose for a while, but after looking around, came to the conclusion that the live actioncable was completely eating up my memory.This especially gets bad when the user reloads the page multiple times that subscribes to actioncable, causing additional process to become active, completely stopping the server, only being cured by a nginx reboot.
I currently run a 2core 1GB memory SSD run VPS server for alpha testing, perhaps at tops 20 concurrent users.Should I be running into performance problems with such load? or should tuning the code or redis, passenger fix this?
I know its hard to say any definitive things without more specifics, but could a ballpark estimate be done with the information?
After some gogoling and testing Nginx settings, adding this directive to the nginx settings for passenger has seemed to dramatically improve the performance issue.
location /special_websocket_endpoint {
passenger_app_group_name foo_websocket;
passenger_force_max_concurrent_requests_per_process 0;
}
more info here
https://www.phusionpassenger.com/library/config/nginx/tuning_sse_and_websockets/
20 concurrent users plus multiple tabs per user is still less than about 100 concurrent websocket connections, it is not that lot.
First thing I'd look for is leaks - when for some reason websocket connection or other resources (open files etc.) does not get freed when actual user disconnects. Make sure you're running fresh versions of rails/passenger, as there was a bug in rails causing similar behaviour (see https://blog.phusion.nl/2016/07/07/actioncable-under-stress-p1/ for details)
Also while actioncable+passenger inside nginx allows you to run everything inside single process, it is not a good idea when you expect some load.
When running a clean nginx and separate rails servers for regular requests and cable - at least other parts of the app will continue some kind of working in such conditions.
We have been trouble shooting a 503 error stating servers are at capacity.
we have removed the potential problems after monitoring the time stamp of the occurence. Everything tests healthy in our dashboard, we stopped instances and restarted them, as well as creating new instances to see if that would help. It is not a big website so we are nowhere near capacity..
Sorry On to the question...Has anybody run into this problem and how did they go about trouble shooting beyond what we have already done.
Thanks,
Found the answer...When we created our new AMI in AWS for our new Instance. We had an AMI that had the same image and no code deployment. So the two AMI's cancelled eachother out and our load balancer just stopped and didn't serv up anything else...
We had to Terminate the AMI, stop and restart the app instances and the and most importantly had to go to code deployment and re deploy the code.
Hope this helps somebody else down the road.
This would sound very newbie but I've just added a centralized logging service (Splunkstorm free version) to my rails app on heroku and it completely changed my life. I don't know why i never thought of this before.
I can just read all the logs from web interface without running heroku logs --tail which spawns a new dyno everytime i do it.
Which makes me curious: Does adding this type of logging service affect how much i have to pay to heroku? I mean, it's sending out packets every time something happens.
Nope!
Bandwidth is included in the dyno pricing (including the one you get for free).
There is a soft limit at 2TB of bandwidth, but you're unlikely to come anywhere near that from logging.
I've used capistrano for a long time, but always for sites that weren't critical. If something went wrong, a few minutes of downtime weren't a big problem.
Now I'm working on a more critical service and need to cover my edge cases. One of which is if my local connection to a server becomes interrupted in the middle of a deployment.
One solution I can think of is to do deployments directly from the server, inside of a screen session. This seems like a reasonable and obvious solution, but I'm surprised I've never read about it elsewhere or even seen it recommended in the capistrano documentation.
Any pointers / tips are welcome. Thanks!
There is only a very small time window during a typical Capistrano deploy where a dropped connection would cause trouble. This window is when the current release is linked to the new version and the server is told to restart. If your connection drops before or after that, you're fine.
If you positively need to be safe from disconnects 100%, you can log onto the server, open a screen session and do a cap deploy from the latest release folder.
I'm running an application that kicks off a Rufus Scheduler process in an initializer. The application is running with Passenger in production and I've noticed a couple weird behaviors:
First, in order to restart the server and make sure the initializer gets run, you have to both touch tmp/restart.txt and load the app in a browser. At that point, the initializer fires. The horrible thing is that if you only do the touch, the processes scheduled by Rufus get reset and aren't rescheduled until you load the app in a browser.
This alone I can deal with. But this leads to the second problem: I'll notice that the scheduled process hasn't run, so I load a page and suddenly the log file is telling me that it's running the initializers as if I'd rebooted. So, at some point, Passenger is randomly rebooting as if I'd touched tmp/restart.txt and wiping out my scheduled processes.
I have an incredibly poor understanding of Passenger and Rails's integration, so I don't know whether this occasional rebooting is aberrant or all part of the architecture. Can anyone offer any wisdom on this situation?
What you describe is the way Passenger works. It spawns new instances of the application when traffic warrants them, and shuts them down after periods of inactivity to free resources.
You should read the Passenger documentation, particularly the Resource Control and Optimization section. There are settings which can prevent the application from being shut down by Passenger, if that is what you want.
Using the PassengerPoolIdleTime setting, you could keep at least one process running, but you'll almost certainly want Passenger to start up other instances of the app as necessary. This thread on the Rufus Scheduler Google Group mentions using lock files to prevent more than one process from starting the scheduler, that may be useful to you.