Is it possible to do something like the Github zero downtime deploy on Heroku using Unicorn on the Cedar stack?
I'm not entirely sure how the restart works on Heroku and what control we have over restarting processes, but I like the possibility of zero downtime deploys and up until now, from what I've read, it's not possible
There are a few things that would be required for this to work.
First off, we'd need backwards compatible migrations. I leave that up to our team to figure out.
Secondly, we'd want to migrate the db right after a push, but before the restart (assuming our migrations are fully backwards compatible, this should not affect anything)
Thirdly, we'd want to instruct Unicorn to launch a new master process and fork some workers, then swap the PIDs and gracefully shut down the old process/workers
I've scoured the docs but I can't find anything that would indicate this is possible on Heroku. Any thoughts?
I can't address migrations, but the part about restarting processes and avoiding wait time:
There is an beta feature for heroku called preboot. After a deploy, it boots your new dynos first and waits a while before switching traffic and killing the old ones:
https://devcenter.heroku.com/articles/labs-preboot/
I also wrote a blog post that has some measurements on my app's performance improvements using this feature:
http://ylan.segal-family.com/blog/2012/08/27/deploy-to-heroku-with-near-zero-downtime/
You might be interested in their feature called preboot.
Taken from their documentation:
This feature provides seamless deploys by booting web dynos with new code before killing existing web dynos.
Some apps take a long time to boot up, and this can cause unacceptable delays in serving HTTP requests during deployment.
There are a few caveats:
You must have at least two web dynos to use this feature. If you have your web process type scaled to 1 or 0, preboot will be disabled.
Whoever is doing the deployment will have to wait a few minutes before the new code starts serving user requests; this happens later than it would without preboot (but in the meanwhile, user requests are still served promptly by old dynos).
There will be a short period (a minute or two) where heroku ps shows the status of the new code, but user requests are still being served by old code.
There is much more information about it, so refer to their documentation.
It is possible, but requires a fair amount of forward planning. As of Rails 3.1 there's three tasks that need carrying out
Upload the new code
Run any database migrations
Sync the assets
Uploading code and restarting is fairly straightforward, the main problem lies with the other two, but the way round them is the pretty much the same.
Essentially you need to:
Make the code compatible with the migration you need to run
Run the migration, and remove any code written specifically for it
For instance, if you want to remove a column, you’ll need to deploy a patch telling ActiveRecord to ignore it first. Only then you can deploy the migration, and clean up that patch.
In short, you need to consider your database and the code compatability an work around them so that the two can overlap in terms of versioning.
An alternative to this method might be to have two versions of the application running on Heroku at the same time. When you deploy, switch the domain to the other version, do the deploy, and switch it back again. This will help in most instances, but again, database compat is an issue.
Personally, I would say that if your deployments are significant to require this sort of consideration, taking parts of the application offline are probably the safest answer. By breaking up an application into several smaller applications can help mitigate this and is a mechanism that I use regularly.
No - this is currently not possible using Unicorn on Heroku cedar. I've been bugging Heroku about this for weeks.
Here was Heroku Support's reply to my email on March 8, 2012:
Hi, you could enable maintenance mode when doing a deploy, at least your users would see a maintenance page instead of an error, and also request queue wouldn't build up.
We're definitely aware this is a pain and we're working to offer rolling / zero-downtime deploys in the future. We have no ETA to announce, though.
Related
I created a simple stock screener (filters out stocks given certain criteria) in Rails. On my localhost the stocks update instantly, but on Heroku it can take anywhere from 10-15 seconds before the stock list is updated.
My Heroku app is here: http://fruthscreener.herokuapp.com/
Github is here: github dot com/anfruth/Fruth_Screener_Rails
The code involved in updating the queries can be found in the user_stocks model and in the stocks controller under def create.
Any ideas why this is happening and suggestions as to how to fix it?
Thank you.
Not slow for me
--
Heroku
The only thing which will slow Heroku down is if your db connection is "off-site"
We've had apps before which ran super slowly due to the case that the database provider was a different host, in a different country.
Heroku runs on AWS, meaning it will run super fast if you have all the dependencies in the same data-center. One of the drawbacks of using one of these powerful "cloud" hosting providers is they need to keep all requests local to help their system run quickly; hence if your DB is "off-site", it will slow it down profusely.
You must remember that Rails apps can't run unless they have a db connection; so if your connectivity is slow, your app's performance is going to be hit hard
-
Postgres
If your app is running slow on Heroku, the best thing to do is to make sure you're using Heroku's postgres database. This is deployed on Heroku's AWS cloud, meaning it's on the same network as your app, hence allowing it to run as quickly as possible
You'll need to change your app's database connection to the new production server like this:
#config/database.yml
production:
.... #-> your Heroku db details here
This will allow you to run heroku run rake db:migrate after you push this new code to Heroku - which should define the db structure for you, allowing you to populate it as you wish
It sounds like you would benefit from using New Relic or another performance management package for Heroku in order to find out what is causing you trouble exactly. The free tier of New Relic should be enough to get you started.
By the way, if your app is a Heroku free tier app (one single web dyno), then your dyno will go to sleep when not in use, and you may be encountering dyno spin-up costs, which are frequently about 5-15 seconds. Repeat the same query several times in several minutes and see if the slowness persists for every request, or only the first one.
Just wondering how everyone pushes updates to their production server on Heroku, without bringing the app down for a few couple of seconds?
Pushing to Heroku (especially using something like Unicorn), takes a while for the web app to load. Especially when there are end-users trying to access the site. They end up with 503 pages.It takes up to 30 secs to a minute for Unicorn processes to load.
There are TWO things you need to accomplish this, and it's not trivial.
1) Migrations need to be backwards compatible (i.e., run while the app is live). See this article about that: http://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/
2) Deploy using TWO heroku apps. I opened a ticket with Heroku on this topic and this was their reply:
We're currently working on a solution to provide a zero-downtime
deploy but do not have an ETA on when this might be available.
In the meantime a workaround that might be possible would be to deploy
to two separate apps. You can push new code to the second app, spin it
up, then move the domain names over to the second app. Wash and repeat
on the next deploy. This is less than ideal but might get you the
desired result in the interim.
If you do this, you will want to automate as much as possible since there's a lot of ways to mess that up. Here's an article on that topic: http://casperfabricius.com/site/2009/09/20/manage-and-rollback-heroku-deployments-capistrano-style/
Why?
These two solutions must both be done because the database migrations must work in BOTH (live and to-be-live) versions of the code. Once you have that working, THEN you can solve the second problem of having the application itself not seem like it went down. There is no supported way to spin up and spin down individual dynos once a push is started.
UPDATE:
There is a beta feature available with Heroku now. To use do the following prior to pushing:
heroku labs:enable -a APP_NAME preboot
This will change the behavior of the app during pushes. It will push up a parallel instance that will warm up for two minutes. Almost exactly two minutes after the push, all traffic will route to the new app. Be careful with migrations, as I mentioned above, as they are still an issue.
Heroku is currently testing their new preboot feature in beta. You might wanna check it out. Unfortunately it only works for ≥2 web dynos. And it also doesn't seem to work for heroku scale web=… which would be important to make it work with HireFireApp.com.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
A little about myself. I am 24 years old, I graduated from NC State with a Master's in Analytics last year. Statistics, mathematics, that kind of thing. I don't have a strong programming background, which is pretty important for my question. If I say anything that doesn't make any sense, that is why. Ever since graduation, I have been working full time on a Rails app with a few other people. My programming experience is mainly Ruby on Rails (1.2 years.) I know R, SAS (statistical languages, not helpful for this question.)
Obviously, that means it has been over a year in development, and we aren't done yet. The main developer is an excellent programmer, just that he has a full time job already, and does this app in his spare time. Due to him not having enough time recently, I have been given practically full responsibility for the app.
We have it deployed on Slicehost right now. The app is at a point where we don't need to program anything else (unless we think of more features.) The reason I am asking if we should migrate to Heroku is that it seems to me that Heroku is a simple platform to which to deploy. Slicehost seems too complicated for me. The other developer dealt with it, and not me. I looked at how to deploy the app on Heroku, and it looks like I would be able to do it. We need our app to scale if it needs to, which Heroku offers. As far as money, I would start it at the minimum (free) and see how it goes. I can pay for additional features if I need to.
We are using Redmine for project management and repository (not git, which I think we need to use on Heroku.) Is git similar to Redmine? Is it easy to use?
Right now, on Slicehost, we have 4 daemons (constantly running processes.) We have 8 delayed_job workers. I know the command line to start the daemons and delayed_job workers. Would these work on Heroku?
I am wondering if I can still use RAILS_ENV=production script/console with Heroku.
The user interface is a javascript file. In development mode, if I do script/server in a terminal, and go to http://localhost:3000 in a browser, I can see it. Would Heroku load this page the way I want?
We have a working website for the app, with our own domain name. I don't really know what DNS is, so I probably wouldn't be able to link the Heroku app to it, unless there is an easy way. I think Heroku links it to appname.heroku.com as a default.
Based on my programming experience, would Heroku be easy enough for me to use, should I find another job, or should I commit seppuku?
Yes, you should definitely deploy your application in heroku. To do this, this is what you will need to do:
Make sure you have git installed in your computer
Create a heroku account here
Install the heroku gem and do the rest as mentioned in this page
Track your application with git, and create your heroku application as shown here
After you do this, heroku will provide you with a URL for your application, such as http://blah-bleep-123.heroku.com. Now, the next step would be to associate your domain to this heroku URL.
Configure your domain DNS Server as shown in this page. Mind you, after you change your DNS, it might take upto 48 hours for it to work. You can change your domain DNS by logging into the site where you bought your domain, for e.g. godaddy.com, hostingdude.com, etc.
Add this code to your ApplicationController. You can follow this from this page as well
class ApplicationController
before_filter :ensure_domain
APP_DOMAIN = 'www.mydomain.com'
def ensure_domain
if request.env['HTTP_HOST'] != APP_DOMAIN
# HTTP 301 is a "permanent" redirect
redirect_to "http://#{APP_DOMAIN}", :status => 301
end
end
end
Make sure you migrate all your database in heroku, by doing heroku rake db:migrate
After you have completed all these steps, you should be good. Check out your domain URL, everything should work pretty :).
If you see any errors in your page, you can view the log by heroku logs
You can access console as heroku console
With features as these, heroku is very convenient to work with.
Please let me know if you need more help.
It seems to me like you should seriously consider Heroku. I have used it for weekend projects and we use it at work as well, quite successfully. Deployment is a breeze, you don't have to worry about setup (for the most part) and system administration. It's super easy to add modules and "pay as you grow".
As for your needs, you could (I believe) run your redmine on Heroku itself, being a rails app. The only thing is that you mention you use Redmine as "repository" and I'm not sure I understand what you mean, since Redmine is not a version control system. Redmine has integration points for various VCS (SVN, git, Mercurial, CVS, and others). Yes, Heroku uses git and that is what you would need to use in order to push code to the server. If you're familiar with Mercurial, it's pretty similar.
For delayed jobs, Heroku offers free cron jobs that run once a day and hourly ones for a fee (see cron add-on). There is also a delayed job plugin (see this) but I don't have any experience with it.
You should be able to access the Rails console (see heroku docs). Just run 'heroku console' and voila, you're there.
If your app works by running script/server, it should work out of the box in heroku too.
As for the DNS, getting it to work with your custom domain is not hard. Out of the box, you can access your app with appname.heroku.com, to set up your custom domain check heroku docs here, but basically you have to add the custom domain add-on (free unless you want subdomains), configure heroku to respond to your domain's requests (couple of simple commands) and set your DNS provider to point to Heroku (there's even a short video in the docs on how to do this with GoDaddy).
The only drawback I've seen with Heroku, and it's not a huge one, is that if your app does not receive any traffic for an extended period of time, the instances kind of "go to sleep", making the next request to arrive somewhat slow (sometimes even timing out), but once the instance is awake, everything is good to go.
All in all, I think Heroku is a great way to take a ton of the burden off of you as a dev and making a lot of things really easy to implement without having to go into the nitty gritty of setting up a server. The downside: once you start growing, it can become somewhat expensive, but hey, if you're growing it probably means you have the cash now to hire someone that can take care of the nitty-gritty.
You might also want to take a look at this blog post which compares Slicehost and Heroku
Best of lucks
YES, go for it.
If you've managed thus far on the strength of your 'programming experience' then you'll be fine. Have some confidence and ship something! To quote Paul Graham:
The reason to launch fast is not so much that it's critical to get your product to market early, but that you haven't really started working on it till you've launched. Launching teaches you what you should have been building. Till you know that you're wasting your time. So the main value of whatever you launch with is as a pretext for engaging users.
The functionality you outline is easily replicated and well documented and it's free to start with. What else could you ask for?
If you have the free time, you might as well sign up for a free account and give it a shot.
HOWEVER, this is going to come with some pretty severe headaches.
Version control will be one, since heroku uses git, but another one that no one's mentioned yet is that your 12 processes ("dynos" in heroku speak) would cost you $35 * 11 = $385 per month! You can set up an hourly cron for $3/month that will flush your delayed_job queue (instead of having workers running at all times), but is that going to be adequate? (If you're running 8 workers, I'm guessing not). This may or may not require some code changes.
Once you get it set up, deployment and admin is really easy (nonexistent), but it'll cost you if you start needing new features.
YES, go for it.
Its good deployment environment any its fast and easy scale out and scale in feature.
even you can use for your testing or demo usage its provide you free account usage of 1 dynos per app.
List of add-on tools available you can add as per your requirement.
I have been hosting a site on Heroku for a few months that is very soon to go into production.
Since I began with them, there have been at least three significant outages, one of which was the disastrous Amazon outage last month and another of which is a multi-hour outage happening today.
I believe in Heroku's vision and I think they are a great company, but I am faced with the ultimate problem: if they can't keep sites up and running, everything I like about them doesn't really matter.
Is Heroku a reliable provider to run a production site on Rails?
Are there any other providers I might look into that have a better reputation for reliability than Heroku?
In my opinion, downtime can happen with almost any provider. What you need to see is how well or badly the host handles the downtime and the effort they make in keeping the customer updated about possible resolution.
In my opinion Heroku is a great place to host your app. The advantages and ease of deploying there covers up for the recent (and rare) downtime FOR ME.
I am user of Heroku with Amazon RDS plugin for the past 7-8 months and my conclusion is there is nothing to appreciate about Heroku except their architecture. Here is why I think:
Even though it is sold for $250 million+ they were still NOT using the Amazon multiple zones feature of Amazon. Below is the link how SmugMug survived amazon crash by using Amazon's multiple zones feature.
http://don.blogs.smugmug.com/2011/04/24/how-smugmug-survived-the-amazonpocalypse/
No phone contact support in the event of issues (not application but Heroku's), lot to learn from Rackspace
The application I am hosting, people will starve if it goes down for few hours on Friday forget about 60 hours downtime.
I see intermittent deployment and connectivity issues. Please visit this link for a confirmation:
http://status.heroku.com/
I know developers love it because they throw a cheap web process called 'dyno' for free.
So far Heroku does not offer multiple availability zone redundancy. If you want something more reliable than Heroku you can create your own EC2 instances in multiple availability zones. Of course this will require significantly more server upkeep, admin, and deployment time.
I have seem Heroku to be reliable. I highly recommended it for starting out and validating your idea. I believe when you start your project you want get it out quickly (to customer or to public).
As mentioned in other comments at some point you might need to switch over to EC2 as you might need zone redundancy and it might actually become cheaper to run of EC2 especially if you already have an SA in the company.
No. It is not. As a customer I've experienced multiple critical outages. These things happen and I get that. But what makes Heroku unreliable is their nearly non-existent support when things do go wrong. I would use caution when evaluating Heroku or any provider for that matter and really understand what you're paying for. Paying as much as I did for Heroku I expected more.
As an example one of their databases went offline early on a Sunday. I immediately was made aware, not from Heroku but from our customers and new relic alerts. I contacted Heroku support just to get the ball rolling as I began to troubleshoot. 24 hours later I had literally no responses from Heroku. I could not fork, follow, or take snapshot of the database as they suggest (because they were experiencing issues) so I basically sat on my hand and waited. Hoping that somebody would respond as I frantically attempt to recover somehow, someway.
Was this their fault. No. Not at all. I should/could have done something to mitigate this failure. But as much as I pay for their servies each month I expected something resembling a response to my critical issue.
Our our app is hosted by Heroku and went down mutliple times over the last 12 months.
Two times it was caused by one of the third-party apps that Heroku offers:
We used Zerigo (recommended by Heroku) for our DNS. This has caused our site to go down twice - one time it took over 12 hours te recover. This is absolutely crazy for something like DNS, so we have switched to a more reliable provider.
The Redistogo app went down once.
Heroku does bring some benefits, but be careful about the apps you select.
In my org i build simple SPA productivity apps, and have been using Heroku to host them for the last year after migrating away from a physical box server to cloud VMs.
I've had multiple days lost due to Heroku development hindering outages. Usually while running apps stay online, and work, when Heroku goes down you can't push updates or restart apps.
Lets also not forget the ridiculous times for scheduled maintenance (usually 2PM EST, midweek....REALLY?)
As of writing this, the Logging system for Heroku has now been acting up (more or less down) for over 24 hour.
Thankfully my apps aren't mission critical. While I like Heroku's ease of use, it's just not worth this much headache for what is nothing other than an AWS middle-man.
That said, I'm moving over to just pure AWS EC2 instances.
Anybody know a nice way to restart a mongrel cluster via capistrano in a "rolling" style, eg, one mongrel at a time. Would be great to have a bit of wait time in there as well for each, to let the mongrel load the rails app up as well.
I've done some searching, and haven't found too much, so looking for help before I dive into the mongrel_cluster gem myself.
Thanks!
I agree with the seesaw approach more than the rolling approach you are seeking. The problem is that you end up in situations where load balancing can throw users back and forth between different versions of the application while you are transitioning.
The solutions we came up with (before finding SeeSaw, which we don't use) was to take half of the mongrels off line from the load balancer. Shut them down. Update them. Start them up. Put those mongrels back online in the load balancer and take the other half off. Shut the second half down. Update the second half. Start them up. This greatly minimizes the time where you have two different versions of the application running simultaneously.
I wrote a windows bat file to do this. (Deploying on Windows is not recommended, btw)
It is very important to note that having database migrations can make the whole approach a little dangerous. If you have only additive migrations, you can run those at any time before the deployment. If you are removing columns, you need to do it after the deployment. If you are renaming columns, it is better to split it into a create a new column and copy data into it migration to run before deployment and a separate script to remove the old column after deployment. In fact, it may be dangerous to use your regular migrations on a production database in general if you don't make a specific effort to organize them. All of this points to making more frequent deliveries so each update is lower risk and less complex, but that's a subject for another response.
Seesaw is a gem found in the Rails Oceania Rubyforge Project that provides this kind of functionality to mongrel clusters. However, the project may be suffering from some bit-rot not havain had a release since 2007. Still worth a look even just to pinch the ideas :)
#!/bin/bash
for PIDFILE in /tmp/mongrel.*; do
PID=$(cat ${PIDFILE})
kill ${PID}
${RUN_MONGREL_CMD} ${PID}
sleep 2
done