I was hoping you folks could recommend a "best practice" for source controlling binary assets for a Rails site in git.
My main concern is that as we work on the site, constantly adding and removing 500kb+ images from our git repo, the repo will eventually get pretty fat and unwieldy and we'll either have to manually remove those images from history (prone to disaster, as far as I can tell) or put up with a long initial download and extra wasted space on disk.
What are some alternatives for separating the app's logic from the assets? Git submodules? Anything else?
Thanks!
It seems this is pretty well covered elsewhere on SO and elsewhere. A start (after 5 seconds of searching):
On how binary files are tracked: http://osdir.com/ml/git/2009-04/msg01221.html
Seemingly relevant: Managing large binary files with git
If you're really seeing bloat in your repo and are wanting to keep the main rails app repo free of this, you can look into git submodules - all changes to the images would be kept within a separate assets (for example) repo, keeping your main repo free of incurred bloat.
Add the images folder to your .gitignore file. e.g.
/app/assets/images/*
Related
Recently, on my latest deploy to Heroku, I got a warning advising not to use AssetSync.
remote: ###### WARNING:
remote: You are using the `asset_sync` gem.
remote: See https://devcenter.heroku.com/articles/please-do-not-use-asset-sync for more information.
The original problem we were trying to solve by using AssetSync was that we were getting a huge slug size caused by the large assets in our application. Out of the 300MB that Heroku allows us, we were probably using close to 230MB - even though our git repo is only around 80MB.
We solved this by using AssetSync to synchronise all our compiled assets to a S3 bucket to be served through Cloudfront. After AssetSync runs, we have a hook that deletes all the precompiled assets to reduce the slug size. Basically, the workflow during slug compilation looked like this:
Let Heroku precompile the assets
AssetSync syncs all compiled assets to S3
All local copies of the compiled assets are deleted
The linked article argues a few points on why it's bad and what to use instead.
Using Asset Sync can cause failures. It is difficult to debug,
unnecessary, and adds extra complexity. Don’t use it. Instead, use a
CDN.
[...]
You should now use a CDN instead. Rather than
copying your assets over to S3 after they are precompiled, the CDN
grabs them from your website. Here are some reasons why that’s better.
Canonical assets
[...] It allows you to have single, authoritative places where you
store information. If you need to change that information, you only
need to change it in one place. [...] What happens if someone has a
failed deploy after assets get synced? What if someone modifies a file
in the S3 bucket? Instead of fixing one copy of assets, now you must
fix two.
Deploy determinism
If you’re debugging inside of a dyno with heroku run bash and you run
rake assets:precompile this doesn’t just modify your local copy. It
actually modifies the copy on S3 as well. [...] The sync part of
asset_sync can also fail if there’s a glitch in the network. What if
you only write part of a file, or only half of your assets are synced?
These things happen.
Although I agree with their points, the question remains: what's the recommended way to deploy a Heroku application that becomes huge when precompiled assets are stored in the slug?
The question is which assets files are making the slug huge?
By default, the Rails assets pipeline should only be used for small and limited internal assets (like JS, CSS, some logos, etc.).
It's not a great idea to store a huge amount of external or big files as Rails assets for many reasons aside the pipeline (like it's making your Git directory big in size too).
Using Rails 3 and Ruby 2. I have the following apps:
App A
App B
Now app A and app B talk to each other via API. This is SOA. As the apps grow bigger, I found that the apps have CSS and Javascripts that can be re-used across the apps. I want to separate the assets (CSS, JS) to a different repo (e.g. app C), so that app A and B can just use the same assets repo.
Here are my questions:
For deployment, app C should precompile and minify, and app A and B should point to that repo. What are the options?
For development, app C should not be minified, but how should app A and B talk to app C continuously to get all changes on CSS/JS in app C?
I have searched through Google, but couldn't get much results. Perhaps my keywords were wrong. Please feel free to name the terms/conventions used for this practice.
If there are other better practices, please feel free to suggest too.
Simplest option might be to use a git submodule to share the assets files themselves between your two projects/apps.
Another option is to create your own gem with those assets files in it, and use that gem in both of your projects. This would require a little more work up front, but would be easier to manage versions down the track (git submodules need a few more manual steps to keep things updated as you change the submodule repository).
I've been using gitignore for password/key protection on my Ruby on Rails apps, so that my config files aren't committed to and publicly available for view on Github.
Going through the Github Developer's API tutorial suggests using environment variables to store these. If I'm not deploying is there any reason I shouldn't use gitignore to store my keys?
There's no strongly compelling reason to prefer environment variables over files, but I'm not sure I'd feel comfortable relying on .gitignore and keeping the files in the source tree. If you opt for files, you might want to consider moving them outside the source tree to avoid any mistakes.
If you've committed the files at any point in the past (particularly to a public site like GitHub), then adding them to .gitignore later isn't going to remove the copies of previous versions that were committed; they'll still be in the repository, and someone could browse back to a version where they weren't ignored.
Additionally, depending on how you run your app, having them in the same directory as the app may make you more vulnerable to web-based security holes. If you keep them outside the app tree, you may gain some limited protection from exploits that can read arbitrary data from the app tree. I don't know that this is all that much of a concern (an exploit that can read within the app tree can probably read outside it too), but it wouldn't hurt to keep them in another directory.
I tend to prefer environment variables because it makes deploying to Heroku easier, so I use them in development too. I'd probably use a file for anything more than simple login/password pairs (RSA keys, etc), and store it in my home directory or somewhere under /var/lib or /etc or /usr/local or something.
There is a gem designed for solving such problems specifically
Figaro: https://github.com/laserlemon/figaro
In a nutshel, Figaro provides a clean and simple way to configure your app and keep the private stuff ... private.
All of your passwords, API keys and anything confidential can be saved in a local yml and won't be pushed to remote git repo.
For every project it's like having two parts: the Rails application and then all documents and pictures related to it.
I wonder how you organize them both.
Do you put everything under the same project root folder and apply Git on that folder or don't you protect your documents with Git at all?
Then, what if the docs are too sensitive or only for owners of that project. Then I probably should't have it under the same folder right?
How have you structured both your Rails code and belonging business documents?
Share your solutions!
If you're deploying with capistrano, as a lot of Rails apps are, the standard solution seems to be to keep these sorts of assets within the shared folder, and then get cap to symlink them into the application at the point of deploy.
I have a bunch of images in SVN I would like to move out and put on S3. How have you dealt with keeping images out of your ruby on rails apps and out of SVN?
You don't have to put your images into your repository. You could still have them on your server, but it doesn't really matter where they're linked from. If you don't use any plugins to manage your assets then you can just remove them from your repository, upload all of them to S3 and update all links pointing to them.
If you do use some kind of plugin like paperclip or attachment_fu then you'll have to tell it where to find your files.