I have a ruby on rails project
I have some few thousand files associated with the site.
Is there a simple way to move all the files from virtual host to cloud (amazon).
Basically I have a lot of images and I am using paperclip gem to store the images.There are already a few users and have thousands of images . Now I have planned to migrate to cloud(amazon). I have used aws-s3 gem for this purpose and it works fine for new images.
But How do I put the old images which were already uploaded ?
Do I have to manually zip and reupload ? or is there a better way ?
Thank you
Hi I would right a simple rake task to accomplish this. It would look like something like this (not tested):
desc "port files to s3"
task :port => :environment do
AWS::S3::Base.establish_connection!(
:access_key_id => S3_CONFIG['access_key_id'],
:secret_access_key => S3_CONFIG['secret_access_key']
)
Images.all.each do |image|
new_image_path = "/images/#{image.id}/#{image.file_name_with_extension}"
AWS::S3::Object.store(new_image_path,open(image.current_path_to_image),S3_CONFIG['bucket_name'],:access => :public_read)
end
end
A few notes:
if you are using paperclip to manage and access the files you will want to store them in s3 so paperclip can access them. To do that use Paperclip::Interpolations.interpolate method. like:
new_image_path = Paperclip::Interpolations.interpolate("/accounts/:account_id/images/:id/:style/:basename.:extension",image,:small)
And this goes without saying -- make sure you test this in a staging environment!
Good luck.
Related
I would like to migrate from Paperclip to Carrier Wave or Refile because of this. The solution written here is impressive, but strikes me as complex and perhaps brittle.
My Rails4 app has 100's of images in production that were uploaded with Paperclip. Files are stored on production server. I have looked for a complete set of steps to follow to migrate, but keep coming up empty.
Is there a set of steps one can follow that allows for migration without necessitating application code re-write?
Alternatively, is there another way to persist uploaded files in Paperclip when form validation fails?
What am I missing here?
UPDATE:
Tried the solution detailed here by https://stackoverflow.com/users/646389/galatians . My paperclip :path and :url interpolations make use of :id_partition. I don't see a way this can be reconciled with an uploaded Image that is staged, but not yet saved.
I migrated to Carrierwave. Here are the relevant stats:
Time to work on and fail at coding a solution for persistent files
across form reloading with Paperclip - 4 hours. See OP update for the issue I could not overcome.
Time to Migrate to carrierwave, adjust the relevant models, controllers, and forms, and test. - 2 hours. Not so bad.
This key info helped me adjust the paths correctly. Keeping the path identical was important to me for avoiding having to move images to a new location in production:
Carrierwave code for generating paperclip-like :path and :url info here.
Paperclip interpolation info here.
This link got me on the right track, although my default :path made use of :id_partition not :id.
UPDATE:
Migration breaks this paradigm:
#protocol.images.each do |i|
tmp=i.dup
tmp.avatar = File.open(i.avatar.current_path)
tmp.save!
#dest.images << tmp
end
See: Duplicating a record that contains a carrierwave avatar : Getting "can't convert nil into Integer" error
You don't need to do anything because the only data needed to get images is already stored in DB.
I just installed the gem asset_sync and I am trying to get set up with my AWS account. When I run bundle exec rake assets:precompile I get the following errror:
AssetSync::Config::Invalid: Fog provider can't be blank, Fog directory can't be blank
I understand the simply reason that I am getting this error, namely that I havent pushed the Fog provider or directory to heroku. What I am stumped about is where to put the Following code (Taken from the Fog README). In config/initializers/fog.rb? Is this all I need to do to start using fog, other than installing the gem?
require 'rubygems'
require 'fog'
# create a connection
connection = Fog::Storage.new({
:provider => 'AWS',
:aws_access_key_id => YOUR_AWS_ACCESS_KEY_ID,
:aws_secret_access_key => YOUR_AWS_SECRET_ACCESS_KEY
})
# First, a place to contain the glorious details
directory = connection.directories.create(
:key => "fog-demo-#{Time.now.to_i}", # globally unique name
:public => true
)
not a problem, getting started tends to be the hardest part.
The answer is, it depends. I'd actually venture to say it would be best to put this in your environment based initializers, ie config/init/development or config/init/production, etc. Relatedly, you probably will not want to generate a new directory every time you start your app (there is an account level limit of 100 total I believe). So you might want to either set a key for each environment for that create or simply create the directory somewhere outside the initializers (and within the initializer you can assume it exists).
If you want to use that directory directly, you'll still need to create a reference, but you can create a local reference without making any api calls with #new like this:
directory = connection.directories.new(:key => ...)
As for asset_sync, it needs those keys and a reference to the directory key, which you will probably want to provide via ENV vars (to avoid checking your credentials into version control). You can find details on which keys and how to set them here: https://github.com/rumblelabs/asset_sync#built-in-initializer-environment-variables (the readme also describes how to do it via initializers, but that probably isn't the best plan).
Hope that helps!
I'm trying to figure out how to pull data from a database without the need to place a connection string at the top of each ruby file.
I'm learning the basics of ruby via a little Sinatra app I'm putting together which pulls data from a MSSQL database.
So far I've managed to create various simple erb pages that display data from the MSSQL database by using the following code structure at the top of each file:-
<% client = TinyTds::Client.new(:username => 'name', :password => 'password', :dataserver => 'hostname', :database => 'database') %>
<% data = client.execute("SELECT * from tablename") %>
From the books, guides and online tutorials I've found based on lots of configs to do with PostgreSQL or MySQL databases it seems to me I need to be creating a central file to store my connection data (such as a database.yml file) and then referencing that somewhere/somehow in my app.
Would that be correct, and should I be doing that in my main.rb file so that each of my .erb files do not require the connection string or do I have to still refer to the database in each .erb file also?
I've noted references to creating database config variables such as:-
db_config = YAML.load(File.Open("/path_to_file/database.yml")) [ENV['RAILS_ENV']]
but that clearly seems suited to Rails apps.
Could I do something similar for my sinatra driven 'app'?
Thanks.
This should work:
require "sinatra"
require "sinatra/config_file"
config_file 'path/to/config.yml'
DB = TinyTds::Client.new(
:username => settings.name,
:password => settings.password,
:dataserver => settings.hostname,
:database => settings.database
)
get '/' do
#result = DB.do_stuff
haml :index
end
What I would suggest though, is that you look for an ORM that supports TinyTDS and use that to set up the database connection and run queries. I use Sequel a lot and I know it supports TinyTDS, but I'm sure others do too.
I'd also suggest not putting things like database settings in a file that gets checked in to source control, as it's sensitive information. I prefer to put this kind of thing into environment variables for production, and use a config file to load the environment variables in development (for convenience). Using the above example:
DB = TinyTds::Client.new(
:username => ENV["name"],
:password => ENV["password"],
:dataserver => ENV["hostname"],
:database => ENV["database"]
)
Those env variables are loaded into the production server's memory, which makes them a little bit more secure. For development, I load YAML files before starting up Sinatra, adding each value to an ENV var. I'm not suggesting this is a standard way, but it's how I do it.
I just upgraded to Rails 3.2 and I'm using Amazon S3 with Paperclip to upload photos to my app.
Before my Image Urls would be:
http://s3.amazonaws.com/dealphotos.website.com/photos/428/large/Sandisk120Drive?1334754504
Now my Image Urls on Localhost are:
http://s3.amazonaws.com/dealphotos.website.com/deals/photos/000/000/428/large/Sandisk120Drive?1334754504
Notice the additional 000/000's - even if I take them out and visit the link it says:
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>"Long String of Numbers"</RequestId>
<HostId>
"Gives me a really long string of letters and numbers"
</HostId>
My Deal Model:
has_attached_file :photo, :styles => { :small =>"268x160>", :large =>"350x250>" },
:storage => :s3,
:bucket => 'dealphotos.website.com',
:s3_credentials => {
:access_key_id => ENV['S3_KEY_SPICY'],
:secret_access_key => ENV['S3_SECRET_SPICY']
}
And the images dont show up!
Before all I had to do was pull from Heroku and all my images and files would go to my development. What's going on?
You are most likely using the latest version of Paperclip (version >= 3).
Since this is a major version of paperclip there are documented incompatibilities with older versions of paperclip.
One MAJOR (and in my books good) change is that the default storage location of assets has changed.
Earlier it used to be /system/:attachment/:id/:style/:filename.:extension (from memory). This would be disastrous if you had 2 attachments with the same name (Company logo and Product logo) for example.
Now the new path used by paperclip is /system/:class/:attachment/:id_partition/:style/:filename.:extension
This means all your assets will be stored in the directory specified by above. :id_partitions are used so that the number of nodes (files/directories) in one directory doesnt exceed 1000.
I hope i've been able to explain the reason WHY you are seeing the problem. The default path has changed.
You have 2 options now -
Move old assets into the correct place in the new directory scheme (painful but recommended).
Move new assets into the old directory structure and add a :url + :path option in your attachment definition in the model to continue using the older scheme. (See https://github.com/thoughtbot/paperclip/blob/master/UPGRADING ).
Since it appears that you have images in BOTH the new structure and the old one. So no matter what you decide files will need to be moved from one structure to another. Unless the assets are bookmarked in some way, I suggest you move the older assets into the new directory structure.
Check out CarrierWave gem to file uploads. It works with 3.2 without any problems ;)
http://railscasts.com/episodes/253-carrierwave-file-uploads
I have many credentials that I have to handle in order to hook my app up to amazon s3 and other services.
I got my heroku app up and running with s3, and it works great. I defined my s3 access credentials following this example: http://devcenter.heroku.com/articles/config-vars
However, I want now to be able to have access to s3 from my local development environment. Obviously, the config vars that I defined on heroku aren't available on my localhost. How can I define these keys locally? Also, I'm looking in particular for a solution that is secure (for example if I define my keys in plain text in an intializer or something, I don't want that file to be pushed on heroku).
For background, here is what I add to my model to get paperclip running with s3
has_attached_file :photo,
:storage => :s3,
:bucket => 'bucket_name',
:s3_credentials => {
:access_key_id => ENV['S3_KEY'],
:secret_access_key => ENV['S3_SECRET']
}
The best place to define stuff like this, if you don't want it shared, is probably an initializer.
# config/initializers/s3_constants.rb
if Rails.env.development?
S3_KEY = "mys3key"
S3_SECRET = "mys3secret"
end
Ensure this file is added to .gitignore so it won't be pushed along with the rest of your repository.
Realistically speaking, constants that differ on a per-environment basis should really be located in the file for that environment (say development.rb here)... but those files should also really be added to your version control system, and if you definitely, definitely want this data excluded from git, then a separate file that you do not commit is probably your best bet.
Just define the environment variables in your .bash_profile file like any other environment variable. Maybe leave a comment to demarcate the section as Rails-specific environment variables.
#~/.bash_profile
# Rails constants
S3_KEY="blady"
S3_SECRET="bladybloo123"
Also, maybe you want to change the name to something more specific so that you can have more than one s3 connection defined...
heroku provides heroku config:add and you provide KEY=value. see config vars documentation