I am using Paperclip (w/ Amazon s3) on Rails 3. I want to attach a new file to my model without replacing the old file. I don't want the old file to be accessible, I only want to have it there on s3 as a back up.
Do you know if there is a way of telling paperclip to take care of it, itself?
in post.rb I have:
has_attached_file :sound,
:storage => :s3,
:s3_credentials => "....",
:styles => {:mp3 => {:format => :mp3}},
:processors => [:sound_processor],
:s3_host_alias => '....',
:bucket => '....',
:path => ":attachment/:id/:style/out.:extension",
:url => ":s3_alias_url"
and the processor is as follows:
class Paperclip::SoundProcessor < Paperclip::Processor
def initialize file, options = {}, attachment = nil
super
#format = options[:format] || "mp3"
#current_format = File.extname(#file.path)
#basename = File.basename(#file.path, #current_format)
end
def make
src = #file
dst = Tempfile.new([#basename,".#{#format}"])
dst.binmode
cmd = "ffmpeg -y -ab 128k -t 600 -i #{File.expand_path(src.path)} #{File.expand_path(dst.path)}"
Paperclip.log(cmd)
out = `#{cmd}`
raise Paperclip::PaperclipError, "processor does not accept the given audio file" unless $?.exitstatus == 0
dst
end
end
Here's what I do. I timestamp the filename before saving it (to prevent another file with the same name overwriting the original), and force paperclip to never delete anything.
before_create :timestamp_filename
def timestamp_filename
fname = Time.now.to_s(:db).gsub(/[^0-9]/,'') + '_' + sound_file_name
sound.instance_write(:file_name, fname)
end
# override paperclip's destroy files
# method to always keep them around
def destroy_attached_files
true
end
It's quite straightforward to add versioning to your models, recently I user CarrierWave and paper_trail in combo to achieve this. We were deploying to Heroku so S3 was in the mix too.
The reason I'm posting this as an answer is because, although controversial, I don't think libraries like PaperClip should support backing up files, a library specific to solve that problem feels better to me personally.
Here is a good resource that you can take a look at: http://eggsonbread.com/2009/07/23/file-versioning-in-ruby-on-rails-with-paperclip-acts_as_versioned/
It creates new versions of a file. Hope this helps.
Related
Setup:
ruby 1.8.6
rails 2.2.2
attachment_fu - not sure (it's vendorized), but last entry in CHANGELOG is "Apr 17 2008"
aws-s3 (0.6.3)
aws-sdk (2.1.13)
aws-sdk-core (2.1.13)
aws-sdk-resources (2.1.13)
I have a model which uses attachment_fu like so:
has_attachment :storage => :s3,
:path_prefix => "vip_uploads#{("_"+RAILS_ENV) if RAILS_ENV != "production"}",
:max_size => 100.megabytes,
:processor => :mp3
The s3 stuff is all set up fine - if i remove the processor option then the upload to s3 works.
My mp3 processor, which converts wav files to mp3s, looks like this:
module Technoweenie # :nodoc:
module AttachmentFu # :nodoc:
module Processors
module Mp3Processor
def self.included(base)
base.send :extend, ClassMethods
base.alias_method_chain :process_attachment, :processing
end
module ClassMethods
end
protected
def process_attachment_with_processing
self.convert_to_mp3
end
# Convert to mp3 and set some metadata
def convert_to_mp3(options={})
#do the conversion with ffmpeg
mp3_temp_path = "#{self.temp_path}.mp3"
cmd = "ffmpeg -i #{self.temp_path} -metadata title=\"#{self.filename.gsub("\.wav","")}\" -metadata artist=\"Vip Studio Sessions\" -metadata album=\"Uploads\" -vn -ar 44100 -ac 2 -ab 320k -f mp3 #{mp3_temp_path}"
`#{cmd}`
#copy this file back into temp_data
self.copy_to_temp_file(mp3_temp_path)
#update attributes
self.filename = self.filename.gsub(/\.wav$/,".mp3")
self.content_type = "audio/mpeg"
self.set_size_from_temp_path
end
end
end
end
end
All of the conversion stuff seems to be working, in that it makes a new mp3 file in the tmp folder, with the filename saved in mp3_temp_path, and it makes a record in the database. But for some reason, the resulting file isn't then pushed up to s3. I've got a feeling i just need to set some accessor to do with temp_data or temp_file or something. I've tried this
self.temp_path = mp3_temp_path
and
self.temp_data = File.read(mp3_temp_path)
and
self.temp_path = write_to_temp_file(File.read(mp3_temp_path))
Currently, as you can see in my code, i'm trying this:
self.copy_to_temp_file(mp3_temp_path)
but none of them work. These attempts were based on looking in the pre-existing processors, eg for rmagick, and seeing what they do. They seem to do the same thing as me, but since all of them are about thumbnailing it's easy to lose something in the translation.
Can anyone see what i'm missing? thanks, Max
Ok, answering my own question - it's amazing what a new day can bring. It suddenly occurred to me to look more closely at the attachment_fu source code, in which i noticed this...
def after_process_attachment
if #saved_attachment
if respond_to?(:process_attachment_with_processing) && thumbnailable? && !attachment_options[:thumbnails].blank? && parent_id.nil?
temp_file = temp_path || create_temp_file
attachment_options[:thumbnails].each { |suffix, size| create_or_update_thumbnail(temp_file, suffix, *size) }
end
save_to_storage
#temp_paths.clear
#saved_attachment = nil
callback :after_attachment_saved
end
end
I hadn't set this instance variable #saved_attachment to true (or truthy), so it was not going ahead with save_to_storage. It kind of fails silently - the record is still happily saved to the database , it just doesn't store the file.
in summary, the answer was that, if i'm happy with the resulting mp3 (ie it was an mp3 in the first place, or I successfully convert it to an mp3), then i need to set #saved_attachment = true in my processor.
I'm having an invalid encoding error that doesn't let me save the image to a carrierwave uploader.
require 'rqrcode_png'
img = RQRCode::QRCode.new( 'test', :size => 4, :level => :h ).to_img.to_s
img.valid_encoding?
=> false
I'm not sure if this is what you're looking for, in my case I needed to associate the generated QR code with a Rails model using carrierwave, what I ended up doing was saving the image to a temp file, associating that file with the model and afterwards deleting the temp file, here's my code:
def generate_qr_code!
tmp_path = Rails.root.join('tmp', "some-filename.png")
tmp_file = RQRCode::QRCode.new(self.hash_value).to_img.resize(200,200).save(tmp_path)
# Stream is handed closed, we need to reopen it
File.open(tmp_file.path) do |file|
self.qr_code = file
end
File.delete(tmp_file.path)
self.save!
end
Is there a recommended technique for migrating a large set of paperclip S3 images to a new :url and :path format?
The reason for this is because after upgrading to rails 3.1, new versions of thumbs are not being shown after cropping (previously cached version is shown). This is because the filename no longer changes (since asset_timestamp was removed in rails 3.1). I'm using :fingerprint in the url/path format, but this is generated from the original, which doesn't change when cropping.
I was intending to insert :updated_at in the url/path format, and update attachment.updated_at during cropping, but after implementing that change all existing images would need to be moved to their new location. That's around half a million images to rename over S3.
At this point I'm considering copying them to their new location first, then deploying the code change, then moving any images which were missed (ie uploaded after the copy), but I'm hoping there's an easier way... any suggestions?
I had to change my paperclip path in order to support image cropping, I ended up creating a rake task to help out.
namespace :paperclip_migration do
desc 'Migrate data'
task :migrate_s3 => :environment do
# Make sure that all of the models have been loaded so any attachments are registered
puts 'Loading models...'
Dir[Rails.root.join('app', 'models', '**/*')].each { |file| File.basename(file, '.rb').camelize.constantize }
# Iterate through all of the registered attachments
puts 'Migrating attachments...'
attachment_registry.each_definition do |klass, name, options|
puts "Migrating #{klass}: #{name}"
klass.find_each(batch_size: 100) do |instance|
attachment = instance.send(name)
unless attachment.blank?
attachment.styles.each do |style_name, style|
old_path = interpolator.interpolate(old_path_option, attachment, style_name)
new_path = interpolator.interpolate(new_path_option, attachment, style_name)
# puts "#{style_name}:\n\told: #{old_path}\n\tnew: #{new_path}"
s3_copy(s3_bucket, old_path, new_path)
end
end
end
end
puts 'Completed migration.'
end
#############################################################################
private
# Paperclip Configuration
def attachment_registry
Paperclip::AttachmentRegistry
end
def s3_bucket
ENV['S3_BUCKET']
end
def old_path_option
':class/:id_partition/:attachment/:hash.:extension'
end
def new_path_option
':class/:attachment/:id_partition/:style/:filename'
end
def interpolator
Paperclip::Interpolations
end
# S3
def s3
AWS::S3.new(access_key_id: ENV['S3_KEY'], secret_access_key: ENV['S3_SECRET'])
end
def s3_copy(bucket, source, destination)
source_object = s3.buckets[bucket].objects[source]
destination_object = source_object.copy_to(destination, {metadata: source_object.metadata.to_h})
destination_object.acl = source_object.acl
puts "Copied #{source}"
rescue Exception => e
puts "*Unable to copy #{source} - #{e.message}"
end
end
Didn't find a feasible method for migrating to a new url format. I ended up overriding Paperclip::Attachment#generate_fingerprint so it appends :updated_at.
I have the following:
attachments.each do |a|
Rails.logger.info a.filename
tempfile = Tempfile.new("#{a.filename}", "#{Rails.root.to_s}/tmp/")
Rails.logger.info tempfile.path
end
Where attachments is from paperclip.
Here's the output:
billgates.jpg
/Users/bhellman/Sites/cline/tmp/billgates.jpg20101204-17402-of0u9o-0
Why is the file name getting 20101204-17402-of0u9o-0 appended to at the end? That's breaking everything with paperclip etc. Anyone seen this before? For the life of I have no idea what's doing it?
Thanks
UPDATE
Paperclip: Paperclip on github
a is the attachment file
tempfile = Tempfile.new("#{a.filename}", "#{Rails.root.to_s}/tmp/")
tempfile << a.body
tempfile.puts
attachments.build(
:attachment => File.open(tempfile.path)
)
best make sure your tempfile has the correct extension, saving you to trying and change it after:
file = Tempfile.new(['hello', '.jpg'])
file.path # => something like: "/tmp/hello2843-8392-92849382--0.jpg"
more here: http://apidock.com/ruby/v1_9_3_125/Tempfile/new/class
The first argument for Tempfile.new is just a basename. To make sure each Tempfile is unique the characters are appended to the end of the file.
You should use Paperclip's API for this:
tempfiles = []
attachments.each do |a|
# use Attachment#to_file to get a :filesystem => file, :s3 => tempfile
tempfiles << a.to_file
end
tempfiles.each do |tf|
Rails.logger.debug tf.filename
end
attachment = attachments.build(
:attachment => File.open(tempfile.path)
)
# change the displayed file name stored in the db record here
attachment.attachment_file_name = a.filename # or whatever else you like
attachment.save!
The best way I found to deal with this was to specify the file extension in the Paperclip attribute. For example:
has_attached_file :picture,
:url => "/system/:hash.jpg",
:hash_secret => "long_secret_string",
:storage => :s3,
:s3_credentials => "#{Rails.root}/config/s3.yml"
Note that the :url is declared as '.jpg' rather than the traditional .:extension.
Good luck!
I'm having a ton of trouble with uploading multiple attachments with paperclip and processing them with a watermark.
I have 2 models, Ad and Photo.
The Ad has_many :photos and the Photo belongs_to :ad.
To be more exact in /models/ad.rb I have:
class Ad < ActiveRecord::Base
has_many :photos, :dependent => :destroy
accepts_nested_attributes_for :photos, :allow_destroy => true
end
and my photo.rb file looks like this:
class Photo < ActiveRecord::Base
belongs_to :ad
has_attached_file :data,
:styles => {
:thumb => "100x100#",
:first => {
:processors => [:watermark],
:geometry => '300x250#',
:watermark_path => ':rails_root/public/images/watermark.png',
:position => 'SouthEast' },
:large => {
:processors => [:watermark],
:geometry => '640x480#',
:watermark_path => ':rails_root/public/images/watermark.png',
:position => 'SouthEast' }
}
end
In my view I use this to add the file fields
<% f.fields_for :photos do |p| %>
<%= p.label :data, 'Poza:' %> <%= p.file_field :data %>
<% end %>
In my controller, in the edit action i use 4.times {#ad.photos.build} to generate the file fields.
It all works fine and dandy if I don't use the watermark processor, if I use a normal has_attached_file declaration, like this:
has_attached_file :data,
:styles => {
:thumb => "100x100#",
:first => '300x250#',
:large => '640x480#'
}
But when I use the watermark processor I always get this error:
NoMethodError in PublicController#update_ad
You have a nil object when you didn't expect it!
You might have expected an instance of ActiveRecord::Base.
The error occurred while evaluating nil.[]
..............................
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/nested_attributes.rb:350:in `assign_nested_attributes_for_collection_association'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/nested_attributes.rb:345:in `each'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/nested_attributes.rb:345:in `assign_nested_attributes_for_collection_association'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/nested_attributes.rb:243:in `photos_attributes='
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2746:in `send'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2746:in `attributes='
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2742:in `each'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2742:in `attributes='
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2628:in `update_attributes'
/home/alexg/Sites/vandmasina/app/controllers/public_controller.rb:217
/home/alexg/Sites/vandmasina/app/controllers/public_controller.rb:216:in `update_ad'
The parameters are ok, as far as I can say
Parameters:
{"commit"=>"Salveaza modificarile",
"ad"=>{"price"=>"6000",
"oras"=>"9",
"photos_attributes"=>{"0"=>{"data"=>#<File:/tmp/RackMultipart20100928-5130-b42noe-0>},
"1"=>{"data"=>#<File:/tmp/RackMultipart20100928-5130-r0ukcr-0>},
"2"=>{"data"=>#<File:/tmp/RackMultipart20100928-5130-mb23ei-0>},
"3"=>{"data"=>#<File:/tmp/RackMultipart20100928-5130-1bpkm3b-0>}},
The Watermark processor /lib/paperclip_processors/watermark.rb looks like this:
module Paperclip
class Watermark < Processor
class InstanceNotGiven < ArgumentError;
end
def initialize(file, options = {},attachment = nil)
super
#file = file
#current_format = File.extname(#file.path)
#basename = File.basename(#file.path, #current_format)
#watermark = ':rails_root/public/images/watermark.png'
#current_geometry = Geometry.from_file file # This is pretty slow
#watermark_geometry = watermark_dimensions
end
def watermark_dimensions
return #watermark_dimensions if #watermark_dimensions
#watermark_dimensions = Geometry.from_file #watermark
end
def make
dst = Tempfile.new([#basename, #format].compact.join("."))
watermark = " \\( #{#watermark} -extract #{#current_geometry.width.to_i}x#{#current_geometry.height.to_i}+#{#watermark_geometry.height.to_i /
2}+#{#watermark_geometry.width.to_i / 2} \\) "
command = "-gravity center " + watermark + File.expand_path(#file.path) + " " +File.expand_path(dst.path)
begin
success = Paperclip.run("composite", command.gsub(/\s+/, " "))
rescue PaperclipCommandLineError
raise PaperclipError, "There was an error processing the watermark for #{#basename}" if #whiny_thumbnails
end
dst
end
end
end
I have tried the processor in a normal app, without multiple attachments and it works perfect. It doesn't work with nested_attributes as far as I can tell.
The app is rails 2.3.5 with ruby 1.8.7 and paperclip 2.3.11
If you can provide any help it would be appreciated a lot, since I've been trying to figure this out for 2 days now :)
If you use rails 3 and paperclip > 2.3.3, try https://gist.github.com/843418 source.
Oh, man, that was a tough one!
You have few errors in your code and none is related to nested models. My explanation is for Rails 3 & Paperclip 2.3.3
a) the :rails_root thing doesn't work. This interpolation is used only in url/path and not on custom options. So you should replace it with Rails.root.join("public/images...")
b) you simply ignore the :watermark_path option and you use only hardcoded path (in initialization method). So it doesn't matter what you have in your :styles as it always go for .../images/watermark.png. The :rails_root thingy there again so it cannot work.
c) when you pass a parameter to Paperclip.run("composite", "foo bar there") it actually executes this command:
composite 'foo bar there'
can you see the single quotes? Because of that the composite command see your parameters as one huge parameter and doesn't understand it at all. If you pass it as an array, then every item is enclosed in the quotes, not the array as a whole.
So here is the improved version of watermark processor:
module Paperclip
class Watermark < Processor
class InstanceNotGiven < ArgumentError;
end
def initialize(file, options = {},attachment = nil)
super
#file = file
#current_format = File.extname(#file.path)
#basename = File.basename(#file.path, #current_format)
# PAWIEN: use default value only if option is not specified
#watermark = options[:watermark_path] || Rails.root.join('public/images/watermark.png')
#current_geometry = Geometry.from_file file # This is pretty slow
#watermark_geometry = watermark_dimensions
end
def watermark_dimensions
return #watermark_dimensions if #watermark_dimensions
#watermark_dimensions = Geometry.from_file #watermark
end
def make
dst = Tempfile.new([#basename, #format].compact.join("."))
dst.binmode
begin
# PAWIEN: change original "stringy" approach to arrayish approach
# inspired by the thumbnail processor
options = [
"-gravity",
"center",
"#{#watermark}",
"-extract",
"#{#current_geometry.width.to_i}x#{#current_geometry.height.to_i}+#{#watermark_geometry.height.to_i / 2}+#{#watermark_geometry.width.to_i / 2}",
File.expand_path(#file.path),
File.expand_path(dst.path)
].flatten.compact.join(" ").strip.squeeze(" ")
success = Paperclip.run("composite", options)
rescue PaperclipCommandLineError
raise PaperclipError, "There was an error processing the watermark for #{#basename}" if #whiny_thumbnails
end
dst
end
end
end
Hope that helped you!
UPDATE: You need to use latest paperclip gem from github
gem 'paperclip', '>= 2.3.3', :git => "http://github.com/thoughtbot/paperclip.git"
In this version the formats of #run were changed once again so I've updated the code. It really should work as I've created test application and it's doing what supposed.
UPDATE 2: Repo with working example:
git://repo.or.cz/paperclip-mass-example.git
Just from a quick glance it looks like watermark_path should be "#{Rails.root}/..." though it looks like you have a lot going on here.
Also, I don't see your form as in form_for. Make sure you have {:multipart => true}