I deployed a Sinatra Ruby app with nginx and unicorn and I'm getting a "failed to allocate memory" when uploading large files (About 300MB or more). I'm currently using paperclip to upload images along with with datamapper(MySQL). Client-side, I'm using jQuery-file-upload by blueimp. My server is the cheapest digital ocean machine, which has 512MB RAM, but I don't understand why this process is taking up my RAM (Don't uploads get stored by parts in a temp file and then moved to the final destination?). I want to understand how the upload process works and how to get around this problem in my current 512MB RAM server. I hope you guys can help me.
Upload code:
post '/client/create/media' do
file_hash = params["files"][0]
title = params[:title][0]
description = params[:description][0]
if (fileAllowed())
paperclip_hash = make_paperclip_hash(file_hash)
begin
mediaObject = Mediaobject.create(:file => paperclip_hash, :client => Client.get(session[:id]),
:random => SecureRandom.hex(16), :name => title[0..49], :type => type.to_s,
:description => description)
rescue Exception => e
puts e.message
end
else
#nothing
end
end
MediaObject class:
class Mediaobject
include DataMapper::Resource
include Paperclip::Resource
property :id, Serial
property :name, String, :length => 64
property :type, String
property :description, Text
property :random, String, :length => 32
belongs_to :client
has n, :containers, :through => Resource
has_attached_file :file,
:url => "/client/media/:id",
:path => "#{Dir.pwd}/media/:userid/:random-:id.:extension"
end
The error occurs when creating the MediaObject instance and it says "failed to allocate memory" only.
Thanks.
I decided to do the upload manually, without paperclip, and it worked just fine. So paperclip seems to be the problem here.
Related
Update:
So this is definitely related to the :hash_data option, specifically the :updated_at segment. Somehow the files are being saved to the S3 bucket with a different :updated_at value than Paperclip uses to read the file. Could this be due to some race condition, considering that it occurs intermittently? As I mentioned below, this issue began after upgrading Paperclip to 4.2.1.
I will greatly appreciate any thoughts/tips you guys have on this. Thank you!
When uploading images via Paperclip to S3 bucket, it sometimes saves the files with a different filename than that returned by the attachment#url method. For example, an image is saved to
main_event_photos_46_47fd4f3c2fea17fbb7a0bd27c648911557f9e12b_main.png
However calling #event.main_event_photo.url(:main) returns
main_event_photos_46_15744de74a36207b672356b5ad4c6b30eb4ba85f_main.png
So as you can see, the :hash section of the interpolation does not match, and I have no way of finding the actual url besides opening the bucket in the S3 console. This issue seems to occur about half the time. Sometimes uploading the exact same file does save properly, and the url method accesses it correctly.
This issue began occurring after we upgraded Rails/Ruby/Paperclip. We're now using:
Ruby 2.1.5
Rails 4.2.0
Paperclip 4.2.1
Note that on development, files always save correctly (local filesystem). I have scoured Stackoverflow and Google to no avail. Please let me know if I can provide any additional information. Thank you!
EDIT:
Model:
has_attached_file :main_event_photo, {
:styles => { :original => {:geometry => "1280x800#", :format => 'png'},
:main => {:geometry => "640x400#", :format => 'png'},
:thumb => {:geometry => "330x220#", :format => 'png'}
},
:convert_options => {:original => '-quality 80',
:main => '-quality 80',
:thumb => '-quality 80'
},
:default_style => :main
}.merge!(PAPERCLIP_STORAGE_OPTIONS) # this is defined in the config/environments
validates_attachment_content_type :main_event_photo, :content_type => ['image/jpeg', 'image/png', 'image/gif', 'image/x-png', 'image/pjpeg']
validates_attachment_presence :main_event_photo
Form (basically):
<%= simple_form_for(#event, :url => { :action => #event.id.nil? ? "create" : "update" }) do |f| %>
<%= f.file_field :main_event_photo %>
<% end %>
Note we have many models with Paperclip attachments, and the issue occurs on each.
So this turned out to be the result of a bug. I upgraded Rails to 4.2.1.rc1 which was released last week, and the issue was resolved. If anyone wants more information, check out the thread on Github: https://github.com/thoughtbot/paperclip/issues/1772. It includes a workaround for those who can't upgrade Rails.
I am using ck editor in rails having database mongo db. I followed the link https://github.com/galetahub/ckeditor . I am succes in doing work with the help of ckeditor.
since my view.html.erb code is like this
<%= f.cktext_area :description, :toolbar => 'Easy', :width => 800, :height => 200 %><br>
and my show page is
<%= raw#department.description %>
it does not works for file cases.
I have my model attachment_file.rb is
class Ckeditor::AttachmentFile < Ckeditor::Asset
has_mongoid_attached_file :data,
:url => "/ckeditor_assets/attachments/:id/:filename",
:path => ":rails_root/public/ckeditor_assets/attachments/:id/:filename"
validates_attachment_size :data, :less_than => 100.megabytes
validates_attachment_presence :data
def url_thumb
#url_thumb ||= Ckeditor::Utils.filethumb(filename)
end
end
It is working for image cases but not working for zip file or any attachement. when it comes to the file cases it can upload file successfully with its path. but to download that file by user it doesnot work. I mean backend works properly for all features. But lacks to download that uploaded file stops by
`javascript:void(0)/*130*/
i have found the answer of this problem . First run this in terminal.
$ sudo chmod -R 777 /usr/share/ruby-rvm/gems/ruby-1.9.3-p194/gems/ckeditor-3.7.1
follow this path in your computer since i am using linux and my gem file locates here.
/usr/share/ruby-rvm/gems/ruby-1.9.3-p194/gems/ckeditor-3.7.1/vendor/assets/javascripts/ckeditor/plugins/attachment/dialogs
and open attachement.js file and edit it with the code that u find from tha above link.
click
Now ck editor will works for file attachment also.
Seem like you encounter this bug in CKeditor:
It sugests adding before filter as fix eg:
# app/model/department.rb
before_save :fix_ckeditor_attachment_paths
def fix_ckeditor_attachment_paths
if self.description.index(/_cke_saved_href/)
self.description = self.body.gsub(/_cke_saved_href/, 'href')
end
end
This spec passes just fine on my CentOS development box, but fails when run on my Macbook (10.7).
app/models/issue.rb
class Issue < ActiveRecord::Base
has_attached_file :cover_image,
:styles => {
:thumbnail => ["80x80>"],
:large => ["600x600>"]
},
:path => ":rails_root/public/images/issues/:id/:style_:basename.:extension",
:url => "issues/:id/:style_:basename.:extension"
validates_attachment_content_type :cover_image, :content_type => ['image/jpeg', 'image/png']
attr_protected :cover_image_file_name, :cover_image_content_type, :cover_image_file_size
end
spec/models/issue_spec.rb
require 'spec_helper'
describe Issue do
it 'should not accept invalid image types' do
%w[spec/factories/assets/docs/image.jpg spec/factories/assets/docs/image.txt].each do |file_path| #image.jpg is a text file with a .jpg extension
issue = Factory.build(:issue, :cover_image => File.new(Rails.root + file_path))
issue.should have(2).errors_on(:cover_image)
end
end
end
On my CentOS development box, I get no failures when I run the test. On my Macbook I get this error:
Failure/Error: issue.should have(2)errors_on(:cover_image)
expected 2 errors on :cover_image, got 0
On both machines if I try to create an issue by uploading one of the invalid image types in a browser, it fails with the error message I expect and passes if I use a valid image type.
paperclip-2.4.2
rspec-2.6.0
factory_girl_rails-1.2.0
CentOS: ImageMagick-6.2.8 (yum package)
Mac OS X: ImageMagick-6.7.2-0 (Macports)
It seems my whole approach to testing paperclip was off.
Paperclip Shoulda Matchers are working on both machines.
This is an odd question, but only commensurate with the strange behavior I'm seeing. My app is Rails 3/Paperclip/S3.
Symptoms:
All images are uploading regardless of their title.
When uploading a .pdf or .doc, if the title has no spaces like my_doc.pdf, it uploads fine.
When uploading a .pdf or .doc with spaces, such as My Doc.pdf, it fails, either with error broken pipe or by the file silently failing to upload to S3.
When uploading a .pdf or .doc with numbers, such as mydoc20.pdf, it also fails as above.
I imagine there are two possible solutions to this problem.
Fix the broken pipe error directly (preferred method).
Automatically rename every uploaded file to remove spaces and numbers before it is saved to S3 - while not fixing it at the source, I imagine this would allay the issue.
I would greatly appreciate any help you can give me in fixing 1 and/or 2.
Code
# Upload.rb model
class Upload < ActiveRecord::Base
has_attached_file :document,
:storage => :s3,
:s3_credentials => "#{::Rails.root.to_s}/config/s3.yml",
:path => "/docs/:style/:id/:basename.:extension"
has_attached_file :photo,
:styles => {:medium => "200x300>", :thumb => "100x150>" },
:storage => :s3,
:s3_credentials => "#{::Rails.root.to_s}/config/s3.yml",
:path => "/photos/:style/:id/:basename.:extension"
# s3.yml
development:
bucket: dev_bucket_name
access_key_id: dev_acc_key
secret_access_key: dev_sec_key
production:
bucket: my_production_bucket
access_key_id: my_access_key_id
secret_access_key: my_secret_key
# environment.rb is empty with regard to uploading.
# uploads_controller.rb
def edit
#candidate = Candidate.find(current_user.user_type_id)
render :layout => 'forms'
end
def update
#candidate = Candidate.find(params[:id])
if #candidate.update_attributes(params[:candidate])
flash[:notice] = "Profile updated successfully."
redirect_to :action => "show", :id => params[:id]
else
flash[:notice] = "There was an error updating your profile."
render :action => "edit", :id => params[:id]
end
end
I don't believe there are any methods involved. I almost hope there is something obviously wrong with my approach because that means it'll get fixed :).
For part two this should do it:
#s = "Really Important!*() Document version#123123.newest.pdf"
#s.gsub!(' ','_').downcase! #this will make everything lowercase and replace all spaces with underscores
#s.gsub!(/[^a-zA-Z._]+/,'') #this will remove all numbers and special characters except . and _
puts #s #prints "really_important_document_version.newest.pdf"
Edit: After some more research into paperclip I found the following: http://blog.wyeworks.com/2009/7/13/paperclip-file-rename
Check that link out, I believe it is what you are looking for.
Edit 2: In my initial read of your post I missed the part about pulling out numbers as well, I have modified the regulat expression code to account for that.
I'm having a ton of trouble with uploading multiple attachments with paperclip and processing them with a watermark.
I have 2 models, Ad and Photo.
The Ad has_many :photos and the Photo belongs_to :ad.
To be more exact in /models/ad.rb I have:
class Ad < ActiveRecord::Base
has_many :photos, :dependent => :destroy
accepts_nested_attributes_for :photos, :allow_destroy => true
end
and my photo.rb file looks like this:
class Photo < ActiveRecord::Base
belongs_to :ad
has_attached_file :data,
:styles => {
:thumb => "100x100#",
:first => {
:processors => [:watermark],
:geometry => '300x250#',
:watermark_path => ':rails_root/public/images/watermark.png',
:position => 'SouthEast' },
:large => {
:processors => [:watermark],
:geometry => '640x480#',
:watermark_path => ':rails_root/public/images/watermark.png',
:position => 'SouthEast' }
}
end
In my view I use this to add the file fields
<% f.fields_for :photos do |p| %>
<%= p.label :data, 'Poza:' %> <%= p.file_field :data %>
<% end %>
In my controller, in the edit action i use 4.times {#ad.photos.build} to generate the file fields.
It all works fine and dandy if I don't use the watermark processor, if I use a normal has_attached_file declaration, like this:
has_attached_file :data,
:styles => {
:thumb => "100x100#",
:first => '300x250#',
:large => '640x480#'
}
But when I use the watermark processor I always get this error:
NoMethodError in PublicController#update_ad
You have a nil object when you didn't expect it!
You might have expected an instance of ActiveRecord::Base.
The error occurred while evaluating nil.[]
..............................
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/nested_attributes.rb:350:in `assign_nested_attributes_for_collection_association'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/nested_attributes.rb:345:in `each'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/nested_attributes.rb:345:in `assign_nested_attributes_for_collection_association'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/nested_attributes.rb:243:in `photos_attributes='
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2746:in `send'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2746:in `attributes='
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2742:in `each'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2742:in `attributes='
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2628:in `update_attributes'
/home/alexg/Sites/vandmasina/app/controllers/public_controller.rb:217
/home/alexg/Sites/vandmasina/app/controllers/public_controller.rb:216:in `update_ad'
The parameters are ok, as far as I can say
Parameters:
{"commit"=>"Salveaza modificarile",
"ad"=>{"price"=>"6000",
"oras"=>"9",
"photos_attributes"=>{"0"=>{"data"=>#<File:/tmp/RackMultipart20100928-5130-b42noe-0>},
"1"=>{"data"=>#<File:/tmp/RackMultipart20100928-5130-r0ukcr-0>},
"2"=>{"data"=>#<File:/tmp/RackMultipart20100928-5130-mb23ei-0>},
"3"=>{"data"=>#<File:/tmp/RackMultipart20100928-5130-1bpkm3b-0>}},
The Watermark processor /lib/paperclip_processors/watermark.rb looks like this:
module Paperclip
class Watermark < Processor
class InstanceNotGiven < ArgumentError;
end
def initialize(file, options = {},attachment = nil)
super
#file = file
#current_format = File.extname(#file.path)
#basename = File.basename(#file.path, #current_format)
#watermark = ':rails_root/public/images/watermark.png'
#current_geometry = Geometry.from_file file # This is pretty slow
#watermark_geometry = watermark_dimensions
end
def watermark_dimensions
return #watermark_dimensions if #watermark_dimensions
#watermark_dimensions = Geometry.from_file #watermark
end
def make
dst = Tempfile.new([#basename, #format].compact.join("."))
watermark = " \\( #{#watermark} -extract #{#current_geometry.width.to_i}x#{#current_geometry.height.to_i}+#{#watermark_geometry.height.to_i /
2}+#{#watermark_geometry.width.to_i / 2} \\) "
command = "-gravity center " + watermark + File.expand_path(#file.path) + " " +File.expand_path(dst.path)
begin
success = Paperclip.run("composite", command.gsub(/\s+/, " "))
rescue PaperclipCommandLineError
raise PaperclipError, "There was an error processing the watermark for #{#basename}" if #whiny_thumbnails
end
dst
end
end
end
I have tried the processor in a normal app, without multiple attachments and it works perfect. It doesn't work with nested_attributes as far as I can tell.
The app is rails 2.3.5 with ruby 1.8.7 and paperclip 2.3.11
If you can provide any help it would be appreciated a lot, since I've been trying to figure this out for 2 days now :)
If you use rails 3 and paperclip > 2.3.3, try https://gist.github.com/843418 source.
Oh, man, that was a tough one!
You have few errors in your code and none is related to nested models. My explanation is for Rails 3 & Paperclip 2.3.3
a) the :rails_root thing doesn't work. This interpolation is used only in url/path and not on custom options. So you should replace it with Rails.root.join("public/images...")
b) you simply ignore the :watermark_path option and you use only hardcoded path (in initialization method). So it doesn't matter what you have in your :styles as it always go for .../images/watermark.png. The :rails_root thingy there again so it cannot work.
c) when you pass a parameter to Paperclip.run("composite", "foo bar there") it actually executes this command:
composite 'foo bar there'
can you see the single quotes? Because of that the composite command see your parameters as one huge parameter and doesn't understand it at all. If you pass it as an array, then every item is enclosed in the quotes, not the array as a whole.
So here is the improved version of watermark processor:
module Paperclip
class Watermark < Processor
class InstanceNotGiven < ArgumentError;
end
def initialize(file, options = {},attachment = nil)
super
#file = file
#current_format = File.extname(#file.path)
#basename = File.basename(#file.path, #current_format)
# PAWIEN: use default value only if option is not specified
#watermark = options[:watermark_path] || Rails.root.join('public/images/watermark.png')
#current_geometry = Geometry.from_file file # This is pretty slow
#watermark_geometry = watermark_dimensions
end
def watermark_dimensions
return #watermark_dimensions if #watermark_dimensions
#watermark_dimensions = Geometry.from_file #watermark
end
def make
dst = Tempfile.new([#basename, #format].compact.join("."))
dst.binmode
begin
# PAWIEN: change original "stringy" approach to arrayish approach
# inspired by the thumbnail processor
options = [
"-gravity",
"center",
"#{#watermark}",
"-extract",
"#{#current_geometry.width.to_i}x#{#current_geometry.height.to_i}+#{#watermark_geometry.height.to_i / 2}+#{#watermark_geometry.width.to_i / 2}",
File.expand_path(#file.path),
File.expand_path(dst.path)
].flatten.compact.join(" ").strip.squeeze(" ")
success = Paperclip.run("composite", options)
rescue PaperclipCommandLineError
raise PaperclipError, "There was an error processing the watermark for #{#basename}" if #whiny_thumbnails
end
dst
end
end
end
Hope that helped you!
UPDATE: You need to use latest paperclip gem from github
gem 'paperclip', '>= 2.3.3', :git => "http://github.com/thoughtbot/paperclip.git"
In this version the formats of #run were changed once again so I've updated the code. It really should work as I've created test application and it's doing what supposed.
UPDATE 2: Repo with working example:
git://repo.or.cz/paperclip-mass-example.git
Just from a quick glance it looks like watermark_path should be "#{Rails.root}/..." though it looks like you have a lot going on here.
Also, I don't see your form as in form_for. Make sure you have {:multipart => true}