When I upload a file , and send this file to s3 ( 5 MB of csv file ) on heroku I got error H13 ( Connection Cloesed without response). I added a line ( environment.rb and application.rb ) lines:
if Rack::Utils.respond_to?("key_space_limit=")
Rack::Utils.key_space_limit = 262144 # 4 times the default size
end
but It did not help me.
My code is:
def import_from_csv(csv_file, user, organization, related_object, after_import= nil)
file_upload = FileUpload.upload_to_s3!(user, csv_file)
delay.process_csv(file_upload, related_object, organization, after_import)
end
Related
I'm writing an Rails app which could upload files to storage. Big files are splitted into chunks from client (with JS) and upload parts to server.
As in development, I could simply open existed file and write following bytes into that.
(I'm using CarrierWave gem)
File.open(#up_file.link.path, "ab") do |f|
f.write(up_file_params[:link].read)
end
# This code worked when I upload to '/public' folder in development
However, now I want to use a FTP server to storage files. But I can't Concatenate new bytes with existed bytes.
def get_ftp_connection # create a new FTP connection
ftp = Net::FTP.new
ftp.connect(ENV['ftp_host'], ENV['ftp_port'])
begin
ftp.passive = ENV['ftp_passive']
ftp.login(ENV['ftp_user'], ENV['ftp_passwd'])
yield ftp
ensure
ftp.quit
end
end
.....
def create
.....
get_ftp_connection #up_file do |ftp|
full_path = ::File.dirname "#{ENV['ftp_folder']}/#{#up_file.link.path}"
base_name = File.basename(#up_file.link.to_s)
ftp.chdir(full_path)
ftp.putbinaryfile(up_file_params[:link].read, base_name)
end
end
I got ArgumentError (string contains null byte): at putbinaryfile... , any help :(
As mentioned in the comments you could completely download the file first and then upload to the ftp server. If that's not an option for whatever reason, you could append to the remote file as it's being uploaded:
ftp.storbinary("APPE #{base_name}", up_file_params[:link], Net::FTP::DEFAULT_BLOCKSIZE)
I have a problem with my attachment system on web page. I store them on amazon S3 using paperclip. I have an option to copy attachment to new file. Everything works fine until there are polish special characters in title, like: ŁĄKA.jpg. Then I get an error:
Saving error: Appendix Paperclip::Errors::NotIdentifiedByImageMagickError
/Users/michal/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/activerecord-4.2.5/lib/active_record/validations.rb:79:in `raise_record_invalid'
/Users/michal/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/activerecord-4.2.5/lib/active_record/validations.rb:43:in `save!'
My code:
instance.appendixes.select {|a| a.temporary? && !a.appendix.exists?}.each do |a|
a.appendix = S3File.new(a.s3path)
a.process = false
a.appendix_url = nil
puts "CREATING NEW FILE from (temporary?) appendix: #{a.id}, path: #{a.s3path}, is_public: #{a.is_public}, determine_is_public: #{a.determine_is_public}"
a.is_public = a.determine_is_public
logger.debug("CREATING NEW FILE from (temporary?) appendix: #{a.id}, path: #{a.s3path}, is_public: #{a.is_public}, determine_is_public: #{a.determine_is_public}")
a.save! # bo delayed_job
end
I'm getting error on a.save! when path is like: appendixes/appendixes/242/original/%25C5%2581A%25CC%25A8KA.jpg, but works like charm when it is: appendixes/appendixes/243/original/laka.jpg or another file name without polish letters. Anybody had this kind of problem or have suggestions how to fix it?
Ok, I found what was wrong. I had to replace in a.s3path, the last part with original name (łąka.jpg) and everything works fine. So when I have:
S3File.new(appendixes/appendixes/243/original/łąka.jpg) it works good and finds the correct file on s3 server.
I'm attempting to download files from my S3 File server via a rails app that I've written. However, I'm having a difficult time trying to figure out how to accomplish this. I've been attempting to use this reference from Amazon's blog to get it working.
in the get method in my controller, I have the following:
asset = current_user.assets.find_by_id(params[:id])
File.open('filename', 'wb') do |file|
reap = s3.get_object({ bucket:'bucket-name', key: URI.encode(asset.uploaded_file.url)}, target: file)
end
however I'm getting the following error:
uninitialized constant AssetsController::s3
I'm using the gem aws-sdk. Any suggestions would be much appreciated.
uninitialized constant AssetsController::s3
You need to define s3, the below should work
asset = current_user.assets.find_by_id(params[:id])
File.open('filename', 'wb') do |file|
s3 = Aws::S3::Client.new
reap = s3.get_object({ bucket:'bucket-name', key: URI.encode(asset.uploaded_file.url)}, target: file)
end
I need to use the tmp folder on Heroku (Cedar) for writing some temporarily data, I am trying to do that this way:
open("#{Rails.root}/tmp/#{result['filename']}", 'wb') do |file|
file.write open(image_url).read
end
But this produce error
Errno::ENOENT: No such file or directory - /app/tmp/image-2.png
I am trying this code and it's running properly on localhost, but I cannot make it work on Heroku.
What is the proper way to save some files to the tmp directory on Heroku (Cedar stack)?
Thank you
EDIT:
I am running method with Delayed Jobs that needs to has access to the tmp file.
EDIT2:
What I am doing:
files.each_with_index do |f, index|
unless f.nil?
result = JSON.parse(buffer)
filename = "#{Time.now.to_i.to_s}_#{result['filename']}" # thumbnail name
thumb_filename = "#{Rails.root}/tmp/#{filename}"
image_url = f.file_url+"/convert?rotate=exif"
open("#{Rails.root}/tmp/#{result['filename']}", 'wb') do |file|
file.write open(image_url).read
end
img = Magick::Image.read(image_url).first
target = Magick::Image.new(150, 150) do
self.background_color = 'white'
end
img.resize_to_fit!(150, 150)
target.composite(img, Magick::CenterGravity, Magick::CopyCompositeOp).write(thumb_filename)
key = File.basename(filename)
s3.buckets[bucket_name].objects[key].write(:file => thumb_filename)
# save path to the new thumbnail to database
f.update_attributes(:file_url_thumb => "https://s3-us-west-1.amazonaws.com/bucket/#{filename}")
end
end
I have in database information about images. These images are stored in Amazon S3 bucket. I need to create thumbnails to these images. So I am going through one image by another one, load the image, temporarily save it, then resize it and afterwards I will upload this thumbnail to S3 bucket.
But this procedure doesn't seems to be working on Heroku, so, how could I do that (my app is running on Heroku)?
Is /tmp included in your git repo? Removed in your .slugignore? The directory may just not exist out on Heroku.
Try tossing in a quick mkdir before the write:
Dir.mkdir(File.join(Rails.root, 'tmp'))
Or even in an initializer or something...
Here's an elegant way
f = File.new("tmp/filename.txt", 'w')
f << "hi there"
f.close
Dir.entries(Dir.pwd.to_s + ("/tmp")) # See your newly created file in /tmp
Don't forget that whenever your app restarts (for any reason, including those outside your control), your files will be deleted, as they are only stored ephemerally.
Try it with heroku restart, you will see the new file you created is no longer there
I have ~16,000 images I'm trying to upload to Amazon. Right now, they're on my local file system. I'd like to upload them to S3 using Paperclip, but I do NOT want to upload them to my server first. I'm using Heroku and they limit slug size.
Is there a way to use a rake task to upload the images directly from my local file system to S3 via Paperclip?
You can configure your app to use Amazon S3 for paperclip storage in development (see my example) and upload the files using a rake task like this:
Lets's say your folder of images was in your_app_folder/public/images, you can create a rake task similar to this.
namespace :images do
desc "Upload images."
task :create => :environment do
#images = Dir["#{RAILS_ROOT}/public/images/*.*"]
for image in #images
MyModel.create(:image => File.open(image))
end
end
end
Yes. I did something similar on my first personal Rails project. Here's a previous SO question (Paperclip S3 download remote images) whose answer links to the where I found my answer so long ago (http://trevorturk.com/2008/12/11/easy-upload-via-url-with-paperclip/).
Great answer Johnny Grass and great question Chris. I had a few hundred tif files on my local machine, Heroku, paperclip, and s3. Some of the tiff files were > 100MB, so getting heroku to pay attention for that long required delayed job and some extra work. Since this was a mostly one time batch process (5 different image forms created from each with 5 x uploads), the idea of a rake task fit perfectly. Here, in case it helps, is the rake task I created assuming like Johnny wrote that your development database has current data (use pg backup to get fresh set of ids) and is connected to S3.
I have a model called "Item" with an attachment "image". I wanted to check if existing Items already had an image, and if not, upload a new one. The effect is to mirror a directory of source files. Good extensions might be to check the dates and see if the local tif if updated.
# lib/image_management.rake
namespace :images do
desc 'upload images through paperclip with postprocessing'
task :create => :environment do
directory = "/Volumes/data/historicus/_projects/deeplandscapes/library/tifs/*.tif"
images = Dir[directory]
puts "\n\nProcessing #{ images.length } images in #{directory}..."
items_with_errors = []
items_updated = []
items_skipped = []
images.each do |image|
# find the needed record
image_basename = File.basename(image)
id = image_basename.gsub("it_", "").gsub(".tif", "").to_i
if id > 0
item = Item.find(id) rescue nil
# check if it has an image already
if item
unless item.image.exists?
# create the image
success = item.update_attributes(:image => File.open(image))
if success
items_updated << item
print ' u '
else
items_with_errors << item
print ' e '
end
else
items_skipped << item
print ' s '
end
else
print "[#{id}] "
end
else
print " [no id for #{image_basename}] "
end
end
unless items_with_errors.empty?
puts "\n\nThe following items had errors: "
items_with_errors.each do |error_image|
puts "#{error_image.id}: #{error_image.errors.full_messages}"
end
end
puts "\n\nUpdated #{items_updated.length} items."
puts "Skipped #{items_skipped.length} items."
puts "Update complete.\n"
end
end