I created an ActiveJob to process my carrier waves uploads. However, when I upload more than one image, I get the following error for the second file:
Errno::ENOENT (No such file or directory # rb_sysopen - C:/Users/tdavi/AppData/Local/Temp/RackMultipart20180830-392-z2s2i.jpg)
Here's the code in my controller:
if #post.save
files = params[:post_attachments].map { |p|
{image: p['photo'][:image].tempfile.path, description: p['photo'][:decription]}
}
ProcessPhotosJob.perform_later(#post.id, files.to_json)
format.html { render :waiting }
end
And my ActiveJob
require 'json'
class ProcessPhotosJob < ApplicationJob
queue_as :default
def perform(post_id, photos_json)
post = Post.friendly.find(post_id)
photos = JSON.parse photos_json
photos.each do |p|
src_file = File.new(p['image'])
post.post_attachments.create!(:photo => src_file, :description => p[:description])
end
post.processed = true
post.save
end
end
When I upload only one file to upload, it works okay.
You should not pass Tempfile to the queued jobs.
First of all - TempFiles can be deleted automatically by Ruby (docs, explanation)
If you would like to upload file(s) and process them later (in a background), then I would suggest you check this question.
Related
For a variety of reasons I am migrating my uploads from ActiveStorage (AS) to CarrierWave (CW).
I am making rake task and have the logic sorted out - I am stumped at how to feed the AS blob into the CW file.
I am trying something like ths:
#files.each.with_index(1) do | a, index |
if a.attachment.attached?
a.attachment.download do |file|
a.file = file
end
a.save!
end
end
This is based on these two links:
https://edgeguides.rubyonrails.org/active_storage_overview.html#downloading-files
message.video.open do |file|
system '/path/to/virus/scanner', file.path
# ...
end
and
https://github.com/carrierwaveuploader/carrierwave#activerecord
# like this
File.open('somewhere') do |f|
u.avatar = f
end
I tested this locally and the files are not mounted via the uploader. My question(s) would be:
am I missing something obvious here?
is my approach wrong and needs a new one?
Bonus Karma Question:
I can't seem to see a clear path to set the CW filename when I do this?
Here is my final rack task (based on the accepted answer) - open to tweaks. Does the job for me:
namespace :carrierwave do
desc "Import the old AS files into CW"
task import: :environment do
#files = Attachment.all
puts "#{#files.count} files to be processed"
puts "+" * 50
#files.each.with_index(1) do | a, index |
if a.attachment.attached?
puts "Attachment #{index}: Key: #{a.attachment.blob.key} ID: #{a.id} Filename: #{a.attachment.blob.filename}"
class FileIO < StringIO
def initialize(stream, filename)
super(stream)
#original_filename = filename
end
attr_reader :original_filename
end
a.attachment.download do |file|
a.file = FileIO.new(file, a.attachment.blob.filename.to_s)
end
a.save!
puts "-" * 50
end
end
end
desc "Purge the old AS files"
task purge: :environment do
#files = Attachment.all
puts "#{#files.count} files to be processed"
puts "+" * 50
#files.each.with_index(1) do | a, index |
if a.attachment.attached?
puts "Attachment #{index}: Key: #{a.attachment.blob.key} ID: #{a.id} Filename: #{a.attachment.blob.filename}"
a.attachment.purge
puts "-" * 50
#count = index
end
end
puts "#{#count} files purged"
end
end
Now in my case I am doing this in steps - I have branched my master with this rake task and the associated MCV updates. If my site was in true production would probably run the import rake task first then confirm all went well THEN purge the old AS files.
The file object you get from the attachment.download block is a string. More precisely, the response from .download is the file, "streamed and yielded in chunks" (see documentation). I validated this by calling file.class to make sure the class is what I expected.
So, to solve your issue, you need to provide an object on which .read can be called. Commonly that is done using the Ruby StringIO class.
However, considering Carrierwave also expects a filename, you can solve it using a helper model that inherits StringIO (from blogpost linked above):
class FileIO < StringIO
def initialize(stream, filename)
super(stream)
#original_filename = filename
end
attr_reader :original_filename
end
And then you can replace a.file = file with a.file = FileIO.new(file, 'new_filename')
I have a very strange bug i been fighting with for a day now.
I'm trying to make an automatic uploader for uploading hundreds of PDF-files and putting them into categories from my local computer to my remote Amazon s3 server using Carrierwave.
When i run it with :file in my pdf_uploader everything is working as it should be, but when i run it with fog trying to save them on the Amazon s3 server everything is duplicated. And not like 1,1,2,2,3,3 but 1,2,3,1,2,3
(i already have two image_uploaders and i tried uploading from a local directory earlier on with images, so the server and connection should be set up right)
Here are my files:
#routes.rb
get "upload_pdfs/now" => "pdf_categories#upload_pdfs", as: "upload_pdfs"
#pdf_categories_controller.rb
def upload_pdfs
path = "/Users/bruger/Desktop/DENHER"
Dir.foreach(path) do |category_path|
next if folder_or_file_excluded? category_path
pdf_category = PdfCategory.create!(title: category_path)
Dir.foreach("#{path}/#{category_path}") do |file_path|
next if folder_or_file_excluded? file_path
pdf_file_without_file_ending = remove_last_obj_in_arr file_path.split(".")
new_pdf = Pdf.new(
title: pdf_file_without_ending,
pdf_category_id: pdf_category.id,
)
#### SOMETHING IS HAPPENING HERE! ###
File.open(path + "/" + category_path + "/" + file_path) do |file_from_path|
new_pdf.file = file_from_path
new_pdf.save!
end
#### SOMETHING IS HAPPENING HERE! ###
end # file_path
end # category_path
redirect_to root_path
end
def is_integer? thing
thing.to_i.to_s == thing
end
def remove_last_obj_in_arr array
new_array = array.first array.size - 1
new_array.join(".")
end
def folder_or_file_excluded? folder_file
list = %w(Fravalgt Fravalg Valgt fra fravalg fravalgt JPEG . .git DS_Store .. .DS_Store image_upload.rb)
list.include? folder_file
end
#pdf_uploader.rb
class PdfUploader < CarrierWave::Uploader::Base
storage :fog
def store_dir
"pdf/#{mounted_as}/#{model.id}"
end
def extension_whitelist
%w(pdf PDF)
end
end
Have you experienced anything like this or do you have a clue?
Please. Any help will be appreciated!
I am having a scenario where the photos I am uploading has to store in a AWS S3 buckets and call the images in through email but after the Rails and corresponding gems upgradation I could not store the images in S3. I upgraded my aws-s3 version from 0.6.1 to 0.6.3, aws-sdk from 1.3.5 to 1.3.9, right_aws from 3.0.0 to 3.0.5 and finally Rails version from 3.2.1 to 4.2.6.
I have tested by putting puts commands, it is going to all the methods but I doubt whether there is any syntax change in upload method at #type (Here #type is the 2 bucket names photo_screenshots and indicator_screenshots).
Please help me.
This is my lib/screenshot.rb:
class Screenshot
attr_reader :user_id, :report_id, :type
def initialize(user_id, report_id, type)
#user_id, #report_id, #type = user_id, report_id, type
capture
resize(500, 686) if #type == 'report_screenshots'
upload
delete_local_copy
end
def capture
if Rails.env.production?
phantom = Rails.root.join('vendor/javascripts/phantomjs_linux/bin/phantomjs')
url = Rails.application.config.custom.domain_url + "users/#{#user_id}/reports/#{#report_id}"
end
js = Rails.root.join("vendor/javascripts/#{#type}.js")
image = Rails.root.join("public/#{#type}/#{#report_id}.png")
`/bin/bash -c "DISPLAY=:0 #{phantom} #{js} #{url} #{image}"`
end
def resize(width, height)
path = "public/#{#type}/#{#report_id}.png"
img = Magick::Image::read(path).first
#img.thumbnail!(width, height)
img.change_geometry("#{width}x#{height}") do |cols, rows, img|
img.thumbnail!(cols, rows)
end
img.write(path)
end
def upload
file_name = Rails.root.join("public/#{#type}/#{#report_id}.png")
s3config = YAML.load_file(Rails.root.join('config', 's3.yml'))[Rails.env]
s3 = RightAws::S3.new(s3config["access_key_id"], s3config["secret_access_key"])
#type == 'report_screenshots' ? s3.bucket("my_project.#{Rails.env}", true).put("#{#type}/#{#report_id}.png", File.open(file_name), {}, 'public-read', { 'content-type' => 'image/png' }) : s3.bucket("my_project.#{Rails.env}", true).put("indicator_screenshots/#{#report_id}.png", File.open(file_name), {}, 'public-read', { 'content-type' => 'image/png' })
report = Report.find(#report_id)
#type == 'report_screenshots' ? report.update_attribute(:report_screenshot_at, Time.now) : report.update_attribute(:indicator_screenshot_at, Time.now)
end
def delete_local_copy
file_name = Rails.root.join("public/#{#type}/#{#report_id}.png")
File.delete(file_name)
end
def self.delete_s3_copy(report_id, type)
s3config = YAML.load_file(Rails.root.join('config', 's3.yml'))[Rails.env]
s3 = RightAws::S3.new(s3config["access_key_id"], s3config["secret_access_key"])
s3.bucket("my_project.#{Rails.env}").key("#{type}/#{report_id}.png").delete
end
end
Whenever I click on send an email, this is what happens:
controller:
def send_test_email
if #report.photos.empty?
Rails.env.development? ? Screenshot.new(#user.id, #report.id, Rails.application.config.custom.indicator_screenshot_bucket) : Screenshot.delay.new(#user.id, #report.id, Rails.application.config.custom.indicator_screenshot_bucket)
else
Rails.env.development? ? Screenshot.new(#user.id, #report.id, "photo_screenshots") : Screenshot.delay.new(#user.id, #report.id, "photo_screenshots")
end
ReportMailer.delay.test_report_email(#user, #report)
respond_to do |format|
format.json { render :json => { :success => true, :report_id => #report.id, :notice => 'Test email was successfully sent!' } }
end
end
This is RAILS_ENV=production log:
New RightAws::S3Interface using shared connections mode Opening new
HTTPS connection to my_project.production.s3.amazonaws.com:443 Opening
new HTTPS connection to s3.amazonaws.com:443 2016-09-26T10:48:46+0000:
[Worker(delayed_job host:ip-172-31-24-139 pid:8769)] Job
Screenshot.new (id=528) FAILED (16 prior attempts) with Errno::ENOENT:
No such file or directory # rb_sysopen -
/var/www/html/project/my_project/public/photo_screenshots/50031.png
2016-09-26T10:48:46+0000: [Worker(delayed_job host:ip-172-31-24-139
pid:8769)] Job Screenshot.new (id=529) RUNNING
2016-09-26T10:48:46+0000: [Worker(delayed_job host:ip-172-31-24-139
pid:8769)] Job Screenshot.new (id=529) FAILED (16 prior attempts) with
Magick::ImageMagickError: unable to open file
`public/report_screenshots/50031.png' # error/png.c/ReadPNGImage/3733
2016-09-26T10:48:46+0000: [Worker(delayed_job host:ip-172-31-24-139
pid:8769)] 2 jobs processed at 1.6978 j/s, 2 failed
This is AWS production log:
New RightAws::S3Interface using shared connections mode
2016-09-26T16:00:30+0530: [Worker(host:OSI-L-0397 pid:7117)] Job
Screenshot.new (id=50) FAILED (6 prior attempts) with Errno::ENOENT:
No such file or directory # rb_sysopen -
/home/abcuser/Desktop/project/my_project/public/photo_screenshots/10016.png
2016-09-26T16:00:30+0530: [Worker(host:OSI-L-0397 pid:7117)] Job
Screenshot.new (id=51) RUNNING 2016-09-26T16:00:30+0530:
[Worker(host:OSI-L-0397 pid:7117)] Job Screenshot.new (id=51) FAILED
(6 prior attempts) with Magick::ImageMagickError: unable to open file
`public/report_screenshots/10016.png' # error/png.c/ReadPNGImage/3667
2016-09-26T16:00:30+0530: [Worker(host:OSI-L-0397 pid:7117)] 2 jobs
processed at 0.2725 j/s, 2 failed
You can try uploading in a simpler approach.
Uploading images to a fixed bucket with different folders for each object or application.The s3 keeps a limitation on the number of buckets creattion whereas there is no
limitation for content inside a bucket.
This code will upload image for a user to s3 using aws-sdk gem. The bucket and the image uploaded are made public
so that the images uploaded are directly accessible. The input it takes is the image complete path
where it is present, folder in which it should be uploaded and user_id for whom it should
be uploaded.
def save_screenshot_to_s3(image_location, folder_name,user_id)
service = AWS::S3.new(:access_key_id => ACCESS_KEY_ID,
:secret_access_key => SECRET_ACCESS_KEY)
bucket_name = "app-images"
if(service.buckets.include?(bucket_name))
bucket = service.buckets[bucket_name]
else
bucket = service.buckets.create(bucket_name)
end
bucket.acl = :public_read
key = folder_name.to_s + "/" + File.basename(image_location)
s3_file = service.buckets[bucket_name].objects[key].write(:file => image_location)
s3_file.acl = :public_read
user = User.where(id: user_id).first
user.image = s3_file.public_url.to_s
user.save
end
for handling the screenshot part, in your capture method you have use done something like this.
`/bin/bash -c "DISPLAY=:0 #{phantom} #{js} #{url} #{image}"`
Is the /bin/bash thing really required, change it to below code and it should work.
`DISPLAY=:0 "#{phantom}" "#{js}" "#{url}" "#{image}"`
let it be if it breaks something else.
Since you are aware of the final image location which is image. Pass this directly to save_screenshot_to_s3 and you should be able to save it. This will save the image path to user too if you pass your user_id as specified in method
I have a small Rails 3.2.1 app that uses CarrierWave 0.5.8 for file uploads to S3 (using Fog)
I want users to be able to select some images that they'd like to download, then zip them up and send them a zip. Here is what I've come up with:
def generate_zip
#A collection of Photo objects. The Photo object has a PhotoUploader mounted.
photos = Photo.all
tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
zip.close
photos.each do |photo|
file_to_add = photo.photo.file
zip = Zip::ZipFile.open(tmp_filename)
zip.add("tmp/", file_to_add.path)
zip.close
end
#do the rest.. like send zip or upload file and e-mail link
end
This doesn't work because photo.photo.file returns an instance of CarrierWave::Storage::Fog::File instead of a regular file.
EDIT: The error this leads to:
Errno::ENOENT: No such file or directory - uploads/photos/name.jpg
I also tried the following:
tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
zip.close
photos.each do |photo|
processed_uri = URI.parse(URI.escape(URI.unescape(photo.photo.file.authenticated_url)).gsub("[", "%5B").gsub("]", "%5D"))
file_to_add = CarrierWave::Uploader::Download::RemoteFile.new(processed_uri)
zip = Zip::ZipFile.open(tmp_filename)
zip.add("tmp/", file_to_add.path)
zip.close
end
But this gives me a 403. Some help would be greatly appreciated.. It probably is not that hard I'm just Doing it Wrong™
I've managed to solve the problem with help from #ffoeg
The solution offered by #ffoeg didn't work quite so well for me since I was dealing with zip files > 500 MB which caused me problems on Heroku. I've therefor moved the zipping to a background process using resque:
app/workers/photo_zipper.rb:
require 'zip/zip'
require 'zip/zipfilesystem'
require 'open-uri'
class PhotoZipper
#queue = :photozip_queue
#I pass
def self.perform(id_of_object_with_images, id_of_user_to_be_notified)
user_mail = User.where(:id => id_of_user_to_be_notified).pluck(:email)
export = PhotoZipper.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)
Notifications.zip_ready(export.archive_url, user_mail).deliver
end
# Zipfile generator
def self.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)
object = ObjectWithImages.find(id_of_object_with_images)
photos = object.images
# base temp dir
temp_dir = Dir.mktmpdir
# path for zip we are about to create, I find that ruby zip needs to write to a real file
# This assumes the ObjectWithImages object has an attribute title which is a string.
zip_path = File.join(temp_dir, "#{object.title}_#{Date.today.to_s}.zip")
Zip::ZipOutputStream.open(zip_path) do |zos|
photos.each do |photo|
path = photo.photo.path
zos.put_next_entry(path)
zos.write photo.photo.file.read
end
end
#Find the user that made the request
user = User.find(id_of_user_to_be_notified)
#Create an export object associated to the user
export = user.exports.build
#Associate the created zip to the export
export.archive = File.open(zip_path)
#Upload the archive
export.save!
#return the export object
export
ensure
# clean up the tempdir now!
FileUtils.rm_rf temp_dir if temp_dir
end
end
app/controllers/photos_controller.rb:
format.zip do
#pick the last ObjectWithImages.. ofcourse you should include your own logic here
id_of_object_with_images = ObjectWithImages.last.id
#enqueue the Photozipper task
Resque.enqueue(PhotoZipper, id_of_object_with_images, current_user.id)
#don't keep the user waiting and flash a message with information about what's happening behind the scenes
redirect_to some_path, :notice => "Your zip is being created, you will receive an e-mail once this process is complete"
end
Many thanks to #ffoeg for helping me out. If your zips are smaller you could try #ffoeg's solution.
Here is my take. There could be typos but I think this is the gist of it :)
# action method, stream the zip
def download_photos_as_zip # silly name but you get the idea
generate_zip do |zipname, zip_path|
File.open(zip_path, 'rb') do |zf|
# you may need to set these to get the file to stream (if you care about that)
# self.last_modified
# self.etag
# self.response.headers['Content-Length']
self.response.headers['Content-Type'] = "application/zip"
self.response.headers['Content-Disposition'] = "attachment; filename=#{zipname}"
self.response.body = Enumerator.new do |out| # Enumerator is ruby 1.9
while !zf.eof? do
out << zf.read(4096)
end
end
end
end
end
# Zipfile generator
def generate_zip(&block)
photos = Photo.all
# base temp dir
temp_dir = Dir.mktempdir
# path for zip we are about to create, I find that ruby zip needs to write to a real file
zip_path = File.join(temp_dir, 'export.zip')
Zip::ZipFile::open(zip_path, true) do |zipfile|
photos.each do |photo|
zipfile.get_output_stream(photo.photo.identifier) do |io|
io.write photo.photo.file.read
end
end
end
# yield the zipfile to the action
block.call 'export.zip', zip_path
ensure
# clean up the tempdir now!
FileUtils.rm_rf temp_dir if temp_dir
end
I have a CSV file to parse with two ways:
Upload the file and parse it in real time
Upload the file and parse it with a delayed_job
When I parse the file in real time, there is no problem, but when I parse the file with a delayed job, I get a CSV::Table but with nil fields [nil,nil.....]
This is how I parse the file in the helper:
CSV.parse(file.read, :headers => true, :row_sep =>:auto)
This is how I save the csv file and I create the delayed job:
yeah its CSV, I made a typo, but I still have the problem, I read the file in an action like this:
def create
if params[:dump]
file = File.new(params[:dump][:file].original_filename, "w")
File.open(file.path, "w") { |f| f.write(params[:dump][:file].read.force_encoding('UTF-8')) }
#import = Import.create(:created_by => current_user.id, :import_file_path => file.path, :size => 0)
file.close
file_to_parse = File.open(#import.import_file_path)
if params[:dump][:file].size < 10000
parsed_file = parsed file_to_parse
if parsed_file && parsed_file.count > 0
#imported_contacts, #imported_organizations = extract_contacts_and_organizations_from parsed_file
validate_and_save #imported_organizations if #imported_organizations.any?
validate_and_save #imported_contacts if #imported_contacts.any?
#import.update_attributes(:done => true, :size => (#imported_contacts.size + #imported_organizations.size))
redirect_to org_import_path(current_org, #import)
else
redirect_to new_org_import_path(current_org) , :notice => "fichier vide ou introuvable ou ne pas de format cvs/vcf"
end
else
Delayed::Job.enqueue(ImportJob.new(current_org, current_user, #import.id))
redirect_to org_import_path(current_org, #import)
end
else
redirect_to new_org_import_path(current_org) , :notice => "vous devrez selctionner un fichier"
end
end
In your code, #import object has the :import_file_path variable that points to a tempfile (file.path) uploaded by a user.
Since this tempfile will be automatically removed as soon as the action finishes, the delayed job can't find the file when it attempts to run the job in the background (the action is already finished).
What about to copy a tempfile to some place and use it instead?
In this case, you need to add a line of code that deletes the copied file at the end of the delayed job process.
I hope this works for your problem.