Give a custom file name to each file being zipped - ruby-on-rails

I am downloading multiple files from S3 and zipping them. How do I give a custom name to each file being zipped?
def download_all_files
folder_path = "#{Rails.root}/public/downloads/"
zipfile_name = "#{Rails.root}/public/archive.zip"
FileUtils.remove_dir(folder_path) if Dir.exist?(folder_path)
FileUtils.remove_entry(zipfile_name) if File.exist?(zipfile_name)
Dir.mkdir("#{Rails.root}/public/downloads")
#model_object.each do |attachment|
open(folder_path + "#{attachment.avatar.file.filename}", 'wb') do |file|
file << open("#{attachment.avatar.url}").read
end
end
input_filenames = Dir.entries(folder_path).select {|f| !File.directory? f}
Zip::File.open(zipfile_name, Zip::File::CREATE) do |zipfile|
input_filenames.each do |attachment|
zipfile.add(attachment,File.join(folder_path,attachment))
end
end
send_file(File.join("#{Rails.root}/public/", 'archive.zip'), :type => 'application/zip', :filename => "#{Time.now.to_date}.zip")
end

Looks like you have 2 ways you can accomplish this. To the end user, both of these should yield the same result, just depends how you want them on your server.
1. Change it while zipping the files together
This will leave the files as they are currently named on your system and only change them in the output archive.zip
Looking at this gem it looks like
Zip::File.open(zipfile_name, Zip::File::CREATE) do |zipfile|
input_filenames.each do |filename|
# Two arguments:
# - The name of the file as it will appear in the archive
# - The original file, including the path to find it
zipfile.add(filename, folder + '/' + filename)
end
zipfile.get_output_stream("myFile") { |os| os.write "myFile contains just this" }
end
So, in your code just change the first occurrence of 'attachment' on
zipfile.add(attachment, File.join(folder_path, attachment))
to whatever you want the name to be.
2. Change it while writing the file to your server
Since you are writing the file just a few lines above there
open(folder_path + "#{attachment.avatar.file.filename}", 'wb') do |file|
you could change the file name on this line as well. This would change the filename on your server and in the archive.zip

Related

How to write csv files to S3 from inside a Job?

I have a data backup system for customers of my app. I gather up all associated csv files and zip them. Once that zip file is complete, I attach it in an email. This process breaks on heroku due to their file system. I thought since heroku-16 we could write to the app/tmp directory and that this process might occur within the same transaction and the files would be fine, but that doesn't seem to be the case. I don't even seem to be writing the files to the tmp directory in production (in Dev I am).
So, what I would like to do instead is just write the csv files directly to S3, then Zip those files and also save the .zip to S3...then, pull that file as an email attachment. To do this, I need to generate the csv files and write them to S3 from inside ActiveJob. I use S3 already as part of ActiveStorage, but this process will not utilize ActiveStorage.
Is there's a command for me to manually direct upload to an S3 bucket. I've been digging around in the docs, etc but don't see what I'm after.
The Job (using /tmp)
def perform(company_id, recipient_id)
company = Company.find(company_id)
source_folder = "#{ Rails.root }/tmp"
zipfile_name = "company_#{ company.id }_archive.zip"
zipfile_path = "#{ Rails.root }/tmp/#{ zipfile_name }"
input_filenames = []
# USERS: create a new empty csv file,
# ... then add rows to it
# ... and, add the file name to the list of files array
users_file_name = "#{ company.name.parameterize.underscore }_users_list.csv"
input_filenames << users_file_name
users_csv_file = File.new("#{ Rails.root.join('tmp') }/#{ users_file_name }", 'w')
users_csv_file << company.users.to_csv
users_csv_file.close
...
# gather up the created files and zip them
Zip::File.open(zipfile_path, create: true) do |zipfile|
input_filenames.uniq.each do |filename|
zipfile.add(filename, File.join(source_folder, filename))
end
end
puts "attaching data_export".colorize(:red)
company.data_exports.attach(
io: StringIO.new("#{ Rails.root }/tmp/company_14_#{ Time.current.to_date.to_s }_archive.zip"),
filename: 'company_14_archive.zip',
content_type: 'application/zip'
)
last_id = company.data_exports.last.id
puts "sending mail using company.id: #{ company.id }, recipient_id: #{ recipient_id }, company.data_exports.last.id: #{ last_id }".colorize(:red)
CompanyMailer.mail_data_export(
company.id,
recipient_id,
last_id
)
end
You can upload file like this on S3
key = "file_name.zip"
file_path = "tmp/file_name.zip"
new_s3_client = Aws::S3::Resource.new(region: 'eu-west-1', access_key_id: '123', secret_access_key: '456')
new_bucket = new_s3_client.bucket('public')
obj = new_bucket.object(key)
obj.upload_file(file_path)

RubyZip : Unable to find path of file stored in Active Storage

I am using Rails 5.2 and ActiveStorage to let my users upload files in my app. I have all the uploaded files displayed in a table and I can choose to select which ones I want to download. After selecting, I will be able to download all of them in a zip file. To zip the files, I am using Rubyzip and I can't get to work properly.
I have tried two ways :
1 - I tried this way and faced this error No such file or directory # rb_sysopen - /rails/active_storage/blobs/..../2234.pyThis is my controller :
def batch_download
if params["record"].present?
ids = params["record"].to_unsafe_h.map(&:first)
if ids.present?
folder_path = "#{Rails.root}/public/downloads/"
zipfile_name = "#{Rails.root}/public/archive.zip"
FileUtils.remove_dir(folder_path) if Dir.exist?(folder_path)
FileUtils.remove_entry(zipfile_name) if File.exist?(zipfile_name)
Dir.mkdir("#{Rails.root}/public/downloads")
Record.where(id: ids).each do |attachment|
open(folder_path + "#{attachment.file.filename}", 'wb') do |file|
file << open("#{rails_blob_path(attachment.file)}").read
end
end
input_filenames = Dir.entries(folder_path).select {|f| !File.directory? f}
Zip::File.open(zipfile_name, Zip::File::CREATE) do |zipfile|
input_filenames.each do |attachment|
zipfile.add(attachment,File.join(folder_path,attachment))
end
end
send_file(File.join("#{Rails.root}/public/", 'archive.zip'), :type => 'application/zip', :filename => "#{Time.now.to_date}.zip")
end
else
redirect_back fallback_location: root_path
end
end
2 - Secondly I tried to follow the rubyzip documentation and the error was a bit different.
No such file or directory # rb_file_s_lstat - /rails/active_storage/blobs/..../2234.py
if ids.present?
folder = []
input_filenames = []
Record.where(id: ids).each do |attachment|
input_filenames.push("#{attachment.file.filename}")
pre_path = "/rails/active_storage/blobs/"
path_find = "#{rails_blob_path(attachment.file)}"
folder.push(pre_path + path_find.split('/')[4])
end
container = Hash[folder.zip(input_filenames)]
zipfile_name = "/Users/fahimabdullah/Documents/archive.zip"
Zip::File.open(zipfile_name, Zip::File::CREATE) do |zipfile|
# input_filenames.each do |filename|
container.map do |path, filename|
zipfile.add(filename, File.join(path, filename))
end
zipfile.get_output_stream("myFile") { |f| f.write "myFile contains just this" }
end
I expect it to download a zip file containing all the files inside. And this is my first question so please excuse me if the question was too long. Thank you.
I just had a similar problem but saw this hasn't been answered yet. Even though it's a little older and you probably already solved this, here is my approach. The trick here was to create a tempfile in between
def whatever
zip_file = Tempfile.new('invoices.zip')
Zip::File.open(zip_file.path, Zip::File::CREATE) do |zipfile|
invoices.each do |invoice|
next unless invoice.attachment.attached?
overlay = Tempfile.new(['overlay', '.pdf'])
overlay.binmode
overlay.write(invoice.attachment.download)
overlay.close
overlay.path
zipfile.add(invoice.filename, File.join(overlay.path))
end
end
invoices_zip = File.read(zip_file.path)
UserMailer.with(user: user).invoice_export(invoices_zip, 'invoices.zip').deliver_now
ensure
zip_file.close
zip_file.unlink
end

"cannot open entry for reading while its open for writing" while creating a zip file

I'm trying to create a function that would download a zip that contains files. I followed this tutorial, that says that my code should look like the following if you want to download a CSV (I translated the comments) :
def exporter_zip
# CSV's header
csv = "Title;Content;Publishing date;File name;\n"
# Loop over articles saved in database
Article.all.each do |article|
# Creating the CSV with datas
csv += "#{article.title};#{article.content};#{article.publishing_date.strftime("%Y-%m-%d") if article.publishing_date };#{article.file_name}\n"
end
# Creating the zip file inside the folder
zip_tmp = File.new("#{Rails.root}/db/mon_fichier.zip", "w+")
# opening the file in writing mode
Zip::File.open(zip_tmp.path, Zip::File::CREATE) {
|zipfile|
# Inserting the csv variable's content inside the article.csv file, which is inserted into the zip file
zipfile.get_output_stream("articles.csv") { |f| f.puts csv }
}
# Sending the created file
send_file "#{Rails.root}/db/mon_fichier.zip"
end
Here is how i adapted the code:
class DownloadController < ApplicationController
require 'zip'
def zip
# Creating zip file
zip_tmp = File.new("#{Rails.root}/public/zip-#{Time.now.strftime('%d-%m-%Y')}.zip", 'w+')
FileDetail.all.each do |fichier|
Zip::File.open(zip_tmp.path, Zip::File::CREATE) { |zipfile|
# fichier.filename => file.mp3
# fichier.path => path/to/file.mp3
zipfile.get_output_stream(fichier.filename, fichier.path)
}
end
send_file "#{Rails.root}/public/zip-#{Date.today.to_time}.zip"
end
end
However, while I'm not even sure I'm doing this correctly, I get the following error: cannot open entry for reading while its open for writing, targetting the following line: Zip::File.open(zip_tmp.path, Zip::File::CREATE) { |zipfile|
Can anyone can tell me what's going on? I never did this before so I don't know what went wrong..
Thank you in advance
FileDetail.all.each do |fichier|
Zip::File.open(zip_tmp.path, Zip::File::CREATE) { |zipfile|
# fichier.filename => file.mp3
# fichier.path => path/to/file.mp3
zipfile.get_output_stream(fichier.filename, fichier.path)
}
end
Doing it this way will attempt to create the zip file for each member of your FileDetail. You should code it to only open and create once:
Zip::File.open(zip_tmp.path, Zip::File::CREATE) do |zipfile|
FileDetail.all.each do |fichier|
# fichier.filename => file.mp3
# fichier.path => path/to/file.mp3
zipfile.get_output_stream(fichier.filename, fichier.path)
end
end

Issue with Zip file exports in Ruby / Rails

I am having an issue with my rails action that takes binary (blob) files from my database and packages them up into a nice zip file and then finally sends it out for download. The problem occurs when I try to unzip the file, it says something like "Unable to expand; Error 1 - Operation not permitted." I believe that means that the file is corrupted but I don't know what I am doing wrong. I've included my code below, any help would be greatly appreciated. Thanks!
require 'zip/zip'
require 'zip/zipfilesystem'
def export
#layers = Layer.where('group_id > 1')
temp = Tempfile.new("layers-zip-export")
Zip::ZipOutputStream.open(temp.path) do |zipfile|
#layers.each do |layer|
zipfile.put_next_entry(layer.name)
file = Tempfile.new("temp-" + layer.id.to_s)
file.binmode
file << layer.file
file.rewind
zipfile.write IO.binread(file.path)
file.close
file.unlink
end
end
send_file temp.path, :type => 'application/zip', :filename => "layer-export.zip"
temp.close
end

Downloading and zipping files that were uploaded to S3 with CarrierWave

I have a small Rails 3.2.1 app that uses CarrierWave 0.5.8 for file uploads to S3 (using Fog)
I want users to be able to select some images that they'd like to download, then zip them up and send them a zip. Here is what I've come up with:
def generate_zip
#A collection of Photo objects. The Photo object has a PhotoUploader mounted.
photos = Photo.all
tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
zip.close
photos.each do |photo|
file_to_add = photo.photo.file
zip = Zip::ZipFile.open(tmp_filename)
zip.add("tmp/", file_to_add.path)
zip.close
end
#do the rest.. like send zip or upload file and e-mail link
end
This doesn't work because photo.photo.file returns an instance of CarrierWave::Storage::Fog::File instead of a regular file.
EDIT: The error this leads to:
Errno::ENOENT: No such file or directory - uploads/photos/name.jpg
I also tried the following:
tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
zip.close
photos.each do |photo|
processed_uri = URI.parse(URI.escape(URI.unescape(photo.photo.file.authenticated_url)).gsub("[", "%5B").gsub("]", "%5D"))
file_to_add = CarrierWave::Uploader::Download::RemoteFile.new(processed_uri)
zip = Zip::ZipFile.open(tmp_filename)
zip.add("tmp/", file_to_add.path)
zip.close
end
But this gives me a 403. Some help would be greatly appreciated.. It probably is not that hard I'm just Doing it Wrong™
I've managed to solve the problem with help from #ffoeg
The solution offered by #ffoeg didn't work quite so well for me since I was dealing with zip files > 500 MB which caused me problems on Heroku. I've therefor moved the zipping to a background process using resque:
app/workers/photo_zipper.rb:
require 'zip/zip'
require 'zip/zipfilesystem'
require 'open-uri'
class PhotoZipper
#queue = :photozip_queue
#I pass
def self.perform(id_of_object_with_images, id_of_user_to_be_notified)
user_mail = User.where(:id => id_of_user_to_be_notified).pluck(:email)
export = PhotoZipper.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)
Notifications.zip_ready(export.archive_url, user_mail).deliver
end
# Zipfile generator
def self.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)
object = ObjectWithImages.find(id_of_object_with_images)
photos = object.images
# base temp dir
temp_dir = Dir.mktmpdir
# path for zip we are about to create, I find that ruby zip needs to write to a real file
# This assumes the ObjectWithImages object has an attribute title which is a string.
zip_path = File.join(temp_dir, "#{object.title}_#{Date.today.to_s}.zip")
Zip::ZipOutputStream.open(zip_path) do |zos|
photos.each do |photo|
path = photo.photo.path
zos.put_next_entry(path)
zos.write photo.photo.file.read
end
end
#Find the user that made the request
user = User.find(id_of_user_to_be_notified)
#Create an export object associated to the user
export = user.exports.build
#Associate the created zip to the export
export.archive = File.open(zip_path)
#Upload the archive
export.save!
#return the export object
export
ensure
# clean up the tempdir now!
FileUtils.rm_rf temp_dir if temp_dir
end
end
app/controllers/photos_controller.rb:
format.zip do
#pick the last ObjectWithImages.. ofcourse you should include your own logic here
id_of_object_with_images = ObjectWithImages.last.id
#enqueue the Photozipper task
Resque.enqueue(PhotoZipper, id_of_object_with_images, current_user.id)
#don't keep the user waiting and flash a message with information about what's happening behind the scenes
redirect_to some_path, :notice => "Your zip is being created, you will receive an e-mail once this process is complete"
end
Many thanks to #ffoeg for helping me out. If your zips are smaller you could try #ffoeg's solution.
Here is my take. There could be typos but I think this is the gist of it :)
# action method, stream the zip
def download_photos_as_zip # silly name but you get the idea
generate_zip do |zipname, zip_path|
File.open(zip_path, 'rb') do |zf|
# you may need to set these to get the file to stream (if you care about that)
# self.last_modified
# self.etag
# self.response.headers['Content-Length']
self.response.headers['Content-Type'] = "application/zip"
self.response.headers['Content-Disposition'] = "attachment; filename=#{zipname}"
self.response.body = Enumerator.new do |out| # Enumerator is ruby 1.9
while !zf.eof? do
out << zf.read(4096)
end
end
end
end
end
# Zipfile generator
def generate_zip(&block)
photos = Photo.all
# base temp dir
temp_dir = Dir.mktempdir
# path for zip we are about to create, I find that ruby zip needs to write to a real file
zip_path = File.join(temp_dir, 'export.zip')
Zip::ZipFile::open(zip_path, true) do |zipfile|
photos.each do |photo|
zipfile.get_output_stream(photo.photo.identifier) do |io|
io.write photo.photo.file.read
end
end
end
# yield the zipfile to the action
block.call 'export.zip', zip_path
ensure
# clean up the tempdir now!
FileUtils.rm_rf temp_dir if temp_dir
end

Resources