I received files in mp3 (2 minutes/files) I want to concatenate together and create a bigger file. So I created my model a function to do this using ffmpeg and IO.popen
FileUtils.mkdir_p "#{Rails.root}/tmp/files"
imported_dir = "#{Rails.root}/tmp/files/#{SecureRandom.uuid}"
links.each_with_index do |link, index|
file_path = "#{imported_dir}_#{index}#{File.extname(link)}"
File.open(file_path, 'wb') do |file|
file.write open(link).read
end
concat_list << "file '#{file_path}'\n"
end
File.open("#{imported_dir}.txt", 'w'){ |f| f.write(concat_list)}
io = IO.popen("#{Rails.root}/lib/ffmpeg/ffmpeg -f concat -i #{imported_dir}.txt -c copy #{imported_dir}.mp3").readlines
if sound = Sound.create(user_id: user.id, file: File.open("#{imported_dir}.mp3"), lang: lang, title: title)
audio = FFMPEG::Movie.new("#{imported_dir}.mp3")
if !audio.valid?
puts "//_!_\\\\ Failed reading with ffmpeg (#{sound.id})#{sound.title} //_!_\\\\"
return false
end
end
the problem is that my .txt file containing the file path
file '/home/test/apps/example/releases/20150305224026/tmp/files/4dbe9707-cfef-467b-ab2c-a5e1e1165953_0.mp3'
created files as well but the final file is not created and i got the error message :
No such file or directory # rb_sysopen - /home/test/apps/example/releases/20150305224026/tmp/files/4dbe9707-cfef-467b-ab2c-a5e1e1165953.mp3
If anyone could help me
Make sure the target directory exists and is writable by the user running your app.
Related
I have a data backup system for customers of my app. I gather up all associated csv files and zip them. Once that zip file is complete, I attach it in an email. This process breaks on heroku due to their file system. I thought since heroku-16 we could write to the app/tmp directory and that this process might occur within the same transaction and the files would be fine, but that doesn't seem to be the case. I don't even seem to be writing the files to the tmp directory in production (in Dev I am).
So, what I would like to do instead is just write the csv files directly to S3, then Zip those files and also save the .zip to S3...then, pull that file as an email attachment. To do this, I need to generate the csv files and write them to S3 from inside ActiveJob. I use S3 already as part of ActiveStorage, but this process will not utilize ActiveStorage.
Is there's a command for me to manually direct upload to an S3 bucket. I've been digging around in the docs, etc but don't see what I'm after.
The Job (using /tmp)
def perform(company_id, recipient_id)
company = Company.find(company_id)
source_folder = "#{ Rails.root }/tmp"
zipfile_name = "company_#{ company.id }_archive.zip"
zipfile_path = "#{ Rails.root }/tmp/#{ zipfile_name }"
input_filenames = []
# USERS: create a new empty csv file,
# ... then add rows to it
# ... and, add the file name to the list of files array
users_file_name = "#{ company.name.parameterize.underscore }_users_list.csv"
input_filenames << users_file_name
users_csv_file = File.new("#{ Rails.root.join('tmp') }/#{ users_file_name }", 'w')
users_csv_file << company.users.to_csv
users_csv_file.close
...
# gather up the created files and zip them
Zip::File.open(zipfile_path, create: true) do |zipfile|
input_filenames.uniq.each do |filename|
zipfile.add(filename, File.join(source_folder, filename))
end
end
puts "attaching data_export".colorize(:red)
company.data_exports.attach(
io: StringIO.new("#{ Rails.root }/tmp/company_14_#{ Time.current.to_date.to_s }_archive.zip"),
filename: 'company_14_archive.zip',
content_type: 'application/zip'
)
last_id = company.data_exports.last.id
puts "sending mail using company.id: #{ company.id }, recipient_id: #{ recipient_id }, company.data_exports.last.id: #{ last_id }".colorize(:red)
CompanyMailer.mail_data_export(
company.id,
recipient_id,
last_id
)
end
You can upload file like this on S3
key = "file_name.zip"
file_path = "tmp/file_name.zip"
new_s3_client = Aws::S3::Resource.new(region: 'eu-west-1', access_key_id: '123', secret_access_key: '456')
new_bucket = new_s3_client.bucket('public')
obj = new_bucket.object(key)
obj.upload_file(file_path)
I'm trying to create a function that would download a zip that contains files. I followed this tutorial, that says that my code should look like the following if you want to download a CSV (I translated the comments) :
def exporter_zip
# CSV's header
csv = "Title;Content;Publishing date;File name;\n"
# Loop over articles saved in database
Article.all.each do |article|
# Creating the CSV with datas
csv += "#{article.title};#{article.content};#{article.publishing_date.strftime("%Y-%m-%d") if article.publishing_date };#{article.file_name}\n"
end
# Creating the zip file inside the folder
zip_tmp = File.new("#{Rails.root}/db/mon_fichier.zip", "w+")
# opening the file in writing mode
Zip::File.open(zip_tmp.path, Zip::File::CREATE) {
|zipfile|
# Inserting the csv variable's content inside the article.csv file, which is inserted into the zip file
zipfile.get_output_stream("articles.csv") { |f| f.puts csv }
}
# Sending the created file
send_file "#{Rails.root}/db/mon_fichier.zip"
end
Here is how i adapted the code:
class DownloadController < ApplicationController
require 'zip'
def zip
# Creating zip file
zip_tmp = File.new("#{Rails.root}/public/zip-#{Time.now.strftime('%d-%m-%Y')}.zip", 'w+')
FileDetail.all.each do |fichier|
Zip::File.open(zip_tmp.path, Zip::File::CREATE) { |zipfile|
# fichier.filename => file.mp3
# fichier.path => path/to/file.mp3
zipfile.get_output_stream(fichier.filename, fichier.path)
}
end
send_file "#{Rails.root}/public/zip-#{Date.today.to_time}.zip"
end
end
However, while I'm not even sure I'm doing this correctly, I get the following error: cannot open entry for reading while its open for writing, targetting the following line: Zip::File.open(zip_tmp.path, Zip::File::CREATE) { |zipfile|
Can anyone can tell me what's going on? I never did this before so I don't know what went wrong..
Thank you in advance
FileDetail.all.each do |fichier|
Zip::File.open(zip_tmp.path, Zip::File::CREATE) { |zipfile|
# fichier.filename => file.mp3
# fichier.path => path/to/file.mp3
zipfile.get_output_stream(fichier.filename, fichier.path)
}
end
Doing it this way will attempt to create the zip file for each member of your FileDetail. You should code it to only open and create once:
Zip::File.open(zip_tmp.path, Zip::File::CREATE) do |zipfile|
FileDetail.all.each do |fichier|
# fichier.filename => file.mp3
# fichier.path => path/to/file.mp3
zipfile.get_output_stream(fichier.filename, fichier.path)
end
end
I am having an issue with my rails action that takes binary (blob) files from my database and packages them up into a nice zip file and then finally sends it out for download. The problem occurs when I try to unzip the file, it says something like "Unable to expand; Error 1 - Operation not permitted." I believe that means that the file is corrupted but I don't know what I am doing wrong. I've included my code below, any help would be greatly appreciated. Thanks!
require 'zip/zip'
require 'zip/zipfilesystem'
def export
#layers = Layer.where('group_id > 1')
temp = Tempfile.new("layers-zip-export")
Zip::ZipOutputStream.open(temp.path) do |zipfile|
#layers.each do |layer|
zipfile.put_next_entry(layer.name)
file = Tempfile.new("temp-" + layer.id.to_s)
file.binmode
file << layer.file
file.rewind
zipfile.write IO.binread(file.path)
file.close
file.unlink
end
end
send_file temp.path, :type => 'application/zip', :filename => "layer-export.zip"
temp.close
end
I am uploading an audiofile and make a mp3 version, which works. Additionally I am generating a waveform out of the mp3 as a "png". Wich also works well.
The image was generated and saved but with "mp3" as suffix, which should be "png".
The view has rendered the image correctly with the "mp3" fileextension name.
Now I get a 404 error when the view tries to get the image. The filename was not assumed correctly:
https://mybucket.amazonaws.com/uploads/sound/soundfile/142/waveform_Sky_02.wav
which should be
https://mybucket.amazonaws.com/uploads/sound/soundfile/142/waveform.png
here is my :version code:
version :waveform do
def filename
"watermark.png" if original_filename.present?
end
def convert_to_waveform
cache_stored_file! if !cached?
Dir::Tmpname.create(File.basename(current_path)) do |tempname|
begin
puts system %Q{ffmpeg -y -i "#{current_path}" -f wav "#{tempname}" > /dev/null 2>&1}
FileUtils.rm current_path
Waveform.generate(tempname, current_path, method: :rms, background_color: :transparent)
ensure
FileUtils.rm tempname
end
end
end
process :convert_to_waveform
end
The database saves "waveform_Sky_02.wav"
How can I get this working?
this has been working for me:
def full_filename(for_file=file)
super.chomp('wav') + '.png'
end
or for all filetypes:
def full_filename(for_file=file)
super.chomp(File.extname(super)) + '.png'
end
I have a small Rails 3.2.1 app that uses CarrierWave 0.5.8 for file uploads to S3 (using Fog)
I want users to be able to select some images that they'd like to download, then zip them up and send them a zip. Here is what I've come up with:
def generate_zip
#A collection of Photo objects. The Photo object has a PhotoUploader mounted.
photos = Photo.all
tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
zip.close
photos.each do |photo|
file_to_add = photo.photo.file
zip = Zip::ZipFile.open(tmp_filename)
zip.add("tmp/", file_to_add.path)
zip.close
end
#do the rest.. like send zip or upload file and e-mail link
end
This doesn't work because photo.photo.file returns an instance of CarrierWave::Storage::Fog::File instead of a regular file.
EDIT: The error this leads to:
Errno::ENOENT: No such file or directory - uploads/photos/name.jpg
I also tried the following:
tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
zip.close
photos.each do |photo|
processed_uri = URI.parse(URI.escape(URI.unescape(photo.photo.file.authenticated_url)).gsub("[", "%5B").gsub("]", "%5D"))
file_to_add = CarrierWave::Uploader::Download::RemoteFile.new(processed_uri)
zip = Zip::ZipFile.open(tmp_filename)
zip.add("tmp/", file_to_add.path)
zip.close
end
But this gives me a 403. Some help would be greatly appreciated.. It probably is not that hard I'm just Doing it Wrong™
I've managed to solve the problem with help from #ffoeg
The solution offered by #ffoeg didn't work quite so well for me since I was dealing with zip files > 500 MB which caused me problems on Heroku. I've therefor moved the zipping to a background process using resque:
app/workers/photo_zipper.rb:
require 'zip/zip'
require 'zip/zipfilesystem'
require 'open-uri'
class PhotoZipper
#queue = :photozip_queue
#I pass
def self.perform(id_of_object_with_images, id_of_user_to_be_notified)
user_mail = User.where(:id => id_of_user_to_be_notified).pluck(:email)
export = PhotoZipper.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)
Notifications.zip_ready(export.archive_url, user_mail).deliver
end
# Zipfile generator
def self.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)
object = ObjectWithImages.find(id_of_object_with_images)
photos = object.images
# base temp dir
temp_dir = Dir.mktmpdir
# path for zip we are about to create, I find that ruby zip needs to write to a real file
# This assumes the ObjectWithImages object has an attribute title which is a string.
zip_path = File.join(temp_dir, "#{object.title}_#{Date.today.to_s}.zip")
Zip::ZipOutputStream.open(zip_path) do |zos|
photos.each do |photo|
path = photo.photo.path
zos.put_next_entry(path)
zos.write photo.photo.file.read
end
end
#Find the user that made the request
user = User.find(id_of_user_to_be_notified)
#Create an export object associated to the user
export = user.exports.build
#Associate the created zip to the export
export.archive = File.open(zip_path)
#Upload the archive
export.save!
#return the export object
export
ensure
# clean up the tempdir now!
FileUtils.rm_rf temp_dir if temp_dir
end
end
app/controllers/photos_controller.rb:
format.zip do
#pick the last ObjectWithImages.. ofcourse you should include your own logic here
id_of_object_with_images = ObjectWithImages.last.id
#enqueue the Photozipper task
Resque.enqueue(PhotoZipper, id_of_object_with_images, current_user.id)
#don't keep the user waiting and flash a message with information about what's happening behind the scenes
redirect_to some_path, :notice => "Your zip is being created, you will receive an e-mail once this process is complete"
end
Many thanks to #ffoeg for helping me out. If your zips are smaller you could try #ffoeg's solution.
Here is my take. There could be typos but I think this is the gist of it :)
# action method, stream the zip
def download_photos_as_zip # silly name but you get the idea
generate_zip do |zipname, zip_path|
File.open(zip_path, 'rb') do |zf|
# you may need to set these to get the file to stream (if you care about that)
# self.last_modified
# self.etag
# self.response.headers['Content-Length']
self.response.headers['Content-Type'] = "application/zip"
self.response.headers['Content-Disposition'] = "attachment; filename=#{zipname}"
self.response.body = Enumerator.new do |out| # Enumerator is ruby 1.9
while !zf.eof? do
out << zf.read(4096)
end
end
end
end
end
# Zipfile generator
def generate_zip(&block)
photos = Photo.all
# base temp dir
temp_dir = Dir.mktempdir
# path for zip we are about to create, I find that ruby zip needs to write to a real file
zip_path = File.join(temp_dir, 'export.zip')
Zip::ZipFile::open(zip_path, true) do |zipfile|
photos.each do |photo|
zipfile.get_output_stream(photo.photo.identifier) do |io|
io.write photo.photo.file.read
end
end
end
# yield the zipfile to the action
block.call 'export.zip', zip_path
ensure
# clean up the tempdir now!
FileUtils.rm_rf temp_dir if temp_dir
end