I have a model Report to export all the Pdf attachments done for the bills. I have other models Bill, Upload.
Bill has many uploads
Tech Stack used
Ruby 2.5.8
Rails 5.0.7
Dummy Code
require 'fileutils'
require 'zip'
require 'open-uri'
class Report < ApplicationRecord
main_dir = "tmp/Attachment-#{self.id}"
FileUtils::mkdir_p main_dir
bills.each do |bill|
bill_directory_name = 'Bill_#{bill.id}'
bill.uploads.each do |bill_upload|
begin
FileUtils::cd main_dir do
FileUtils::mkdir_p "#{bill_directory_name}"
file = open(bill_upload.file.url, 'rb')
upload_name = "Invoice-#{bill_upload.file_file_name}-#{bill_upload.id}"
FileUtils::cd "#{bill_directory_name}" do
IO.copy_stream(file, "#{upload_name}")
end
end
rescue => exp
Airbrake.notify(exp)
end
end
end
path = "tmp/Attachment-#{self.id}"
archive = File.join(path, "Attachment-#{self.id}" + '.zip')
FileUtils.rm archive, force: true
Zip::File::open(archive, 'w') do |zipfile|
Dir["#{path}/**/**"].reject{|f| f==archive }.each do |file|
zipfile.add(file.sub(path + '/', ''), file)
end
end
end
Zip folder is created successfully. When I extract zip folder, I can see there are many sub folders created for bills. Each bill folder have uploaded PDF files as well and it's working.
Problem
Most of the time exported bill sub-directory contains PDF uploaded file. But, rarely sometime (once in a 100's of iteration), when I export report, I cannot see PDF file inside some of the bill sub directory. I have added error handling but there is no error generated. So, it's hard to reproduce this issue at my end.
How to confirm that this file copy command is successful IO.copy_stream()?
Please check dummy code and provide your feedback if I need to improve or modify any line of code?
file = open(bill_upload.file.url, 'rb') # File is read from S3
upload_name = "Invoice-#{bill_upload.file_file_name}-#{bill_upload.id}"
FileUtils::cd "#{bill_directory_name}" do
IO.copy_stream(file, "#{upload_name}")
end
Any help would be appreciated.
Related
The guide says that I can save an attachment to disc to run a process on it like this:
message.video.open do |file|
system '/path/to/virus/scanner', file.path
# ...
end
My model has an attachment defined as:
has_one_attached :zip
And then in the model I have defined:
def process_zip
zip.open do |file|
# process the zip file
end
end
However I am getting an error :
private method `open' called
on the zip.open call.
How can I save the zip locally for processing?
As an alternative in Rails 5.2 you can do this:
def process_zip
# Download the zip file in temp dir
zip_path = "#{Dir.tmpdir}/#{zip.filename}"
File.open(zip_path, 'wb') do |file|
file.write(zip.download)
end
Zip::File.open(zip_path) do |zip_file|
# process the zip file
# ...
puts "processing file #{zip_file}"
end
end
That’s an edge guide (note edgeguides.rubyonrails.org in the URL); it applies to the master branch of the rails/rails repository on GitHub. The latest changes in master haven’t been included in a released version of Rails yet.
You’re likely using Rails 5.2. Use edge Rails to take advantage of ActiveStorage::Blob#open:
gem "rails", github: "rails/rails"
I'm working on a project that is migrating data from a customers old_busted DB into rails objects to be worked on later. Similarly, I need to convert these objects into a CSV and upload it to a neutral FTP (this is to allow a coworker to build the example pages through Sugar CRM). I've created rake files to do all of this, and it was successful. Now, I'm going to continue this process for each object that I create in rails (relative to the previous DB) and, best case, wanted these generated when I run rake generate scaffold <object>.
Here is my import rake:
desc "Import Clients from db"
task :get_busted_clients => [:environment] do
#old_clients = Busted::Client.all
#old_clients.each do |row|
#client = Client.new();
#client.client_id = row.NUMBER
#client.save
end
end
Here is my CSV convert/FTP upload rake:
desc "Exports db's to local CSV and uploads them to FTP"
task :export_clients_CSV => [:environment] do
# Required libraries for CSV read/write and NET/FTP IO #
require 'csv'
require 'net/ftp'
# Pull all Editor objects into clients for reading #
clients = Client.all
puts "Creating CSV file for <Clients> and updating column names..."
# Open a new CSV file that uses the column headers from Client #
CSV.open("clients.csv", "wb",
:write_headers => true, :headers => Client.column_names) do |csv|
puts "--Loading each entry..."
# Load all entries from Client into the CSV file row by row #
clients.each do |client|
# This line specifically puts the attributes in the rows WITH RESPECT TO#
# THE COLUMNS
csv << client.attributes.values_at(*Client.column_names)
end
puts "--Done loading each entry..."
end
puts "...Data populated. Finished bulding CSV. Closing File."
puts "------------------------"
# Upload CSV File to FTP server by requesting new FTP connection, assigning credentials
# and informing the client what file to look for and what to name it
puts "Uploading <Clients>..."
ftp = Net::FTP.new('192.168.xxx.xxx')
ftp.login(user = "user", passwd = "passwd")
ftp.puttextfile("clients.csv", "clients.csv")
ftp.quit()
puts "...Finished."
end
I ran rake generate g get_busted and put this in my get_busted_generator.rb:
class GetBustedGenerator < Rails::Generators::NamedBase
source_root File.expand_path('../templates', __FILE__)
def generate_get_busted
copy_file "getbusted.rake", "lib/tasks/#{file_name}.rake"
end
end
After that, I got lost. I can't find anything on templating a rake file or the syntax included to do so.
Rails has been a recent endeavor and I may be overlooking something in terms of design of the solution to my problem.
TL;DR: Is templating a rake file a bad thing? Solution alternatives? If not, whats the syntax for generating either script custom to the object (or point me in the direction, please).
I've been knocking my head around with Heroku, while trying to download a zip file with all my receipt files data.
The files are stored on amazon s3 and it all works fine on my development machine..
I thought it had to do with Tempfile, and abandoned that previous solution, since heroku has some strict policies with their filesystem, so i used the tmp folder, but the problem doesn't seem to be there. I already tried to load directly from s3 (using openUri) to the zip file, but it doesn't seem to work either on Heroku.
What might be wrong with my code for Heroku not loading the files to the zip?
Here is my model method :
def zip_receipts(search_hash=nil)
require 'zip/zip'
require 'zip/zipfilesystem'
t=File.open("#{Rails.root}/tmp/#{Digest::MD5.hexdigest(rand(12).to_s)}_#{Process.pid}",'w')
# t = Tempfile.new(Digest::MD5.hexdigest(rand(12).to_s))
# Give the path of the temp file to the zip outputstream, it won't try to open it as an archive.
Zip::ZipOutputStream.open(t.path) do |zos|
logger.debug("search hash Zip: #{search_hash.inspect}")
self.feed(search_hash).each do |receipt|
begin
require 'open-uri'
require 'tempfile'
#configures filename
filen = File.basename(receipt.receipt_file_file_name)
ext= File.extname(filen)
filen_noext = File.basename(receipt.receipt_file_file_name, '.*')
filen=filen_noext+SecureRandom.hex(10)+ext
logger.info("Info Zip - Filename: #{filen}")
# Create a new entry on the zip file
zos.put_next_entry(filen)
# logger.info("Info Zip - Added entry: #{zos.inspect}")
# Add the contents of the file, reading directly from amazon
tfilepath= "#{Rails.root}/tmp/#{File.basename(filen,ext)}_#{Process.pid}"
open(tfilepath,"wb") do |file|
file << open(receipt.authenticated_url(:original),:ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE).read
end
zos.print IO.binread tfilepath
# logger.info("Info Zip - Extracted from amazon: #{zos.inspect}")
rescue Exception => e
logger.info("exception #{e}")
end # closes the exception begin
end #closes receipts cycle
end #closes zip file stream cycle
# The temp file will be deleted some time...
t.close
#returns the path for send file controller to act
t.path
end
My controller:
def download_all
#user = User.find_by_id(params[:user_id])
filepath = #user.zip_receipts
# Send it using the right mime type, with a download window and some nice file name.
send_file(filepath,type: 'application/zip', disposition: 'attachment',filename:"MyReceipts.zip")
end
And I write also my view and routes, so that it might serve anyone else trying to implement a download all feature
routes.rb
resources :users do
post 'download_all'
end
my view
<%= link_to "Download receipts", user_download_all_path(user_id:user.id), method: :post %>
The problem seemed to be with the search hash, and the sql query, and not the code itself. For some reason, the receipts get listed, but aren't downloaded. So it is an all different issue
In the end i have this code for the model
def zip_receipts(search_hash=nil)
require 'zip/zip'
require 'zip/zipfilesystem'
t=File.open("#{Rails.root}/tmp/MyReceipts.zip_#{Process.pid}","w")
# t = Tempfile.new(Digest::MD5.hexdigest(rand(12).to_s))
#"#{Rails.root}/tmp/RecibosOnline#{SecureRandom.hex(10)}.zip"
puts "Zip- Receipts About to enter"
# Give the path of the temp file to the zip outputstream, it won't try to open it as an archive.
Zip::ZipOutputStream.open(t.path) do |zos|
self.feed(search_hash).each do |receipt|
begin
require 'open-uri'
require 'tempfile'
filen = File.basename(receipt.receipt_file_file_name)
ext= File.extname(filen)
filen_noext = File.basename(receipt.receipt_file_file_name, '.*')
filen=filen_noext+SecureRandom.hex(10)+ext
# puts "Info Zip - Filename: #{filen}"
# Create a new entry on the zip file
zos.put_next_entry(filen)
zos.print open(receipt.authenticated_url(:original),:ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE).read
rescue Exception => e
puts "exception #{e}"
end # closes the exception begin
end #closes receipts cycle
end #closes zip file stream cycle
# The temp file will be deleted some time...
t.close
#returns the path for send file controller to act
t.path
end
I'm using CarrierWave to store files in gridfs, but having problems with opening them from my model.
Here are my configs:
/config/initialize/carrierwave.rb
CarrierWave.configure do |config|
config.grid_fs_database = Mongoid.database.name
config.grid_fs_host = Mongoid.config.master.connection.host
config.storage = :grid_fs
config.grid_fs_access_url = "/files"
end
/app/controllers/gridfs_controller.rb
/require 'mongo'
class GridfsController < ActionController::Metal
def serve
gridfs_path = env["PATH_INFO"].gsub("/files/", "")
begin
gridfs_file = Mongo::GridFileSystem.new(Mongoid.database).open(gridfs_path, 'r')
self.response_body = gridfs_file.read
self.content_type = gridfs_file.content_type
rescue
self.status = :file_not_found
self.content_type = 'text/plain'
self.response_body = ''
end
end
end
/app/uploaders/list_uploader.rb
class ListUploader < CarrierWave::Uploader::Base
storage :grid_fs
def store_dir
"uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end
and in routes
match "/files/uploads/*path" => "gridfs#serve"
So, I have a model, which have a text file
class Campaign
include Mongoid::Document
mount_uploader :list, ListUploader
When I'm calling something like <%=link_to "List", #campaign.list.url %> from my view, it opens fine. But when I'm trying something like File.open("#{campaign.list.url}", "r") from campaign model, it fails. It gives me false even when I'm calling File.exists?("/files/uploads/campaign/list/4eb02c4d6b1c0f02b200000b/list.txt"), which is a proper url for that file. So, the question is how should I call it, to open the file from model? And for some reasons, it is important to open it from model. Any suggestions would help, thank you.
Carrierwave url with mongodb gridfs is not a physical path. Its merely a logical route to download the file from gridfs. Thats why you cannot access it from ruby File.open. Check out the below snippet from rails console trying to open the file from gridfs
File.open(User.first.image.pic.url,'r')
Errno::ENOENT: No such file or directory - /images/uploads/e5a1007d34.jpg
see it throw No such file or directory., So you have to download a file instead opening by
>> require 'open-uri'
>> open('image.jpg', 'wb') do |file|
?> file << open('http://0.0.0.0:3000' + (User.first.image.pic.url)).read
>> p file
>> end
#<File:image.jpg>
=> #<File:image.png (closed)>
What is the best way to get a temporary directory with nothing in it using Ruby on Rails? I need the API to be cross-platform compatible. The stdlib tmpdir won't work.
The Dir object has a method mktmpdir which creates a temporary directory:
require 'tmpdir' # Not needed if you are using rails.
Dir.mktmpdir do |dir|
puts "My new temp dir: #{dir}"
end
The temporary directory will be removed after execution of the block.
The Dir#tmpdir function in the Ruby core (not stdlib that you linked to) should be cross-platform.
To use this function you need to require 'tmpdir'.
A general aprox I'm using now:
def in_tmpdir
path = File.expand_path "#{Dir.tmpdir}/#{Time.now.to_i}#{rand(1000)}/"
FileUtils.mkdir_p path
yield path
ensure
FileUtils.rm_rf( path ) if File.exists?( path )
end
So in your code you can:
in_tmpdir do |tmpdir|
puts "My tmp dir: #{tmpdir}"
# work with files in the dir
end
The temporary dir will be removed automatically when your method will finish.
Ruby has Dir#mktmpdir, so just use that.
require 'tempfile'
Dir.mktmpdir('prefix_unique_to_your_program') do |dir|
### your work here ###
end
See http://www.ruby-doc.org/stdlib-1.9.3/libdoc/tmpdir/rdoc/Dir.html
Or build your own using Tempfile tempfile that is process and thread unique, so just use that to build a quick Tempdir.
require 'tempfile'
Tempfile.open('prefix_unique_to_your_program') do |tmp|
tmp_dir = tmp.path + "_dir"
begin
FileUtils.mkdir_p(tmp_dir)
### your work here ###
ensure
FileUtils.rm_rf(tmp_dir)
end
end
See http://www.ruby-doc.org/stdlib-1.9.3/libdoc/tempfile/rdoc/Tempfile.html for optional suffix/prefix options.
require 'tmpdir' # not needed if you are loading Rails
tmp_dir = File.join(Dir::tmpdir, "my_app_#{Time.now.to_i}_#{rand(100)}")
Dir.mkdir(tmp_dir)
Works for me.
You can use Dir.mktmpdir.
Using a block will get rid of the temporary directory when it closes.
Dir.mktmpdir do |dir|
File.open("#{dir}/foo", 'w') { |f| f.write('foo') }
end
Or if you need multiple temp directories to exist at the same time, for example
context 'when there are duplicate tasks' do
it 'raises an DuplicateTask error' do
begin
tmp_dir1 = Dir.mktmpdir('foo')
tmp_dir2 = Dir.mktmpdir('bar')
File.new("#{tmp_dir1}/task_name", 'w+')
File.new("#{tmp_dir2}/task_name", 'w+')
expect { subject.filepath('task_name') }.to raise_error(TaskFinder::DuplicateTask)
ensure
FileUtils.remove_entry tmp_dir1
FileUtils.remove_entry tmp_dir2
end
end
end
Dir.mktmpdir creates a temporary directory under Dir.tmpdir (you'll need to require 'tmpdir' to see what that evaluates to).
If you want to use your own path, Dir.mktmpdir takes an optional second argument tmpdir if non-nil value is given. E.g.
Dir.mktmpdir(nil, "/var/tmp") { |dir| "dir is '/var/tmp/d...'" }
I started to tackle this by hijacking Tempfile, see below.
It should clean itself up as Tempfile does, but doesn't always yet..
It's yet to delete files in the tempdir.
Anyway I share this here, might be useful as a starting point.
require 'tempfile'
class Tempdir < Tempfile
require 'tmpdir'
def initialize(basename, tmpdir = Dir::tmpdir)
super
p = self.path
File.delete(p)
Dir.mkdir(p)
end
def unlink # copied from tempfile.rb
# keep this order for thread safeness
begin
Dir.unlink(#tmpname) if File.exist?(#tmpname)
##cleanlist.delete(#tmpname)
#data = #tmpname = nil
ObjectSpace.undefine_finalizer(self)
rescue Errno::EACCES
# may not be able to unlink on Windows; just ignore
end
end
end
This can be used the same way as Tempfile, eg:
Tempdir.new('foo')
All methods on Tempfile , and in turn, File should work.
Just briefly tested it, so no guarantees.
Update: gem install files, then
require "files"
dir = Files do
file "hello.txt", "stuff"
end
See below for more examples.
Here's another solution, inspired by a few other answers. This one is suitable for inclusion in a test (e.g. rspec or spec_helper.rb). It makes a temporary dir based on the name of the including file, stores it in an instance variable so it persists for the duration of the test (but is not shared between tests), and deletes it on exit (or optionally doesn't, if you want to check its contents after the test run).
def temp_dir options = {:remove => true}
#temp_dir ||= begin
require 'tmpdir'
require 'fileutils'
called_from = File.basename caller.first.split(':').first, ".rb"
path = File.join(Dir::tmpdir, "#{called_from}_#{Time.now.to_i}_#{rand(1000)}")
Dir.mkdir(path)
at_exit {FileUtils.rm_rf(path) if File.exists?(path)} if options[:remove]
File.new path
end
end
(You could also use Dir.mktmpdir (which has been around since Ruby 1.8.7) instead of Dir.mkdir but I find the API of that method confusing, not to mention the naming algorithm.)
Usage example (and another useful test method):
def write name, contents = "contents of #{name}"
path = "#{temp_dir}/#{name}"
File.open(path, "w") do |f|
f.write contents
end
File.new path
end
describe "#write" do
before do
#hello = write "hello.txt"
#goodbye = write "goodbye.txt", "farewell"
end
it "uses temp_dir" do
File.dirname(#hello).should == temp_dir
File.dirname(#goodbye).should == temp_dir
end
it "writes a default value" do
File.read(#hello).should == "contents of hello.txt"
end
it "writes a given value" do
# since write returns a File instance, we can call read on it
#goodbye.read.should == "farewell"
end
end
Update: I've used this code to kickstart a gem I'm calling files which intends to make it super-easy to create directories and files for temporary (e.g. unit test) use. See https://github.com/alexch/files and https://rubygems.org/gems/files . For example:
require "files"
files = Files do # creates a temporary directory inside Dir.tmpdir
file "hello.txt" # creates file "hello.txt" containing "contents of hello.txt"
dir "web" do # creates directory "web"
file "snippet.html", # creates file "web/snippet.html"...
"<h1>Fix this!</h1>" # ...containing "<h1>Fix this!</h1>"
dir "img" do # creates directory "web/img"
file File.new("data/hello.png") # containing a copy of hello.png
file "hi.png", File.new("data/hello.png") # and a copy of hello.png named hi.png
end
end
end # returns a string with the path to the directory
Check out the Ruby STemp library: http://ruby-stemp.rubyforge.org/rdoc/
If you do something like this:
dirname = STemp.mkdtemp("#{Dir.tmpdir}/directory-name-template-XXXXXXXX")
dirname will be a string that points to a directory that's guaranteed not to exist previously. You get to define what you want the directory name to start with. The X's get replaced with random characters.
EDIT: someone mentioned this didn't work for them on 1.9, so YMMV.