I'm using JPEGCAM to allow users to take a profile pic with their web cam. This uploads a temporary file as so:
def ajax_photo_upload
File.open(upload_path, 'w:ASCII-8BIT') do |f|
f.write request.raw_post
end
# #user.photo = File.open(upload_path)
#user.assign_attributes(
:photo => File.open(upload_path),
:orig_filename => "#{current_user.full_name}.jpg"
)
if #user.save
respond_to do |format|
.....
private
def upload_path # is used in upload and create
file_name = session[:session_id].to_s + '.jpg'
File.join(::Rails.root.to_s, 'public', 'temp', file_name)
end
What's the best way to go about deleting this temporary file safely? Thanks
When you know that you are done with the file:
File.delete(path_to_file) if File.exist?(path_to_file)
Another thing: make sure that you always close files that you have opened, an operating system can only handle a certain number of open files/file descriptors and you'll may run into strange bugs when you pass that limit... So when you want to open files in Ruby always either use the block form:
File.open(path) do |f|
# ...
end
and Ruby will close the file automatically for you. If the block form is not usable, you have to close files by yourself:
f = File.open(path)
# ...
f.close
So make sure to close the file that you pass to #user.assign_attributes(...)...
If you are sure you are done with it, why not just use FileUtils.rm or FileUtils.rm_f?
FileUtils.rm_f(upload_path)
http://www.ruby-doc.org/stdlib-1.9.3/libdoc/fileutils/rdoc/FileUtils.html#method-c-rm_f
You could also ignore this in Rails, and have a cron that wakes up and deletes files older than a day from the temp directory that match these temp files. That has the benefit of some margin for error if a file fails to be reprocessed - you don't rm it immediately - and the file operation is not done on the request/response loop for Rails, which will then respond a bit faster.
Related
I have the following code to connect my rails app to my FTP. This works great. However, I want to use open-uri to open the csv file so I can parse it. Any ideas how to do this? I think it's an easy thing to do but I'm missing something.
require 'net/ftp'
ftp = Net::FTP.new
ftp.connect("xxx.xxx.xx.xxx",21)
ftp.login("xxxxx","xxxx")
ftp.chdir("/")
ftp.passive = true
puts ftp.list("TEST.csv")
You'll need to use #gettextfile.
A) Get the file to a local temporary file and read its content
# Creating a tmp file can be done differently as well.
# It may also be omitted, in which case `gettextfile`
# will create a file in the current directory.
Dir::Tmpname.create(['TEST', ['.csv']) do |file_name|
ftp.gettextfile('TEST.csv', file_name)
content = File.read(file_name)
end
B) Pass a block to gettextfile and get the content one line at a time
content = ''
ftp.gettextfile('TEST.csv') do |line|
content << line
end
I need to serve some data from my database in a zip file, streaming it on the fly such that:
I do not write a temporary file to disk
I do not compose the whole file in RAM
I know that I can do streaming generation of zip files to the filesystemk using ZipOutputStream as here. I also know that I can do streaming output from a rails controller by setting response_body to a Proc as here. What I need (I think) is a way of plugging those two things together. Can I make rails serve a response from a ZipOutputStream? Can I get ZipOutputStream give me incremental chunks of data that I can feed into my response_body Proc? Or is there another way?
Short Version
https://github.com/fringd/zipline
Long Version
so jo5h's answer didn't work for me in rails 3.1.1
i found a youtube video that helped, though.
http://www.youtube.com/watch?v=K0XvnspdPsc
the crux of it is creating an object that responds to each... this is what i did:
class ZipGenerator
def initialize(model)
#model = model
end
def each( &block )
output = Object.new
output.define_singleton_method :tell, Proc.new { 0 }
output.define_singleton_method :pos=, Proc.new { |x| 0 }
output.define_singleton_method :<<, Proc.new { |x| block.call(x) }
output.define_singleton_method :close, Proc.new { nil }
Zip::IoZip.open(output) do |zip|
#model.attachments.all.each do |attachment|
zip.put_next_entry "#{attachment.name}.pdf"
file = attachment.file.file.send :file
file = File.open(file) if file.is_a? String
while buffer = file.read(2048)
zip << buffer
end
end
end
sleep 10
end
end
def getzip
self.response_body = ZipGenerator.new(#model)
#this is a hack to preven middleware from buffering
headers['Last-Modified'] = Time.now.to_s
end
EDIT:
the above solution didn't ACTUALLY work... the problem is that rubyzip needs to jump around the file to rewrite the headers for entries as it goes. particularly it needs to write the compressed size BEFORE it writes the data. this is just not possible in a truly streaming situation... so ultimately this task may be impossible. there is a chance that it might be possible to buffer a whole file at a time, but this seemed less worth it. ultimately i just wrote to a tmp file... on heroku i can write to Rails.root/tmp less instant feedback, and not ideal, but neccessary.
ANOTHER EDIT:
i got another idea recently... we COULD know the compressed size of the files if we do not compress them. the plan goes something like this:
subclass the ZipStreamOutput class as follows:
always use the "stored" compression method, in other words do not compress
ensure we never seek backwards to change file headers, get it all right up front
rewrite any code related to TOC that seeks
I haven't tried to implement this yet, but will report back if there's any success.
OK ONE LAST EDIT:
In the zip standard: http://en.wikipedia.org/wiki/Zip_(file_format)#File_headers
they mention that there's a bit you can flip to put the size, compressed size and crc AFTER a file. so my new plan was to subclass zipoutput stream so that it
sets this flag
writes sizes and CRCs after the data
never rewinds output
furthermore i needed to get all the hacks in order to stream output in rails fixed up...
anyways it all worked!
here's a gem!
https://github.com/fringd/zipline
I had a similar issue. I didn't need to stream directly, but only had your first case of not wanting to write a temp file. You can easily modify ZipOutputStream to accept an IO object instead of just a filename.
module Zip
class IOOutputStream < ZipOutputStream
def initialize io
super '-'
#outputStream = io
end
def stream
#outputStream
end
end
end
From there, it should just be a matter of using the new Zip::IOOutputStream in your Proc. In your controller, you'd probably do something like:
self.response_body = proc do |response, output|
Zip::IOOutputStream.open(output) do |zip|
my_files.each do |file|
zip.put_next_entry file
zip << IO.read file
end
end
end
It is now possible to do this directly:
class SomeController < ApplicationController
def some_action
compressed_filestream = Zip::ZipOutputStream.write_buffer do |zos|
zos.put_next_entry "some/filename.ext"
zos.print data
end
compressed_filestream .rewind
respond_to do |format|
format.zip do
send_data compressed_filestream .read, filename: "some.zip"
end
end
# or some other return of send_data
end
end
This is the link you want:
http://info.michael-simons.eu/2008/01/21/using-rubyzip-to-create-zip-files-on-the-fly/
It builds and generates the zipfile using ZipOutputStream and then uses send_file to send it directly out from the controller.
Use chunked HTTP transfer encoding for output: HTTP header "Transfer-Encoding: chunked" and restructure the output according to the chunked encoding specification, so no need to know the resulting ZIP file size at the begginning of the transfer. Can be easily coded in Ruby with the help of Open3.popen3 and threads.
Something I'm not getting about the version process...
I have a zip file with a file inside, and I want to upload the file as a "version" of the zip:
Uploader:
version :specificFile do
process :extract_file
end
def extract_file
file = nil
Zip::ZipFile.open(current_path) do |zip_file|
file = zip_file.select{|f| f.name.match(/specificFile/)}.first
zip_file.extract(file, "tmp/" + file.name.gsub("/", "-")){ true }
end
File.open("tmp/" + file.name.gsub("/", "-"))
end
Usage:
=link_to "Specific File", instance.uploader.specificFile.url
Only this just nets me two copies of the zip. Clearly, there's something I'm missing about how version / process works, and I haven't been able to find documentation that actually explains the magic.
So how do I do this, and what am I missing?
This provided the "why", although it took a bit to understand:
How do you create a new file in a CarrierWave process?
To rephrase, when you go to create a version, carrierwave makes a copy of the file and then passes the process the file path. When the process exits, carrierwave will upload the contents of that path - not the file the process returns, which is what I thought was going on.
Working code:
version :specificFile do
process :extract_file
def full_filename (for_file = model.logo.file)
"SpecificFile.ext"
end
end
def extract_plist
file = nil
Zip::ZipFile.open(current_path) do |zip_file|
file = zip_file.select{|f| f.name.match(/specificFile/)}.first
zip_file.extract(file, "tmp/" + file.name.gsub("/", "-")){ true }
end
File.delete(current_path)
FileUtils.cp("tmp/" + file.name.gsub("/", "-"), current_path)
end
So, to make what I want to happen, happen, I:
Tell carrierwave to use a particular filename. I'm using a hardcoded value but you should be able to use whatever you want.
Overwrite the contents of current_path with the contents you want under the version name. In my case, I can't just overwrite the zip while I'm "in it" (I think), so I make a copy of the file I care about and overwrite the zip via File and FileUtils.
PS - It would be nice to avoid the duplication of the zip, but it doesn't look like you can tell carrierwave to skip the duplication.
I'm providing users with the ability to download an extremely large amount of data via CSV. To do this, I'm using Sidekiq and putting the task off into a background job once they've initiated it. What I've done in the background job is generate a csv containing all of the proper data, storing it in /tmp and then call save! on my model, passing the location of the file to the paperclip attribute which then goes off and is stored in S3.
All of this is working perfectly fine locally. My problem now lies with Heroku and it's ability to store files for a short duration dependent on what node you're on. My background job is unable to find the tmp file that gets saved because of how Heroku deals with these files. I guess I'm searching for a better way to do this. If there's some way that everything can be done in-memory, that would be awesome. The only problem is that paperclip expects an actual file object as an attribute when you're saving the model. Here's what my background job looks like:
class CsvWorker
include Sidekiq::Worker
def perform(report_id)
puts "Starting the jobz!"
report = Report.find(report_id)
items = query_ranged_downloads(report.start_date, report.end_date)
csv = compile_csv(items)
update_report(report.id, csv)
end
def update_report(report_id, csv)
report = Report.find(report_id)
report.update_attributes(csv: csv, status: true)
report.save!
end
def compile_csv(items)
clean_items = items.compact
path = File.new("#{Rails.root}/tmp/uploads/downloads_by_title_#{Process.pid}.csv", "w")
csv_string = CSV.open(path, "w") do |csv|
csv << ["Item Name", "Parent", "Download Count"]
clean_items.each do |row|
if !row.item.nil? && !row.item.parent.nil?
csv << [
row.item.name,
row.item.parent.name,
row.download_count
]
end
end
end
return path
end
end
I've omitted the query method for readabilities sake.
I don't think Heroku's temporary file storage is the problem here. The warnings around that mostly center around the facts that a) dynos are ephemeral, so anything you write can and will disappear without notice; and b) dynos are interchangeable, so the presence of inter-request tempfiles are a matter of luck when you have more than one web dyno running. However, in no situation do temporary files just vanish while your worker is running.
One thing I notice is that you're actually creating two temporary files with the same name:
> path = File.new("/tmp/filename", "w")
=> #<File:/tmp/filename>
> path.fileno
=> 3
> CSV.open(path, "w") do |csv| csv << %w(foo bar baz); puts csv.fileno end
4
=> nil
You could change the path = line to just set the filename (instead of opening it for writing), and then make update_report open the filename for reading. I haven't dug into what Paperclip does when you give it an empty, already-overwritten, opened-for-writing file handle, but changing that flow may well fix the issue.
Alternately, you could do this in memory instead: generate the CSV as a string and give it to Paperclip as a StringIO. (Paperclip supports certain non-file objects, including StringIOs, using e.g. Paperclip::StringioAdapter.) Try something like:
# returns a CSV as a string
def compile_csv(items)
CSV.generate do |csv|
# ...
end
end
def update_report(report_id, csv)
report = Report.find(report_id)
report.update_attributes(csv: StringIO.new(csv), status: true)
report.save!
end
I have a file server which has to be deleted after a user downloads it. I send the file with send_file, but I can't just put a File.delete(path) after that line because send_file returns immediatly, so the file is deleted even before the user receives the complete file.
How do I fix this? I think it's a common problem but haven't found a solution yet.
Here is my code below:
def export_rawdata
#device = Device.find_by_id( params[:id] )
#messages = #device.devmessages.all( :order => "id", :conditions => ["packettime >= ? and packettime <= ?", params[:start_time], params[:end_time]] )
raw_data_path = "#{Rails.root}/tmp/exports/#{#device.s_dev_name}.csv"
FasterCSV.open(raw_data_path, "w+") do |csv|
csv << ["Packet","Created At"]
#messages.each_with_index do |m,i|
x = m.created_at
csv << [m.message, x.strftime('%h %d, %G %r')]
end
end
send_file raw_data_path, :type => "text/csv", :x_sendfile => true, :streaming => false
end
Can Anyone please suggest me, how to delete a file from the server once the download is completed in the client side?
I have read (once more) the section in the "Rails Guides for Controller, Section sending files". You have chosen the options so your method returns immediately, so there is no chance to get the information that the download has completed. Depending on the network and the client behavior, you will never know if the file download was completed.
So you could think about a workaround (no real code, only the idea):
If you know how long the download may need, define the interval (in minutes) that is at least needed.
Store your temporary files in folders that are segmented by time.
Create new folders for new segments, when the time has come.
From time to time, delete old folders (segments older than x * delta minutes) and delete the whole folder.
By that technique, depending on the usage, you will have a constant overhead for older temporary files.