I am using rails server. i am sending core http request.
in request.body contents a file which I want to be uploaded. This request.body is StringIo object. I want to upload this file to my server.
This writes the file to disk in 1mb (1024**2) chunks. Reading the whole file in at once can leave you open to a DOS with huge files.
File.open("where-you-want-the-file", "w") do |f|
while blk = request.body.read(1024**2)
f << blk
end
end
Related
I want to download uploaded files such as .zip, .xlsm, xlsx, pdf... ..., etc., that have been uploaded, I want to eventually compress them into a zip file for download.
However, if the uploaded files are not zipped, they will be downloaded with strange files stored in them.
In this case, .xlsm
Source Code
class DownloadZipsController < ::ApplicationController
def index
file_name = "#{Time.current.strftime("%Y%m%d_%H%M%S")}.zip"
zip_file = Tempfile.new(file_name)
Zip.unicode_names = true
Zip::OutputStream.open(zip_file.path) do |zip|
params[:product_ids].each do |product_id|
product = Product.find(product_id)
zip.put_next_entry("output.zip")
zip.print Net::HTTP.get URI.parse(product.submit_zip_file.url)
end
end
send_file zip_file.path,
type: "application/zip",
disposition: "attachment",
filename: file_name
zip_file.close
rescue => e
logger.debug e.backtrace
redirect_to request.referer, alert: e.message
end
end
Uploaded files are stored in AWS S3.
Is there any solution to this problem?
Office Documents such as .xlsm, .xlsx, .docx, and others are in fact zip files containing the document as a xml file plus additional resources.
The file tree you have shown in your screenshot shows the content of one such document if you interpret it as a zip file and unpack.
It appears that somewhere in your code, you have detected the document file as a zip file and interpreted it as such which resulted in its contents to be unpacked.
This is not apparent from the code you have posted though, so I would assume that you have some additional handling of zip files somewhere else (such as a function to download existing files which may then be send with a wrong content type to the browser, i.e as an application/zip rather than application/vnd.ms-excel.sheet.macroEnabled.12).
My App saves Files to the Filesystem via an base64 encoded API. Here is my (simplified) controller:
def create
file = "name.pdf"
folder = "downloads"
FileUtils::mkdir_p folder
f = File.open File.join(folder, file), "wb"
if f.write Base64.decode64(params[:file])
f.close
end
Now I want to test for an actual file creation. How do I check whether there is a file or not?
Thanks for your input!
I'm writing an Rails app which could upload files to storage. Big files are splitted into chunks from client (with JS) and upload parts to server.
As in development, I could simply open existed file and write following bytes into that.
(I'm using CarrierWave gem)
File.open(#up_file.link.path, "ab") do |f|
f.write(up_file_params[:link].read)
end
# This code worked when I upload to '/public' folder in development
However, now I want to use a FTP server to storage files. But I can't Concatenate new bytes with existed bytes.
def get_ftp_connection # create a new FTP connection
ftp = Net::FTP.new
ftp.connect(ENV['ftp_host'], ENV['ftp_port'])
begin
ftp.passive = ENV['ftp_passive']
ftp.login(ENV['ftp_user'], ENV['ftp_passwd'])
yield ftp
ensure
ftp.quit
end
end
.....
def create
.....
get_ftp_connection #up_file do |ftp|
full_path = ::File.dirname "#{ENV['ftp_folder']}/#{#up_file.link.path}"
base_name = File.basename(#up_file.link.to_s)
ftp.chdir(full_path)
ftp.putbinaryfile(up_file_params[:link].read, base_name)
end
end
I got ArgumentError (string contains null byte): at putbinaryfile... , any help :(
As mentioned in the comments you could completely download the file first and then upload to the ftp server. If that's not an option for whatever reason, you could append to the remote file as it's being uploaded:
ftp.storbinary("APPE #{base_name}", up_file_params[:link], Net::FTP::DEFAULT_BLOCKSIZE)
I have this code, which writes a zip file to disk, reads it back, uploads to s3, then deletes the file:
compressed_file = some_temp_path
Zip::ZipOutputStream.open(compressed_file) do |zos|
some_file_list.each do |file|
zos.put_next_entry(file.some_title)
zos.print IO.read(file.path)
end
end # Write zip file
s3 = Aws::S3.new(S3_KEY, S3_SECRET)
bucket = Aws::S3::Bucket.create(s3, S3_BUCKET)
bucket.put("#{BUCKET_PATH}/archive.zip", IO.read(compressed_file), {}, 'authenticated-read')
File.delete(compressed_file)
This code works already but what I want is to not create the zip file anymore, to save a few steps. I was wondering if there is a way to export the zipfile data directly to s3 without having to first create a tmpfile, read it back, then delete it?
I think I just found the answer to my question.
It's Zip::ZipOutputStream.write_buffer. I'll check this out and update this answer when I get it working.
Update
It does work. My code is like this now:
compressed_filestream = Zip::ZipOutputStream.write_buffer do |zos|
some_file_list.each do |file|
zos.put_next_entry(file.some_title)
zos.print IO.read(file.path)
end
end # Outputs zipfile as StringIO
s3 = Aws::S3.new(S3_KEY, S3_SECRET)
bucket = Aws::S3::Bucket.create(s3, S3_BUCKET)
compressed_filestream.rewind
bucket.put("#{BUCKET_PATH}/archive.zip", compressed_filestream.read, {}, 'authenticated-read')
The write_buffer returns a StringIO and needs to rewind the stream first before reading it. Now I don't need to create and delete the tmpfile.
I'm just wondering now if write_buffer would be more memory extensive or heavier than open? Or is it the other way around?
I'm trying to use OpenURI to download a file from S3, and then save it locally so I can send the file as an attachment with ActionMailer.
Something strange is going on. The images being downloaded and attached are corrupt, the bottom parts of the images are missing.
Here's the code:
require 'open-uri'
open("#{Rails.root.to_s}/tmp/#{a.attachment_file_name}", "wb") do |file|
source_url = a.authenticated_url()
io = open(URI.parse(source_url).to_s)
file << io.read
attachments[a.attachment_file_name] = File.read("#{Rails.root.to_s}/tmp/#{a.attachment_file_name}")
end
a is the attachment from ActionMailer.
What can I try next?
It looks like you're trying to read the file before it's been closed, which could leave part of the file buffer unwritten.
I'd do it like this:
require 'open-uri'
source_url = a.authenticated_url()
attachment_file = "#{Rails.root.to_s}/tmp/#{a.attachment_file_name}"
open(attachment_file, "wb") do |file|
file.print open(source_url, &:read)
end
attachments[a.attachment_file_name] = File.read(attachment_file)
It looks like source_url = a.authenticated_url() will be a string, so parsing the string into a URI then doing to_s on it will be redundant unless URI is doing some normalizing, which I don't think it does.
Based on my sysadmin experience: A side task is cleaning up the downloaded/spooled files. They could be deleted immediately after being attached, or you could have a cron job that runs daily, deleting all spooled files over one day old.
An additional concern for this is there is no error handling in case the URL can't be read, causing the attachment to fail. Using a temp spool file you could check for the existence of the file. Even better, you should probably be prepared to handle an exception if the server returns a 400 or 500 error.
To avoid using a temporary spool file try this untested code:
require 'open-uri'
source_url = a.authenticated_url()
attachments[a.attachment_file_name] = open(source_url, &:read)