I'm writing an Rails app which could upload files to storage. Big files are splitted into chunks from client (with JS) and upload parts to server.
As in development, I could simply open existed file and write following bytes into that.
(I'm using CarrierWave gem)
File.open(#up_file.link.path, "ab") do |f|
f.write(up_file_params[:link].read)
end
# This code worked when I upload to '/public' folder in development
However, now I want to use a FTP server to storage files. But I can't Concatenate new bytes with existed bytes.
def get_ftp_connection # create a new FTP connection
ftp = Net::FTP.new
ftp.connect(ENV['ftp_host'], ENV['ftp_port'])
begin
ftp.passive = ENV['ftp_passive']
ftp.login(ENV['ftp_user'], ENV['ftp_passwd'])
yield ftp
ensure
ftp.quit
end
end
.....
def create
.....
get_ftp_connection #up_file do |ftp|
full_path = ::File.dirname "#{ENV['ftp_folder']}/#{#up_file.link.path}"
base_name = File.basename(#up_file.link.to_s)
ftp.chdir(full_path)
ftp.putbinaryfile(up_file_params[:link].read, base_name)
end
end
I got ArgumentError (string contains null byte): at putbinaryfile... , any help :(
As mentioned in the comments you could completely download the file first and then upload to the ftp server. If that's not an option for whatever reason, you could append to the remote file as it's being uploaded:
ftp.storbinary("APPE #{base_name}", up_file_params[:link], Net::FTP::DEFAULT_BLOCKSIZE)
Related
I want to download uploaded files such as .zip, .xlsm, xlsx, pdf... ..., etc., that have been uploaded, I want to eventually compress them into a zip file for download.
However, if the uploaded files are not zipped, they will be downloaded with strange files stored in them.
In this case, .xlsm
Source Code
class DownloadZipsController < ::ApplicationController
def index
file_name = "#{Time.current.strftime("%Y%m%d_%H%M%S")}.zip"
zip_file = Tempfile.new(file_name)
Zip.unicode_names = true
Zip::OutputStream.open(zip_file.path) do |zip|
params[:product_ids].each do |product_id|
product = Product.find(product_id)
zip.put_next_entry("output.zip")
zip.print Net::HTTP.get URI.parse(product.submit_zip_file.url)
end
end
send_file zip_file.path,
type: "application/zip",
disposition: "attachment",
filename: file_name
zip_file.close
rescue => e
logger.debug e.backtrace
redirect_to request.referer, alert: e.message
end
end
Uploaded files are stored in AWS S3.
Is there any solution to this problem?
Office Documents such as .xlsm, .xlsx, .docx, and others are in fact zip files containing the document as a xml file plus additional resources.
The file tree you have shown in your screenshot shows the content of one such document if you interpret it as a zip file and unpack.
It appears that somewhere in your code, you have detected the document file as a zip file and interpreted it as such which resulted in its contents to be unpacked.
This is not apparent from the code you have posted though, so I would assume that you have some additional handling of zip files somewhere else (such as a function to download existing files which may then be send with a wrong content type to the browser, i.e as an application/zip rather than application/vnd.ms-excel.sheet.macroEnabled.12).
rails 6.1.3.2
aws-sdk-s3 gem
I currently have a rails app in production that uses ActiveStorage to attach image data to a wrapper Image model. It's currently using the local strategy to save images to disk and I am migrating it to S3. I am not using paperclip or anything similar.
I succeeded in setting it up. Currently it is set to use local primarily and have S3 as a mirror so that I can write to two places during the migration. However the documentation says that it will only save new images to S3 upon create and update of a record. I would like to "re-save" all models in production to force the migration to happen. Does anyone know how to do this?
Looks like it was already answered!
If you happen to be stuck with only access to the Rails Console like I was, this solution worked perfectly. If you copy-paste this code into the console, it will begin to produce output of the S3 uploads. After 5k of those, I was done. An immense thank you to Tayden for the solution.
all_services = [ActiveStorage::Blob.service.primary, *ActiveStorage::Blob.service.mirrors]
# Iterate through each blob
ActiveStorage::Blob.all.each do |blob|
# Select services where file exists
services = all_services.select { |file| file.exist? blob.key }
# Skip blob if file doesn't exist anywhere
next unless services.present?
# Select services where file doesn't exist
mirrors = all_services - services
# Open the local file (if one exists)
local_file = File.open(services.find{ |service| service.is_a? ActiveStorage::Service::DiskService }.path_for blob.key) if services.select{ |service| service.is_a? ActiveStorage::Service::DiskService }.any?
# Upload local file to mirrors (if one exists)
mirrors.each do |mirror|
mirror.upload blob.key, local_file, checksum: blob.checksum
end if local_file.present?
# If no local file exists then download a remote file and upload it to the mirrors (thanks #Rystraum)
services.first.open blob.key, checksum: blob.checksum do |temp_file|
mirrors.each do |mirror|
mirror.upload blob.key, temp_file, checksum: blob.checksum
end
end unless local_file.present?
I've seen a lot of threads that open up a file from a directory and rename it. But in my case, the user is uploading a file to a POST endpoint. I'd like to rename that file before uploading to my blob storage.
Here's what I have so far:
def picture
#file = params[:file]
#new_file_name = generate__filename()
#this line didn't work
File.rename(#file, #new_file_name + File.extname(#file))
# begin upload here
end
I'm testing this by selecting the form-data value as file in Postman. How do I rename this file?
Using Rails 3.2, carrier wave, and recently switched to store on Amazon S3. My setup and uploads are all working fine.
1. I have image_uploader.rb to upload and store images. Displaying them all works fine
2. I have file_uploader.rb to upload and store files. I've even taken it a step further to upload ZIP files and extract a version so that both the ZIP file and TXT files are stored in the correct place on S3.
My problem is I run a method on the TXT file. In the past, I used storage :file
With that I was able to:
Dir.chdir("public/uploads/")
import_file = Dir['*.TXT'].first
f = File.new(import_file)
Now, that I'm using storage :fog I can't get seem to retrieve/File.new/Open the file.
I see the file with the usual commands:
#upload1.team_file # stored file
#upload1.team_file.url # url
#upload1.team_file_url(:data_file).to_s # version created
I've been pouring through all kinds of very limited leads on retrieving and/or opening the file, but everything I try seems to return errors, such as:
Errno::ENOENT: No such file or directory - https://teamfiles.s3.amazonaws.com/data_files…
Thoughts on the difference here of retrieving and USING a file from AmazonS3? Thanks!
Pulling from multiple threads, APIs, etc. I'm answering my own question with what I've found. I welcome any corrections or improvements:
To retrieve carrierwave files uploaded to AmazonS3, you have to understand that open(#upload.file_url) or File.open(#upload.file_url) does NOT open the file, it only opens the PATH to the file. (ref: Ruby OpenURI )
I use: open_uri_url = open(#upload.file_url)
You then have to find the specific file in that path that you want. For me, I then find a ZIP file that was uploaded to AmazonS3 and Extract the specific file within the ZIP file that I want with a unique *.ABC extension:
zip_content_file = Zip::File.open(open_uri_url).map{|content| content if content.to_s.split('.').last == "ABC"}.compact.first
Now, from here, where to extract to?? I create a unique directory in the Rails tmp directory to extract the file to, use it and then delete the directory:
tmp_directory = "tmp/extracts/#{#upload.parent_id}/"
FileUtils.mkdir_p(tmp_directory) unless File.directory?(tmp_directory)
extract = zip_content_file.extract(tmp_directory + content_file.to_s)
Now with found from the AmazonS3 stored ZIP file and extracted, I can open, read, etc:
f = File.new(tmp_directory + extract.to_s)
I hope this helps with Carrierwave, AmazonS3, ZIP files and using them once uploaded.
I have this code, which writes a zip file to disk, reads it back, uploads to s3, then deletes the file:
compressed_file = some_temp_path
Zip::ZipOutputStream.open(compressed_file) do |zos|
some_file_list.each do |file|
zos.put_next_entry(file.some_title)
zos.print IO.read(file.path)
end
end # Write zip file
s3 = Aws::S3.new(S3_KEY, S3_SECRET)
bucket = Aws::S3::Bucket.create(s3, S3_BUCKET)
bucket.put("#{BUCKET_PATH}/archive.zip", IO.read(compressed_file), {}, 'authenticated-read')
File.delete(compressed_file)
This code works already but what I want is to not create the zip file anymore, to save a few steps. I was wondering if there is a way to export the zipfile data directly to s3 without having to first create a tmpfile, read it back, then delete it?
I think I just found the answer to my question.
It's Zip::ZipOutputStream.write_buffer. I'll check this out and update this answer when I get it working.
Update
It does work. My code is like this now:
compressed_filestream = Zip::ZipOutputStream.write_buffer do |zos|
some_file_list.each do |file|
zos.put_next_entry(file.some_title)
zos.print IO.read(file.path)
end
end # Outputs zipfile as StringIO
s3 = Aws::S3.new(S3_KEY, S3_SECRET)
bucket = Aws::S3::Bucket.create(s3, S3_BUCKET)
compressed_filestream.rewind
bucket.put("#{BUCKET_PATH}/archive.zip", compressed_filestream.read, {}, 'authenticated-read')
The write_buffer returns a StringIO and needs to rewind the stream first before reading it. Now I don't need to create and delete the tmpfile.
I'm just wondering now if write_buffer would be more memory extensive or heavier than open? Or is it the other way around?