retrieve carrierwave file uploaded to Amazon S3 - ruby-on-rails

Using Rails 3.2, carrier wave, and recently switched to store on Amazon S3. My setup and uploads are all working fine.
1. I have image_uploader.rb to upload and store images. Displaying them all works fine
2. I have file_uploader.rb to upload and store files. I've even taken it a step further to upload ZIP files and extract a version so that both the ZIP file and TXT files are stored in the correct place on S3.
My problem is I run a method on the TXT file. In the past, I used storage :file
With that I was able to:
Dir.chdir("public/uploads/")
import_file = Dir['*.TXT'].first
f = File.new(import_file)
Now, that I'm using storage :fog I can't get seem to retrieve/File.new/Open the file.
I see the file with the usual commands:
#upload1.team_file # stored file
#upload1.team_file.url # url
#upload1.team_file_url(:data_file).to_s # version created
I've been pouring through all kinds of very limited leads on retrieving and/or opening the file, but everything I try seems to return errors, such as:
Errno::ENOENT: No such file or directory - https://teamfiles.s3.amazonaws.com/data_files…
Thoughts on the difference here of retrieving and USING a file from AmazonS3? Thanks!

Pulling from multiple threads, APIs, etc. I'm answering my own question with what I've found. I welcome any corrections or improvements:
To retrieve carrierwave files uploaded to AmazonS3, you have to understand that open(#upload.file_url) or File.open(#upload.file_url) does NOT open the file, it only opens the PATH to the file. (ref: Ruby OpenURI )
I use: open_uri_url = open(#upload.file_url)
You then have to find the specific file in that path that you want. For me, I then find a ZIP file that was uploaded to AmazonS3 and Extract the specific file within the ZIP file that I want with a unique *.ABC extension:
zip_content_file = Zip::File.open(open_uri_url).map{|content| content if content.to_s.split('.').last == "ABC"}.compact.first
Now, from here, where to extract to?? I create a unique directory in the Rails tmp directory to extract the file to, use it and then delete the directory:
tmp_directory = "tmp/extracts/#{#upload.parent_id}/"
FileUtils.mkdir_p(tmp_directory) unless File.directory?(tmp_directory)
extract = zip_content_file.extract(tmp_directory + content_file.to_s)
Now with found from the AmazonS3 stored ZIP file and extracted, I can open, read, etc:
f = File.new(tmp_directory + extract.to_s)
I hope this helps with Carrierwave, AmazonS3, ZIP files and using them once uploaded.

Related

In Rails I want to read excel file form Live Path like http://www.carsa.jp/admin/data.xlsx

I wants to read a excel file existing on Live URL of another website.
When I hit that URL in browser file is downloading. While in my rails app it is giving below error
No such file or directory # rb_sysopen - http://www.carsa.jp/admin/data.xlsx (Errno::ENOENT)
My Rails app code is as below
data = Roo::Excelx.new('http://www.carsa.jp/admin/data.xlsx')
header = data.row(1)
puts header
Note: If I download file and place it within my application it is working fine but the requirement is to read it from the third-party website in a scheduled job as per the above script.
data = Roo::Excelx.new('lib/data.xlsx')
header = data.row(1)
puts header
Try using Roo::Spreadsheet.open instead of Roo::Excelx.new. According to the Roo Readme:
Roo::Spreadsheet.open can accept both paths and File instances.
This should do the trick:
Roo::Spreadsheet.open('http://www.carsa.jp/admin/data.xlsx')

Rails: Rename a Binary File uploaded to API

I've seen a lot of threads that open up a file from a directory and rename it. But in my case, the user is uploading a file to a POST endpoint. I'd like to rename that file before uploading to my blob storage.
Here's what I have so far:
def picture
#file = params[:file]
#new_file_name = generate__filename()
#this line didn't work
File.rename(#file, #new_file_name + File.extname(#file))
# begin upload here
end
I'm testing this by selecting the form-data value as file in Postman. How do I rename this file?

Ruby on Rails and rubyzip: powerpoint modification corrupted on windows

I'm using ruby on rails to modify a powerpoint presentations existing template's xml files (open xml) based on data from my postgres database.
The issue I am facing is that after the file is generated and downloaded from heroku using a windows machine, microsoft powerpoint detects the file as corrupted and attempts to repair. After repairing the file generated in powerpoint, the file opens correctly.
If i download the file from a Linux machine and send the file to a windows machine, the file opens correctly without warning or attempt to repair. The powerpoint generated also opens correctly on Open office.
Technically, these are the steps that I am using to generate the file.
Open the template from the assets folder of my application
Extract the files in a tmp folder using Rubyzip gem
Open and modify the individual files using Nokogiri
Compress the file to a .pptx file
Upload/Save file using active storage
Redirect user to the file for download
Extract files method
def self.extract_files(dir_prefix)
Zip::File.open(Rails.root.join('app', "assets", "ppt", 'test_8.pptx')) do |z|
z.each do |f|
##Extract files in a directory
f_path=File.join("tmp/#{dir_prefix}_destination", f.name)
FileUtils.mkdir_p(File.dirname(f_path))
z.extract(f, f_path) unless File.exist?(f_path)
end
end
end
Opening files for manipulations using:
chartxml = File.open(Rails.root.join('tmp', tmp_extract_folder, 'ppt', 'charts', 'chart5.xml'))
##Manipulation logic here
File.write(Rails.root.join('tmp', tmp_extract_folder, 'ppt', 'charts', 'chart5.xml'), doc.to_xml)
Re-zipping files using:
zf =ZipFileGenerator.new("tmp/#{dir_prefix}_destination", "tmp/#{dir_prefix}_zipped.pptx")
The implementation of the zip file generator is the same as the one provided on Rubyzip's github repo
I'm then storing using active storage using:
ppt = Powerpoint.new(name: "#{dir_prefix}")
ppt.file.attach(
io: File.open("tmp/#{dir_prefix}_zipped.pptx"), filename: 'Synthese.pptx', content_type: 'application/vnd.openxmlformats-officedocument.presentationml.presentation'
)
ppt.save
render :js => "window.location = '#{download_ppt_path(dir_prefix: dir_prefix)}'"
The user then downloads the file:
ppt = Powerpoint.where(name: dir_prefix)
redirect_to rails_blob_path(ppt.first.file, disposition: "attachment")
I tried uncommenting the code which does the manipulation of the xml files, and just unzipping and rezipping; I still face the same issue.

FTP getbinaryfile by file extension?

I am using Net::FTP's getbinaryfile functionality to pull in a zip file with FTP. My system is not aware of the full file name, so I would simply like to search the folder for the zip file extension. Usually I would simply input the filename as *.zip. This does not seem to work.
ftp = Net::FTP.new(domain)
path = "#{Rails.root}/public/ftp/#{self.id}.zip"
ftp.getbinaryfile("*.zip", path)
I used the following code to return the zip file name in the FTP folder. Then using the same code as above, I was able to run getbinaryfile with the correct zip file name.
files = ftp.nlst("*.zip")
I use the following to get all zip files (I'm using SFTP but hopefully this will point you in the right direction)
Net::SFTP.start(domain, user, :password => 'pass') do |sftp|
sftp.dir.glob("/yourdirectory","*.zip").each do |file|
sftp.download!(file, "/local/spot")
end
end

ruby reading files from S3 with open-URI

I'm having some problems reading a file from S3. I want to be able to load the ID3 tags remotely, but using open-URI doesn't work, it gives me the following error:
ruby-1.8.7-p302 > c=TagLib2::File.new(open(URI.parse("http://recordtemple.com.s3.amazonaws.com/music/745/original/The%20Stranger.mp3?1292096514")))
TypeError: can't convert Tempfile into String
from (irb):8:in `initialize'
from (irb):8:in `new'
from (irb):8
However, if i download the same file and put it on my desktop (ie no need for open-URI), it works just fine.
c=TagLib2::File.new("/Users/momofwombie/Desktop/blah.mp3")
is there something else I should be doing to read a remote file?
UPDATE: I just found this link, which may explain a little bit, but surely there must be some way to do this...
Read header data from files on remote server
Might want to check out AWS::S3, a Ruby Library for Amazon's Simple Storage Service
Do an AWS::S3:S3Object.find for the file and then an use about to retrieve the metadata
This solution assumes you have the AWS credentials and permission to access the S3 bucket that contains the files in question.
TagLib2::File.new doesn't take a file handle, which is what you are passing to it when you use open without a read.
Add on read and you'll get the contents of the URL, but TagLib2::File doesn't know what to do with that either, so you are forced to read the contents of the URL, and save it.
I also noticed you are unnecessarily complicating your use of OpenURI. You don't have to parse the URL using URI before passing it to open. Just pass the URL string.
require 'open-uri'
fname = File.basename($0) << '.' << $$.to_s
File.open(fname, 'wb') do |fo|
fo.print open("http://recordtemple.com.s3.amazonaws.com/music/745/original/The%20Stranger.mp3?1292096514").read
end
c = TagLib2::File.new(fname)
# do more processing...
File.delete(fname)
I don't have TagLib2 installed but I ran the rest of the code and the mp3 file downloaded to my disk and is playable. The File.delete would clean up afterwards, which should put you in the state you want to be in.
This solution isn't going to work much longer. Paperclip > 3.0.0 has removed to_file. I'm using S3 & Heroku. What I ended up doing was copying the file to a temporary location and parsing it from there. Here is my code:
dest = Tempfile.new(upload.spreadsheet_file_name)
dest.binmode
upload.spreadsheet.copy_to_local_file(:default_style, dest.path)
file_loc = dest.path
...
CSV.foreach(file_loc, :headers => true, :skip_blanks => true) do |row|}
This seems to work instead of open-URI:
Mp3Info.open(mp3.to_file.path) do |mp3info|
puts mp3info.tag.artist
end
Paperclip has a to_file method that downloads the file from S3.

Resources