Multiple simultaneous remote file downloads to server ruby on rails - ruby-on-rails

How can be multiple files download to server simultaneous and check upload process ( I mean time left )
On rails application I have multiple text fields, they are for remote file urls like www.example.com/abc.pdf etc and these all files should be downloaded to a temp_uploads folder.
for this I have written code in remotefiles controller like below.
def remotefiles
params[:reference_file_url].each do |rfile|
temp_file = File.new("/public/temp_uploads", "w")
open(temp_file, 'wb') do |file|
file << open(rfile).read()
end
end
end
where rfile is remote file url.
I also have defined route remotefiles to call it in ajax
my ajax call sends form data in serialize format and this controller downloads all files 1 by 1.
with this code i have to wait untill all files are downloaded to folder that is obviously not acceptable.( I mean client asked me to reduce wait time by downloading all files simultaneous )
is there any way to get it done, all my code is custom and is not using any gem

For local file upload:
http://blueimp.github.io/jQuery-File-Upload/
For remote file download:
You can use a gem called Sidekiq to write a background job which would use http to download each file. Then update Redis with the status and poll for that status via ajax from the browser.
To download the file you can use HTTPParty
require "httparty"
File.open("myfile.txt", "wb") do |f|
f.write HTTParty.get(remote_file_url).response
end

Let me answer the parallel file download part of the question. you can use a library like Typhoeus or em-http-request for that
#Typhoeus
hydra = Typhoeus::Hydra.new
params[:reference_file_url].each do |rfile|
request = Typhoeus::Request.new(rfile, followlocation: true)
request.on_complete do |response|
#do_something_with response
end
hydra.queue(request)
end
hydra.run

Related

How make rails get data from aws s3 chunk by chunk and send it ti browser as pdf download file?

I have some confusion with how to get file from aws s3 without write it as file but maybe as a tempfile that is deleted automatically. so my friend tell me to buffer stream the data chunk by chunk and send it to browser as downloadable file.
So here it is my code for downloading the file
def download(key)
File.open('filename', 'wb') do |file|
s3.get_object(bucket: 'bucket-test', key:key) do |chunk|
send_data(chunk,:type => application/pdf, :disposition => 'inline')
end
end
end
it comes with and error about seahorse cannot be convert to string. and i dont actually understand that.
How to actually do stream the data from aws (pdf file) and send it to browser as downloadable pdf file? is my code not like what i inteded for?
thank you kindly
Just retrieve the whole file into memory and then send it out:
response = s3.get_object(bucket: 'bucket-test', key: key)
send_data(response.body.read, type: application/pdf, disposition: 'inline')
This method also has the benefit that it will retry network errors, so it's more resilient that the chunked method which disable retries on error.

How to parse a very huge XML file from a remote server rails

I have a very large XML from a remote server which I have to parse and get the data.
I have tried to open the file using the open() function but it is taking more than 15 minutes and still no response.
Then I tried Nokogiri::XML(open(URL)) where URL is the link which contains the data to parse.
Also, I have tried using Net::HTTP::Get but again with no fruitful results.
Can anyone suggest which gem and function can be used to parse the data?
As mentioned before, Nokogiri::XML::Reader is your friend here. The example in the documentation works fine if you have the file locally.
It is also possible to parse the data as soon as it comes in, fully streaming. This involves getting the data in chunks (e.g. using Net::HTTP) and connecting it to the Nokogiri::XML::Reader by means of an IO.pipe.
Example (adapted from this gist):
require 'nokogiri'
require 'net/http'
# setup request
uri = URI("http://example.com/articles.xml")
req = Net::HTTP::Get.new(uri.request_uri)
# read response in a separate thread using a pipe to communicate
rd, wr = IO.pipe
reader_thread = Thread.new do
Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') do |http|
http.request(req) do |response|
response.read_body {|chunk| wr.write(chunk) }
end
wr.close
end
end
# parse the incoming data chunk by chunk
reader = Nokogiri::XML::Reader(rd)
reader.each do |node|
next if node.node_type != Nokogiri::XML::Reader::TYPE_ELEMENT
next if node.name != "article"
# now that we have the desired fragment, put it to use
doc = Nokogiri::XML(node.outer_xml)
puts("Got #{doc.text}")
end
rd.close
# let the reader thread finish cleanly
reader_thread.join
If you are working with large XML files then you can use Nokogiri::XML::Reader class. I have successfully opened 1 GB files without any problems. For optimal performance you could download the file first and then parse it using XML::Reader class localy on your server
The usage is something like this (replace XML_FILE with your path):
Nokogiri::XML::Reader(File.open(XML_FILE)).each do |node|
if node.name == 'Node' && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT
puts node.outer_xml # you can do something like this also Nokogiri::XML(node.outer_xml).at('./Node')
end
end
Heere is the documentation: http://www.rubydoc.info/github/sparklemotion/nokogiri/master/Nokogiri/XML/Reader
Hope it helps

Receive file from server in Chrome extension

I would like to connect my Chrome Extension to be able to download a file from my Ruby on Rails server. In particular, when the user clicks a button on the extension, the extension should send an AJAX request to the server, and then the server should send a file file.zip back to the user.
How would the server send a file back to the extension of the user?
On the Rails side, make sure the returned Content-Type header is application/zip. You can use #send_file or #send_data to send zip file to client. See ActionController::DataStreaming
Use #send_file
send_file '/path/to.zip', :type => 'application/zip'
Or use #send_data
zip_data = generate_zip_data()
send_data zip_data, :type => 'application/zip', :filename => 'data.zip'
For generating dynamic zip file, see other topics:
How can I generate zip file without saving to the disk with Ruby?
On google chrome extension, you might want to create new tab with the zip file url. That causes browser downloads file.
function buttonClickHandler() {
chrome.tabs.create({ url: "the url to zip file"}, function() {});
}

Ruby on Rails - OAuth 2 multipart Post (Uploading to Facebook or Soundcloud)

I am working on a Rails App that Uses OmniAuth to gather Oauth/OAuth2 credentials for my users and then posts out to those services on their behalf.
Creating simple posts to update status feeds work great.. Now I am to the point of needing to upload files. Facebook says "To publish a photo, issue a POST request with the photo file attachment as multipart/form-data." http://developers.facebook.com/docs/reference/api/photo/
So that is what I am trying to do:
I have implemented the module here: Ruby: How to post a file via HTTP as multipart/form-data? to get the headers and data...
if appearance.post.post_attachment_content_type.to_s.include?('image')
fbpost = "https://graph.facebook.com/me/photos"
data, headers = Multipart::Post.prepare_query("title" => appearance.post.post_attachment_file_name , "document" => File.read(appearance.post.post_attachment.path))
paramsarray = {:source=>data, :message=> appearance.post.content}
response = access_token.request(:post, fbpost, paramsarray, headers)
appearance.result = response
appearance.save
end
I but I am getting a OAuth2::HTTPError - HTTP 400 Error
Any assistance would be Incredible... As I see this information will also be needed for uploading files to SoundCloud also.
Thanks,
Mark
Struggled with this myself. The oauth2 library is backed by Faraday for it's HTTP interaction. with a little configuration it supports uploaded files out of the box. First step is to add the appropriate Faraday middleware when building your connection. An example from my code:
OAuth2::Client.new client_id, secret, site: site do |stack|
stack.request :multipart
stack.request :url_encoded
stack.adapter Faraday.default_adapter
end
This adds the multipart encoding support to the Faraday connection. Next when making the request on your access token object you want to use a Faraday::UploadIO object. So:
upload = Faraday::UploadIO.new io, mime_type, filename
access_token.post('some/url', params: {url: 'params'}, body: {file: upload})
In the above code:
io - An IO object for the file you want to upload. Can be a File object or even a StringIO.
mime_type - The mime type of the file you are uploading. You can either try to detect this server-side or if a user uploaded the file to you, you should be able to extract the mime type from their request.
filename - What are are calling the file you are uploading. This can also be determined by your own choosing or you can just use whatever the user uploading the file calls it.
some/url - Replace this with the URL you want to post to
{url: 'params'} - Replace this with any URL params you want to provide
{file: upload} - Replace this with your multipart form data. Obviously one (or more) of the key/value pairs should have an instance of your file upload.
I'm actually using successfully this code to upload a photo on a fb page :
dir = Dir.pwd.concat("/public/system/posts/images")
fb_url = URI.parse("https://graph.facebook.com/#{#page_id}/photos")
img = File.open("myfile.jpg")
req = Net::HTTP::Post::Multipart.new(
"#{fb_url.path}?access_token=#{#token}",
"source" => UploadIO.new(img, "application/jpg", img.path),
"message" => "some messsage"
)
n = Net::HTTP.new(fb_url.host, fb_url.port)
n.use_ssl = true
n.verify_mode = OpenSSL::SSL::VERIFY_NONE
n.start do |http|
#result = http.request(req)
end

Ruby-Rails serve ftp file direct to client

I am new to ruby and to rails, so excuse my question... .
What i want to know is, how to take a file from a ftp server with ruby without saving the file on my rails application harddrive (streaming the filedata direct to the client). I am working with the ruby Net/FTP class.
With the method "retrbinary" from the Net/FTP class i have the following snippet:
ftp.retrbinary('RETR ' + filename, 4096) { |data|
buf << data
}
In my rails view i can do something like this:
send_data( buf )
So how do i combine these two. I dont know how to instanziate a buffer object, fill in the stream and than serve it to the user. Has anybody an idea how to do this?
thank you very much for your support! Your post get me going on. After some cups of coffee i found a working solution. Actually i am doing the following, which works for me:
def download_file
filename = params[:file]
raw = StringIO.new('')
#ftp.retrbinary('RETR ' + filename, 4096) { |data|
raw << data
}
#ftp.close
raw.rewind
send_data raw.read, :filename => filename
end
I will test this in production(real life situation). If this is not working well enough, i have to use a NFS mount.
fin
Do you want the following?
1) Client (browser) sends a request to the Rails server
2) Server should respond with the contents of a file that is located on an ftp server.
Is that it?
If so, then simply redirect the browser to the ftp location. Eg
# in controller
ftp_url = "ftp://someserver.com/dir_name/file_name.txt
redirect_to ftp_url
The above works if the ftp file has anonymous get access.
If you really need to access the file from the server and stream it, try the following:
# in controller
render :text => proc {|response, output|
ftp_session = FTP.open(host, user, passwd, acct)
ftp_session.gettextfile(remotefile) {|data| output.write(data)}
ftp_session.close
}
You should check the headers in the response to see if they're what you want.
ps. Setting up the ftp connection and streaming from a second server will probably be relatively slow. I'd use JS to show a busy graphic to the user.
I'd try alternatives to ftp. Can you set up an NFS connection or mount the remote disk? Would be much faster than ftp. Also investigate large TCP window sizes.

Resources