I am new to ruby and to rails, so excuse my question... .
What i want to know is, how to take a file from a ftp server with ruby without saving the file on my rails application harddrive (streaming the filedata direct to the client). I am working with the ruby Net/FTP class.
With the method "retrbinary" from the Net/FTP class i have the following snippet:
ftp.retrbinary('RETR ' + filename, 4096) { |data|
buf << data
}
In my rails view i can do something like this:
send_data( buf )
So how do i combine these two. I dont know how to instanziate a buffer object, fill in the stream and than serve it to the user. Has anybody an idea how to do this?
thank you very much for your support! Your post get me going on. After some cups of coffee i found a working solution. Actually i am doing the following, which works for me:
def download_file
filename = params[:file]
raw = StringIO.new('')
#ftp.retrbinary('RETR ' + filename, 4096) { |data|
raw << data
}
#ftp.close
raw.rewind
send_data raw.read, :filename => filename
end
I will test this in production(real life situation). If this is not working well enough, i have to use a NFS mount.
fin
Do you want the following?
1) Client (browser) sends a request to the Rails server
2) Server should respond with the contents of a file that is located on an ftp server.
Is that it?
If so, then simply redirect the browser to the ftp location. Eg
# in controller
ftp_url = "ftp://someserver.com/dir_name/file_name.txt
redirect_to ftp_url
The above works if the ftp file has anonymous get access.
If you really need to access the file from the server and stream it, try the following:
# in controller
render :text => proc {|response, output|
ftp_session = FTP.open(host, user, passwd, acct)
ftp_session.gettextfile(remotefile) {|data| output.write(data)}
ftp_session.close
}
You should check the headers in the response to see if they're what you want.
ps. Setting up the ftp connection and streaming from a second server will probably be relatively slow. I'd use JS to show a busy graphic to the user.
I'd try alternatives to ftp. Can you set up an NFS connection or mount the remote disk? Would be much faster than ftp. Also investigate large TCP window sizes.
Related
I have some confusion with how to get file from aws s3 without write it as file but maybe as a tempfile that is deleted automatically. so my friend tell me to buffer stream the data chunk by chunk and send it to browser as downloadable file.
So here it is my code for downloading the file
def download(key)
File.open('filename', 'wb') do |file|
s3.get_object(bucket: 'bucket-test', key:key) do |chunk|
send_data(chunk,:type => application/pdf, :disposition => 'inline')
end
end
end
it comes with and error about seahorse cannot be convert to string. and i dont actually understand that.
How to actually do stream the data from aws (pdf file) and send it to browser as downloadable pdf file? is my code not like what i inteded for?
thank you kindly
Just retrieve the whole file into memory and then send it out:
response = s3.get_object(bucket: 'bucket-test', key: key)
send_data(response.body.read, type: application/pdf, disposition: 'inline')
This method also has the benefit that it will retry network errors, so it's more resilient that the chunked method which disable retries on error.
I have this Telegram bot written in Lua that I am doing as a hobby for a language network. And I have been reading new messages via the getUpdates API call all the time. Now I want to rewrite it to use webhooks, but I have no experience with that whatsoever. I have googled but didn't find anything certain. I kinda feel that WSAPI is the library to use, but I am not sure. Moreover, I am not really sure I need any special library just for reading POST requests (which is all that the Telegram bot API uses). I tried using sockets:
socket = require 'socket'
server = assert(socket.bind("*", 9000))
function read(client, pattern, prefix)
local data, emsg, partial = client:receive(pattern, prefix)
if data then
return data
end
if partial and #partial > 0 then
return partial
end
return nil, emsg
end
while true do
local client = server:accept()
client:settimeout(3)
local msg, err = read(client, '*a')
if not err then
print(msg)
client:close()
end
end
The print(msg) here gives me the full POST request including headers, which I am probably able to parse (the body is supposed to always be a JSON). I am not really that familiar with HTTP requests though and I'm not sure I can just throw away everything that goes before the first {.
My setup is Lua 5.2, Ubuntu x64 16.04 and Nginx. What I need to do is to receive and read POST requests, nothing more.
TL;DR: is it okay to parse the POST request I receive from the code above or am I missing something, like a library that'd make my life easier?
Thanks!
I have a very large XML from a remote server which I have to parse and get the data.
I have tried to open the file using the open() function but it is taking more than 15 minutes and still no response.
Then I tried Nokogiri::XML(open(URL)) where URL is the link which contains the data to parse.
Also, I have tried using Net::HTTP::Get but again with no fruitful results.
Can anyone suggest which gem and function can be used to parse the data?
As mentioned before, Nokogiri::XML::Reader is your friend here. The example in the documentation works fine if you have the file locally.
It is also possible to parse the data as soon as it comes in, fully streaming. This involves getting the data in chunks (e.g. using Net::HTTP) and connecting it to the Nokogiri::XML::Reader by means of an IO.pipe.
Example (adapted from this gist):
require 'nokogiri'
require 'net/http'
# setup request
uri = URI("http://example.com/articles.xml")
req = Net::HTTP::Get.new(uri.request_uri)
# read response in a separate thread using a pipe to communicate
rd, wr = IO.pipe
reader_thread = Thread.new do
Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') do |http|
http.request(req) do |response|
response.read_body {|chunk| wr.write(chunk) }
end
wr.close
end
end
# parse the incoming data chunk by chunk
reader = Nokogiri::XML::Reader(rd)
reader.each do |node|
next if node.node_type != Nokogiri::XML::Reader::TYPE_ELEMENT
next if node.name != "article"
# now that we have the desired fragment, put it to use
doc = Nokogiri::XML(node.outer_xml)
puts("Got #{doc.text}")
end
rd.close
# let the reader thread finish cleanly
reader_thread.join
If you are working with large XML files then you can use Nokogiri::XML::Reader class. I have successfully opened 1 GB files without any problems. For optimal performance you could download the file first and then parse it using XML::Reader class localy on your server
The usage is something like this (replace XML_FILE with your path):
Nokogiri::XML::Reader(File.open(XML_FILE)).each do |node|
if node.name == 'Node' && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT
puts node.outer_xml # you can do something like this also Nokogiri::XML(node.outer_xml).at('./Node')
end
end
Heere is the documentation: http://www.rubydoc.info/github/sparklemotion/nokogiri/master/Nokogiri/XML/Reader
Hope it helps
How can be multiple files download to server simultaneous and check upload process ( I mean time left )
On rails application I have multiple text fields, they are for remote file urls like www.example.com/abc.pdf etc and these all files should be downloaded to a temp_uploads folder.
for this I have written code in remotefiles controller like below.
def remotefiles
params[:reference_file_url].each do |rfile|
temp_file = File.new("/public/temp_uploads", "w")
open(temp_file, 'wb') do |file|
file << open(rfile).read()
end
end
end
where rfile is remote file url.
I also have defined route remotefiles to call it in ajax
my ajax call sends form data in serialize format and this controller downloads all files 1 by 1.
with this code i have to wait untill all files are downloaded to folder that is obviously not acceptable.( I mean client asked me to reduce wait time by downloading all files simultaneous )
is there any way to get it done, all my code is custom and is not using any gem
For local file upload:
http://blueimp.github.io/jQuery-File-Upload/
For remote file download:
You can use a gem called Sidekiq to write a background job which would use http to download each file. Then update Redis with the status and poll for that status via ajax from the browser.
To download the file you can use HTTPParty
require "httparty"
File.open("myfile.txt", "wb") do |f|
f.write HTTParty.get(remote_file_url).response
end
Let me answer the parallel file download part of the question. you can use a library like Typhoeus or em-http-request for that
#Typhoeus
hydra = Typhoeus::Hydra.new
params[:reference_file_url].each do |rfile|
request = Typhoeus::Request.new(rfile, followlocation: true)
request.on_complete do |response|
#do_something_with response
end
hydra.queue(request)
end
hydra.run
I'm trying to upload to my server (on Heroku) a file stored in a password protected FTP.
The problem is that this FTP also dont contain my production IP address on his whitelist (and i cant add it..) so i should use a proxy to connect my rails app this FTP.
I tried this code :
proxy_uri = URI(ENV['QUOTAGUARDSTATIC_URL'] || 'http://login:password#myproxy.com:9293')
Net::HTTP::Proxy(proxy_uri.host, proxy_uri.port,"login","password").start('ftp://login:password#ftp.website.com') do |http|
http.get('/path/to/myfile.gz').body
end
But my http.get returns me lookup ftp: no such host.
I also got this code for FTP download, but i dont know how to make it works with a proxy :
ftp = Net::FTP.new('ftp.myftp.com', 'login', 'password')
ftp.chdir('path/to')
ftp.getbinaryfile('myfile.gz', 'public/myfile.gz', 1024)
ftp.close
Thanks in advance.
I realise that you asked this question over 6 months ago, but I recently had a similar issue and found that this (unanswered) question is the top Google result, so I thought I would share my findings.
mudasobwa's comment below your original post has a link to the net/ftp documentation which explains how to use a SOCKS proxy...
Although you don't mention a specific requirement for a HTTP proxy in your original post, it seems obvious to me that is what you were trying to use. As I'm sure you're aware, this makes the SOCKS documentation totally irrelevant.
The following code has been tested on ruby-1.8.7-p357 using an HTTP proxy that does not require authentication:
file = File.open('myfile.gz', 'w')
http = Net::HTTP.start('myproxy.com', '9293')
resp, data = http.get('ftp://login:password#ftp.website.com')
file.write(data) if resp.code == "200"
file.close unless file.nil?
Source
This should give you a good starting point to figure the rest out for yourself.
To get you going, I would guess that you could use user:pass#myproxy.com for basic auth, or perhaps sending a Proxy-Authorization header in your GET request.