ruby http response script sending large gzipped content - ruby-on-rails

Question

I'm trying to setup a script written in ruby to open a port 2004 on a server. Called the server with http and the port http://<server>:2004/ will then result back a HTTP header + response.
The response is read from a file.
This is working for small content, but not for like 50MB.
Somehow it simply breaks.
By the way I'm testing this script with SoapUI.
Here is the source code, I think this one is pretty self-explanatory.
For better reading I marked the response part big.
#!/bin/ruby
require 'socket'
require 'timeout'
require 'date'
server = TCPServer.open 2004
puts "Listening on port 2004"
#file="dump.request"
loop {
Thread.start(server.accept) do |client|
date = Time.now.strftime("%d-%m-%Y_%H-%M-%S")
file = "#{date}_mt_dump.txt"
puts date
puts "Accepting connection"
#client = server.accept
#resp = "OKY|So long and thanks for all the fish!|OKY"
ticket_id = "1235"
partial_data = ""
i = 1024
firstrun = "yes"
fd = File.open(file,'w')
puts "Attempting receive loop"
puts "Ready to transfer contents to the client"
f = File.open("output.txt.gz","r")
puts "Opened file output.txt.gz; size: #{f.size}"
resp = f.read(f.size)
headers = ["HTTP/1.1 200 OK",
"Content-Encoding: gzip",
"Content-Type: text/xml;charset=UTF-8",
"Content-Length: #{f.size}\r\n\r\n"].join("\r\n")
client.puts headers
#puts all_data.join()
fd.close unless fd == nil
puts "Start data transfer"
client.puts resp
client.close
puts "Closed connection"
puts "\n"
end
}

Answer 1

There are a number of issues I see with your code, some which are conceptual and some of which are technical, but without more information about the error you receive might be impossible to offer a correct response.
It is my initial thought that the issue is caused by the fact that you are opening Gzipped files without using the binary mode flag, so that the file reading stops ate the first EOF character and new line markers might be converted.
A few technical things to consider:
Your loop is infinite. You should really set up signal traps to allow you to exit the script (catching ^C, for example).
Zip files are usually binary files. You should use a binary mode to open the file, or use the IO.binread method if your loading the whole file up to memory.
Your loading the whole file unto the memory before sending it. That's great for small files, but it isn't the best approach for larger files. Loading 50MB up to the RAM for each client, while serving a 100 clients, means 5GB of RAM...
Considering the first two technical points, I would tweek the code a bit like so:
keep_running = true
trap('INT'){ keep_running = false ; raise ::SystemExit}
begin
while(run) {
Thread.start(server.accept) do |client|
date = Time.now.strftime("%d-%m-%Y_%H-%M-%S")
file = "#{date}_mt_dump.txt"
puts date
puts "Accepting connection"
#client = server.accept
#resp = "OKY|So long and thanks for all the fish!|OKY"
ticket_id = "1235"
partial_data = ""
i = 1024
firstrun = "yes"
fd = File.open(file,'bw')
puts "Attempting receive loop"
puts "Ready to transfer contents to the client"
f = File.open("output.txt.gz","br")
puts "Opened file output.txt.gz; size: #{f.size}"
resp = f.read(f.size)
headers = ["HTTP/1.1 200 OK",
"Content-Encoding: gzip",
"Content-Type: text/xml;charset=UTF-8",
"Content-Length: #{f.size}\r\n\r\n"].join("\r\n")
client.puts headers
#puts all_data.join()
fd.close unless fd == nil
puts "Start data transfer"
client.puts resp
client.close
puts "Closed connection"
puts "\n"
end
}
rescue => e
puts e.message
puts e.backtrace
rescue SystemExit => e
puts "exiting... please notice that existing threads will be brutally stoped, as we will not wait for them..."
end
As to my more general pointers:
Your code is opening a new thread per connection. While this is okay for a small load of concurrent connections, your script might grind to a halt if you have a lot of concurrent connections. The context-switching alone (moving between threads) could potentially create a DoS situation.
I recommend that you use a Reactor pattern, where you have a pool of threads. Another option is to fork a few processes listening to the same TCPSocket.
You don't read the data from the socket and you don't parse the HTTP request - this means that someone could potentially fill up the system buffer, which you never empty, by continuously sendings data.
It would be better if you read the information from the socket, or emptied it's buffer, as well as disconnected from any malformed of malicious connections.
Also, most browsers aren't too happy when the response comes in before the request...
You don't catch any exceptions nor print any error messages. This means that your script might throw an exception that will break everything apart. For instance, if your 'server' will reach the 'open file limit' for it's process, the accept method will throw an exception which will shut down the whole script, including existing connections.
I'm not sure why you aren't using one of the many HTTP servers available for Ruby - be it the builtin WEBrick (don't use for production) or one of the native Ruby community gems, such as Iodine.
Here's a short example using Iodine, which has an easy to utilize Http server written in Ruby (no need to compile anything):
require 'iodine/http'
# cache the file, since it's the only response ever sent
file_data = IO.binread "output.txt.gz"
Iodine.on_http do |request, response|
begin
# set any headers
response['content-type'] = 'text/xml;charset=UTF-8'
response['content-encoding'] = 'gzip'
response << file_data
true
rescue => e
Iodine.error e
false
end
end
end
#if in irb:
exit
Or, if you insist on writing your own HTTP server, you can at least use a one of the available IO reactors, such as Iodine (I it wrote for Plezi), to help you handle the thread pool and IO management (you can also use EventMachine, but I don't it like so much - than again, I'm biased, as I wrote the Iodine Library):
require 'iodine'
require 'stringio'
class MiniServer < Iodine::Protocol
# cache the file, since it's the only data sent,
# and make it available to all the connections.
def self.data
#data ||= IO.binread 'output.txt.gz'
end
# The on_opne callback is called when a connection is established.
# We'll use it to initialize the HTTP request's headers Hash.
def on_open
#headers = {}
end
# the on_message callback is called when data is sent from the client to the socket.
def on_message input
input = StringIO.new input
l = nil
headers = #headers # easy access
# loop the lines and parse the HTTP request.
while (l = input.gets)
unless l.match /^[\r]?\n/
if l.include? ':'
l = l.strip.downcase.split(':', 2)
headers[l[0]] = l[1]
else
headers[:method], headers[:query], headers[:version] = l.strip.split(/[\s]+/, 3)
headers[:request_start] = Time.now
end
next
end
# keep the connection alive if the HTTP version is 1.1 or if the connection is requested to be kept alive
keep_alive = (headers['connection'].to_s.match(/keep/i) || headers[:version].match(/1\.1/)) && true
# refuse any file uploads or forms. make sure the request is a GET request
return close if headers['content-length'] || headers['content-type'] || headers[:method].to_s.match(/get/i).nil?
# all is well, send the file.
write ["HTTP/1.1 200 OK",
"Connection: #{keep_alive ? 'keep-alive' : 'close'}",
"Content-Encoding: gzip",
"Content-Type: text/xml;charset=UTF-8",
"Content-Length: #{self.class.data.bytesize}\r\n\r\n"].join("\r\n")
write self.class.data
return close unless keep_alive
# reset the headers, in case another request comes in
headers.clear
end
end
end
Iodine.protocol = MiniServer
# # if running within a larget application, consider:
# Iodine.force_start!
# # Server starts automatically when the script ends.
# # on irb, use `exit`:
exit
Good Luck!