i'm having a trouble with lua.
I need send a request to a website by GET and get the response from website.
Atm all i have is this:
local LuaSocket = require("socket")
client = LuaSocket.connect("example.com", 80)
client:send("GET /login.php?login=admin&pass=admin HTTP/1.0\r\n\r\n")
while true do
s, status, partial = client:receive('*a')
print(s or partial)
if status == "closed" then
break
end
end
client:close()
What should i do to get the response from server ?
I want to send some info to this website and get the result of page.
Any ideas ?
This probably won't work as *a will be reading until the connection is closed, but in this case the client doesn't know how much to read. What you need to do is to read line-by-line and parse the headers to find Content-Length and then after you see two line ends you read the specified number of bytes (as set in Content-Length).
Instead of doing it all yourself (reading and parsing headers, handling redirects, 100 continue, and all that), socket.http will take care of all that complexity for you. Try something like this:
local http = require("socket.http")
local body, code, headers, status = http.request("https://www.google.com")
print(code, status, #body)
I solved the problem passing the header
local LuaSocket = require("socket")
client = LuaSocket.connect("example.com", 80)
client:send("GET /login.php?login=admin&pass=admin HTTP/1.0\r\nHost: example.com\r\n\r\n")
while true do
s, status, partial = client:receive('*a')
print(s or partial)
if status == "closed" then
break
end
end
client:close()
Related
I have this Telegram bot written in Lua that I am doing as a hobby for a language network. And I have been reading new messages via the getUpdates API call all the time. Now I want to rewrite it to use webhooks, but I have no experience with that whatsoever. I have googled but didn't find anything certain. I kinda feel that WSAPI is the library to use, but I am not sure. Moreover, I am not really sure I need any special library just for reading POST requests (which is all that the Telegram bot API uses). I tried using sockets:
socket = require 'socket'
server = assert(socket.bind("*", 9000))
function read(client, pattern, prefix)
local data, emsg, partial = client:receive(pattern, prefix)
if data then
return data
end
if partial and #partial > 0 then
return partial
end
return nil, emsg
end
while true do
local client = server:accept()
client:settimeout(3)
local msg, err = read(client, '*a')
if not err then
print(msg)
client:close()
end
end
The print(msg) here gives me the full POST request including headers, which I am probably able to parse (the body is supposed to always be a JSON). I am not really that familiar with HTTP requests though and I'm not sure I can just throw away everything that goes before the first {.
My setup is Lua 5.2, Ubuntu x64 16.04 and Nginx. What I need to do is to receive and read POST requests, nothing more.
TL;DR: is it okay to parse the POST request I receive from the code above or am I missing something, like a library that'd make my life easier?
Thanks!
I have a very large XML from a remote server which I have to parse and get the data.
I have tried to open the file using the open() function but it is taking more than 15 minutes and still no response.
Then I tried Nokogiri::XML(open(URL)) where URL is the link which contains the data to parse.
Also, I have tried using Net::HTTP::Get but again with no fruitful results.
Can anyone suggest which gem and function can be used to parse the data?
As mentioned before, Nokogiri::XML::Reader is your friend here. The example in the documentation works fine if you have the file locally.
It is also possible to parse the data as soon as it comes in, fully streaming. This involves getting the data in chunks (e.g. using Net::HTTP) and connecting it to the Nokogiri::XML::Reader by means of an IO.pipe.
Example (adapted from this gist):
require 'nokogiri'
require 'net/http'
# setup request
uri = URI("http://example.com/articles.xml")
req = Net::HTTP::Get.new(uri.request_uri)
# read response in a separate thread using a pipe to communicate
rd, wr = IO.pipe
reader_thread = Thread.new do
Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') do |http|
http.request(req) do |response|
response.read_body {|chunk| wr.write(chunk) }
end
wr.close
end
end
# parse the incoming data chunk by chunk
reader = Nokogiri::XML::Reader(rd)
reader.each do |node|
next if node.node_type != Nokogiri::XML::Reader::TYPE_ELEMENT
next if node.name != "article"
# now that we have the desired fragment, put it to use
doc = Nokogiri::XML(node.outer_xml)
puts("Got #{doc.text}")
end
rd.close
# let the reader thread finish cleanly
reader_thread.join
If you are working with large XML files then you can use Nokogiri::XML::Reader class. I have successfully opened 1 GB files without any problems. For optimal performance you could download the file first and then parse it using XML::Reader class localy on your server
The usage is something like this (replace XML_FILE with your path):
Nokogiri::XML::Reader(File.open(XML_FILE)).each do |node|
if node.name == 'Node' && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT
puts node.outer_xml # you can do something like this also Nokogiri::XML(node.outer_xml).at('./Node')
end
end
Heere is the documentation: http://www.rubydoc.info/github/sparklemotion/nokogiri/master/Nokogiri/XML/Reader
Hope it helps
I am using this example. It prints the text based on this site http://www.google.com/robots.txt
local socket = require("socket")
client = socket.connect("google.com", 80)
client:send("GET /robots.txt HTTP/1.0\r\n\r\n")
while true do
s, status, partial = client:receive(1024)
print(s or partial)
if status == "closed" then
break
end
end
client:close()
I use:
local socket = require("socket")
client = socket.connect("www.lua.org", 80)
client:send("GET /pil/9.4.html HTTP/1.0\r\n\r\n")
while true do
s, status, partial = client:receive(1024)
print(s or partial)
if status == "closed" then
break
end
end
client:close()
But, as above, I try with this link, http://www.lua.org/pil/9.4.html and it doesn't work, saying "HTTP/1.0 302 Moved temporarily". Did same on many other sites, got similar results. Why is that so? Thanks a lot
You're receiving a redirect (HTTP 302 code) which should be handled by issuing a new request to the specified Location.
Rather than using raw sockets, you could use the high level HTTP module from the socket library, which provides a request method able to authomatically follow redirects.
I use lua 5.1 and the luaSocket 2.0.2-4 to retrieve a page from a web server. I first check if the server is responding and then assign the web server response to lua variables.
local mysocket = require("socket.http")
if mysocket.request(URL) == nil then
print('The server is unreachable on:\n'..URL)
return
end
local response, httpCode, header = mysocket.request(URL)
Everything works as expected but the request is executed two times. I wonder if I could do Something like (which doesn't work obviously):
local mysocket = require("socket.http")
if (local response, httpCode, header = mysocket.request(URL)) == nil then
print('The server is unreachable on:\n'..URL)
return
end
Yes, something like this :
local mysocket = require("socket.http")
local response, httpCode, header = mysocket.request(URL)
if response == nil then
print('The server is unreachable on:\n'..URL)
return
end
-- here you do your stuff that's supposed to happen when request worked
Request will be sent only once, and function will exit if it failed.
Even better, when request fails, the second return is the reason:
In case of failure, the function returns nil followed by an error message.
(From the documentation for http.request)
So you can print the problem straight from the socket's mouth:
local http = require("socket.http")
local response, httpCode, header = http.request(URL)
if response == nil then
-- the httpCode variable contains the error message instead
print(httpCode)
return
end
-- here you do your stuff that's supposed to happen when request worked
I'm creating an API service which allows people to provide a URL of an image to the API call, and the the service downloads the image to process.
How do I ensure somebody does NOT give me the URL of, like, a 5MB image? Is there a way to limit the request?
This is what I have so far, which basically grabs everything.
req = Net::HTTP::Get.new(url.path)
res = Net::HTTP.start(url.host, url.port) { |http|
http.request(req)
}
Thanks,
Conrad
cwninja unfortunately gave you an answer that will only work for accidental attacks. An intelligent attacker will have no trouble at all defeating that check. There are two main reasons his method should not be used. First, nothing guarantees that the information in a HEAD response will match the corresponding GET response. A properly behaving server certainly will do this, but a malicious actor does not have to follow the spec. The attacker could simply send a HEAD response that says it has a Content-Length that's less than your threshold, but then hand you a huge file in the GET response. But that doesn't even cover the potential for a server to send back a response with the Transfer-Encoding: chunked header set. A chunked response could quite possibly never end. A few people pointing your server at never-ending responses could carry out a trivial resource-exhaustion attack, even if your HTTP client enforces a timeout.
To do this correctly, you need to use an HTTP library that allows you to count the bytes as they're received, and abort if it crosses the threshold. I would probably recommend Curb for this rather than Net::HTTP. (Can you even do this at all with Net::HTTP?) If you use the on_body and/or on_progress callbacks, you can count the incoming bytes and abort mid-response if you receive a file that's too large. Obviously, as cwninja already pointed out, if you receive a Content-Length header larger than your threshold, you want to abort for that too. Curb is also notably faster than Net::HTTP.
Try running this first:
Net::HTTP.start(url.host, url.port) { |http|
response = http.request_head(url.path)
raise "File too big." if response['content-length'].to_i > 5*1024*1024
}
You still have a race condition (someone could swap out the file after you do the HEAD request), but in the simple case this asks the server for the headers it would send back from a GET request.
Another one way to limit downloading size (full code should check response status, exception handling etc. It's just an example)
Net::HTTP.start(uri.host, uri.port) do |http|
request = Net::HTTP::Get.new uri.request_uri
http.request request do |response|
# check response codes here
body=''
response.read_body do |chunk|
body += chunk
break if body.size > MY_SAFE_SIZE_LIMIT
end
break
end
end
Combining the other two answers, I'd like to 1) check the size header, 2) watch the size of chunks, while also 3) supporting https and 4) aggressively enforcing a timeout. Here's a helper I came up with:
require "net/http"
require 'uri'
module FetchUtil
# Fetch a URL, with a given max bytes, and a given timeout
def self.fetch_url url, timeout_sec=5, max_bytes=5*1024*1024
uri = URI.parse(url)
t0 = Time.now.to_f
body = ''
Net::HTTP.start(uri.host, uri.port,
:use_ssl => (uri.scheme == 'https'),
:open_timeout => timeout_sec,
:read_timeout => timeout_sec) { |http|
# First make a HEAD request and check the content-length
check_res = http.request_head(uri.path)
raise "File too big" if check_res['content-length'].to_i > max_bytes
# Then fetch in chunks and bail on either timeout or max_bytes
# (Note: timeout won't work unless bytes are streaming in...)
http.request_get(uri.path) do |res|
res.read_body do |chunk|
raise "Timeout error" if (Time.now().to_f-t0 > timeout_sec)
raise "Filesize exceeded" if (body.length+chunk.length > max_bytes)
body += chunk
end
end
}
return body
end
end