I'm trying to send a POST request but noticed the endpoint I'm trying to access doesn't seem to like the request, so to investigate the situation I redirected the request to localhost while listening with nc and saw the following request:
nc -vlp 444
Connection from 127.0.0.1:53812
POST / HTTP/1.1
Host: localhost:4444
TE: trailers
Cookie:
Content-Type: application/x-www-form-urlencoded
Connection: close, TE
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0
27 -- this line shouldn't be there
username=username&password=password
0 -- also this one
The code I'm using, just in case:
local http = require("socket.http") -- even tried ssl.https
...
function Session:post(url, payload) -- payload = "username=username&password=password"
local response = Response
local body = { }
local r, c, h, s = http.request{
url = url,
method = "POST",
sink = ltn12.sink.table(body),
source = ltn12.source.string(payload),
headers = {
["User-Agent"] = "Mozilla/5.0 (X11; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0",
["Content-Type"] = "application/x-www-form"
},
}
-- you can ignore this
response.code = c
response.status = s
response.body = table.concat(body)
self.cookies:parse(h["set-cookie"])
return response
end
Thinking it was a problem of ltn12 I copied the source code of it directly in my project but soon found out it only returned the payload without actually changing anything
BLOCKSIZE = 2048
function generate_payload(s)
if s then
local i = 1
return function()
local chunk = string.sub(s, i, i+BLOCKSIZE-1)
i = i + BLOCKSIZE
if chunk ~= "" then return chunk
else return nil end
end
else return source.empty() end
end
Managed to fix the issue by adding the Content-Length header.
Related
I wrote some lua script for wrk to generate multiple POST requests.
My problem is that my script is working only for the first request. All further generated requests are exactly the same as the first one. I would like that user variable will be new generated for each POST request:
lowerCase = "abcdefghijklmnopqrstuvwxyz"
characterSet = lowerCase
keyLength = 13
user = ""
math.randomseed(os.time())
for i = 1, keyLength do
rand = math.random(#characterSet)
user = user .. string.sub(characterSet, rand, rand )
end
wrk.path = "/somepath"
wrk.method = "POST"
wrk.body = [[{"username":"]].. user .. [[,"password":"somepassword"}]]
wrk.headers["Content-Type"] = "application/json"
Try something like this. It should execute request3 ~50% of the time, and the other two ~25% of the time. Cheers!
names = { "Maverick", "Goose", "Viper", "Iceman", "Merlin", "Sundown", "Cougar", "Hollywood", "Wolfman", "Jester" }
request1 = function()
headers = {}
headers["Content-Type"] = "application/json"
body = '{"name": ' .. names[math.random(#names)] .. '}'
return wrk.format("POST", "/test1", headers, body)
end
request2 = function()
headers = {}
headers["Content-Type"] = "application/json"
body = '{"name": ' .. names[math.random(#names)] .. '}'
return wrk.format("POST", "/test2", headers, body)
end
request3 = function()
headers = {}
headers["Content-Type"] = "application/json"
body = '{"name": ' .. names[math.random(#names)] .. '}'
return wrk.format("GET", "/test3", headers, body)
end
requests = {}
requests[0] = request1
requests[1] = request2
requests[2] = request3
requests[3] = request3
request = function()
return requests[math.random(0, 3)]()
end
response = function(status, headers, body)
if status ~= 200 then
io.write("------------------------------\n")
io.write("Response with status: ".. status .."\n")
io.write("------------------------------\n")
io.write("[response] Body:\n")
io.write(body .. "\n")
end
end
I'm not familiar with wrk.
I guess you're running that code multiple times within a second. As os.time has second accuracy you'll have the same randomseed and hence the same user name within that second.
From looking into the scripting examples I'd say the script is only evaluated once or maybe once per thread. Those examples implement functions that will be called by wrk. It wouldn't make sense to define those functions for every request.
Add a print to your script to make sure.
Here is an example that counts requests. You probably can put your code into that function
function request()
requests = requests + 1
return wrk.request()
end
This question already has an answer here:
How to get id in responseHeaders location?
(1 answer)
Closed last year.
I am not able to fetch location attribute from response header in karate, while other headers are getting displayed when using below code also from postman i am able to see all the headers including location
Background:
* url APAuthorizationURL
* def sessionTokenPayload = call read('DIGI-3049.feature')
* def authToken = sessionTokenPayload.session_Token
* configure cors = true
* print authToken
#Authorization
Scenario: Authorization_100200
Given header Content-Type = 'application/json'
* print authToken
* def query = {client_id: '0oajtiuveymO1ES7G0h7',response_type : 'token',response_mode: 'okta_post_message', redirect_uri: 'https%3A%2F%2Fqa2-sc.adviser.ad.mlclife.com.au', scope:'advisor',prompt:'none',state:'203475b6-836d-4dcb-8a12-7b2b2b3a2cf9', sessionToken : 'authToken', nonce:'158261204839000', scope : 'advisor' }
And params query
When method get
Then status 200
* print responseHeaders / location header is not printing
* print 'headers:', karate.prevRequest.headers / location header is not printing
This works fine for me. Please paste this into a new Scenario and try:
* url 'http://httpbin.org'
* path 'redirect-to'
* param url = 'http://httpbin.org/get'
* param status_code = 200
* method get
And the first request-response is:
19:04:20.612 [main] DEBUG com.intuit.karate - request:
1 > GET http://httpbin.org/redirect-to?url=http%3A%2F%2Fhttpbin.org%2Fget&status_code=200
1 > Accept-Encoding: gzip,deflate
1 > Connection: Keep-Alive
1 > Host: httpbin.org
1 > User-Agent: Apache-HttpClient/4.5.11 (Java/1.8.0_231)
19:04:21.287 [main] DEBUG com.intuit.karate - response time in milliseconds: 670.62
1 < 302
1 < Access-Control-Allow-Credentials: true
1 < Access-Control-Allow-Origin: *
1 < Connection: keep-alive
1 < Content-Length: 0
1 < Content-Type: text/html; charset=utf-8
1 < Date: Tue, 25 Feb 2020 13:34:21 GMT
1 < Location: http://httpbin.org/get
1 < Server: gunicorn/19.9.0
So you must be missing something or making the wrong request to the wrong end-point. Also make sure you read about the followRedirects configuration in the docs.
I've had a practice on scraping with requests-html package under python 3.6. I've tried relevant websites but only one, poetryfoundation.org, https://www.poetryfoundation.org/poems/browse#page=1&sort_by=recently_added&topics=20, returns the incorrect page. I'll demonstrate it in detail.
Here is the source code, the code just import requests-html and return poems which is wrapped in :
from requests_html import HTMLSession
class Scrapy:
def __init__(self, session):
self.session = session
def request_content(self, url):
page = self.session.get(url)
results = page.html.find('div.c-feature')
a = True
if __name__ == '__main__':
session = HTMLSession()
scrapy = Scrapy(session)
url = 'https://www.poetryfoundation.org/poems/browse#page=1&sort_by=recently_added&topics=20'
scrapy.request_content(url=url)
no matter what parameter I change the parameter in the url, it will just return one incorrect page
Thanks for your time
it is different page when you use requests and selenium,because website is using javascript to process the data
from selenium import webdriver
import requests
url = 'https://www.poetryfoundation.org/poems/browse#page=1&sort_by=recently_added&topics=20'
if __name__ == '__main__':
with requests.Session() as ses:
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36",
"Accept": "*/*",
"Referer": "https://www.poetryfoundation.org/poems/browse",
"Accept-Encoding": "gzip, deflate, br",
}
req = ses.get(url,headers=headers)
A = req.text
dr = webdriver.PhantomJS()
dr.get(url)
B = dr.page_source
dr.close()
print(type(A) == type(B))
print(A == B)
print(len(A),len(B))
Output
True # type(A) == type(B)
False # A == B
365477 482831
I'm trying to set up a simple HTTP web server on ESP8266-01 (1MB flash), which has latest 1.9.3 MicroPython firmware. The purpose is to be able to configure the credentials for the home network, to which the STA interface would ultimately connect.
So the code at high level does this:
Turns on the AP interface
Some one would connect to 192.168.0.1/index.html, which would have a form for username and password. We just need to put admin/admin. Clicking on Submit button should do a POST to 192.168.0.1/configure.html
Configure.html is a web page with a form, where SSID and password would be input. I hope you can follow more details by looking at the code below
I am facing two problems:
The total number of bytes that I receive when index.html form is submitted are not complete. I get to around Referrer (that too partial), all a total of around 560 bytes. This is when I do it from mobile browser. Funny thing, it always gets exactly those many bytes. I can share what I get if that's going to help.
The watch dog timer sometimes reboots my module. I'm doing most of the suggested changes in my code - using small sleeps. Is there any way in MicroPython on ESP8266, through which I can 'feed' the WDT so that it does not 'time-out' and reboot my module?
The following is my code:
import gc
import network
gc.collect()
import machine
gc.collect()
import ubinascii
gc.collect()
import ujson
gc.collect()
import uos
gc.collect()
import utime
gc.collect()
import socket
gc.collect()
import select
gc.collect()
html = """<!DOCTYPE html>
<html>
<head> <title>Ouroboros IoT Login</title> </head>
<body>
<form action="configure.html" method="POST">
Username : <input type="text" name="username"></br>
Password: <input type="password" name="password" ></br>
<input type="submit" value="submit" name="submit">
</form>
</body>
</html>
"""
login_fail_html = """<!DOCTYPE html>
<html>
<head> <title>Ouroboros IoT Login</title> </head>
<body>
<h2>Incorrect Credentials!</h2><br>Please login<br>
<form action="configure.html" method="POST">
Username : <input type="text" name="username"></br>
Password: <input type="password" name="password" ></br>
<input type="submit" value="submit" name="submit">
</form>
</body>
</html>
"""
# Check if file exists
def fileExists(fileName):
try:
uos.stat(fileName)
print("File " + fileName + " found!")
return True
except OSError:
print("No file " + fileName + " found!")
return False
# Turns WiFi ON for configuration
def turn_wifi_on():
# Setup the AP interface
ap_if = network.WLAN(network.AP_IF)
ap_if.active(False)
ap_if.active(True)
# Get the MACADDRESS - without any spaces
macaddress = ubinascii.hexlify(ap_if.config('mac'),'').decode()
ap_if.config(essid="OUB1_"+macaddress, password="12345678")
#ap_if.config(essid="OUB1_"+macaddress)
ap_if.ifconfig(('192.168.0.1', '255.255.255.0', '192.168.0.1', '192.168.0.1'))
# Configure the AP to static IPs
def turn_wifi_off():
ap_if = network.WLAN(network.AP_IF)
ap_if.active(False)
# Find out the stored IoT secret content
def get_iot_secret():
fileName = 'alpha.txt'
if fileExists(fileName):
f = open(fileName)
content_str = f.read()
f.close()
return content_str
else:
return 'asdasrefwefefergf9rerf3n4r23irn1n32f'
# Find out the stored home network credential if exist
def get_wifi_config():
fileName = 'wifi.conf'
if fileExists(fileName):
f = open(fileName)
content_str = f.read()
f.close()
content = ujson.loads(content_str)
return content
else:
return None
# Set the home network credentials
def save_wifi_config(essid, passphrase):
f = open('wifi.conf', 'w')
config = {'essid':essid, 'passphrase':passphrase}
config_str = ujson.dumps(config)
f.write(config_str)
f.close()
# Find out the stored login credentials
def get_login_config():
fileName = 'login.conf'
if fileExists(fileName):
f = open(fileName)
content_str = f.read()
f.close()
content = ujson.loads(content_str)
return content
else:
# No file exists so far, so use the admin/admin credentials
return {'user':'admin','password':'admin'}
# Set the login credentials
def save_login_config(user, password):
f = open('login.conf', 'w')
config = {'user':user, 'password':password}
config_str = ujson.dumps(config)
f.write(config_str)
f.close()
def turn_gpio_on(device_num):
# Device Num to Pin Mapping
if device_num == 0:
pin_num = 0
elif device_num == 1:
pin_num = 2
# Check Pin
pin = machine.Pin(pin_num)
if pin.value() == 0:
pin.on()
# else it is already at HIGH state, nothing to do
def turn_gpio_off(device_num):
# Device Num to Pin Mapping
if device_num == 0:
pin_num = 0
elif device_num == 1:
pin_num = 2
# Check Pin
pin = machine.Pin(pin_num)
if pin.value() == 1:
pin.off()
# else it is already at LOW state, nothing to do
def init_pin(device_num):
# Device Num to Pin Mapping
if device_num == 0:
pin_num = 0
elif device_num == 1:
pin_num = 2
#open GPIO0 in output mode & turn it off by default
pin = machine.Pin(pin_num, machine.Pin.OUT)
# Turn off both GPIO initially
turn_gpio_off(device_num)
# Find out the post parameters in a dictionary
def get_post_params(req):
print("Inside GET POST PARAMS : req = " + req)
post_params = req.split('\r\n')[-1:][0]
# Check if the post body contains the necessary fields
# Split the post_params by &
# params : ['username=', 'password=', 'method=POST', 'url=http%3A%2F%2Ftwig-me.com%2Fv1%2Fusergroups%2FWKMUYXELA9LCC', 'jsondata=', 'submit=submit']
print("post_params : " + post_params)
params = post_params.split('&')
print("Params")
print(params)
# Initialize the key value pair dict
post_dict = {}
# Iterate on each param
for param in params:
# Each param would be like 'method=POST', etc
key_val = param.split('=')
print("Key Val :")
print(key_val)
key = key_val[0]
val = key_val[1]
# Update post_dict
post_dict[key] = val
return post_dict
# This web server takes care of the WiFi configuration
# max_run_sec
def web_server(max_run_sec = None):
# Turn wifi interface ON
turn_wifi_on()
# Create server socket
addr = socket.getaddrinfo('0.0.0.0', 80)[0][-1]
s = socket.socket()
# TODO : If both the wifi and sta are operating simultaneously, then bind only to WiFi
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(addr)
s.listen(1)
# s.settimeout(1)
poller = select.poll()
poller.register(s, select.POLLIN)
# Get the current time since epoch
startTimeEpoch = utime.time()
while True:
events = poller.poll(200) # time in milliseconds
if events:
try:
gc.collect()
res = s.accept()
client_s = res[0]
client_addr = res[1]
req = ''
#while True:
# data = client_s.recv(200)
# if data:
# req += str(data, 'utf8')
# else:
# break
# utime.sleep_ms(50)
req = client_s.recv(4096)
req = req.decode()
print(req)
req = str(req)
# Came here means that there has been some connection!
# Reset the start time epoch in such a case:
startTimeEpoch = utime.time()
# Check route now
if req.find('configure.html') != -1:
print("Got configure request!\r\n")
# Check if the username and password are correct, if not, configure:
login_config = get_login_config()
username = login_config['user']
pwd = login_config['password']
print("Username : " + username + ", pwd : " + pwd)
# Find the POST PARAMETERS sent
# There would be just one entry in the array, so get the 0th index directly
# post_params : 'username=&password=&method=POST&url=http%3A%2F%2Ftwig-me.com%2Fv1%2Fusergroups%2FWKMUYXELA9LCC&jsondata=&submit=submit'
print("Came here A")
post_dict = get_post_params(req)
# Now check if the post_dict has the key and value for username and password as needed?
username_post = post_dict['username']
password_post = post_dict['password']
print("Came here B")
# Check if the password is same as expected
if (username_post == username) and (password_post == pwd):
hidden_input = '<input type="hidden" name="username" value="' + username + '"><input type="hidden" name="passphrase" value="' + pwd + '">'
# Send the login username and password inside the hidden input field
configure_html = "<!DOCTYPE html><html><head> <title>Ouroboros IoT WiFi Configuration Page</title> </head><body><form action=\"configure_wifi.html\" method=\"POST\">WiFi SSID : <input type=\"text\" name=\"essid\"></br>WiFi Password: <input type=\"password\" name=\"passphrase\" ></br>" + hidden_input + "<input type=\"submit\" value=\"submit\" name=\"submit\"></form></body></html>"
# TODO : Also show link to webpage, where from we can change the login credentials
client_s.send(configure_html)
else:
client_s.send(login_fail_html)
elif req.find('configure_wifi.html') != -1:
# Check if the username and password are correct, if not, configure:
login_config = get_login_config()
username = login_config['user']
pwd = login_config['password']
# Get post parameters
post_dict = get_post_params(req)
# Now check if the post_dict has the key and value for username and password as needed?
username_post = post_dict['username']
password_post = post_dict['password']
# Check if the password is same as expected
if (username_post == username) and (password_post == pwd):
# Do some sanity check for handling the new wifi ssid and password
new_wifi_ssid = post_dict['essid']
new_wifi_passphrase = post_dict['passphrase']
# Set the wifi credentials
save_wifi_config(new_wifi_ssid, new_wifi_passphrase)
client_s.send('<!DOCTYPE html><html><head> <title>Ouroboros IoT WiFi Configuration Success</title> </head><body>Configuration successful!<br>Device would go into reboot now!</body></html>')
# Reboot device now
machine.reset()
else:
client_s.send(login_fail_html)
elif req.find('index.html') != -1:
print("Got index.html request!\r\n")
client_s.send(html)
else :
# Do nothing
print("Invalid request received! Show the login page again!\r\n")
client_s.send(html)
client_s.close()
machine.idle()
except OSError:
# Got no request and it timedout!
print("Timed-out, no request received!\r\n")
except Exception as e:
print("Got some exception\r\n")
print(str(e))
finally:
if max_run_sec is not None:
elapsedTime = utime.time() - startTimeEpoch
if elapsedTime > max_run_sec:
# Max run time of web server has elapsed, time to exit this mode!
break
utime.sleep_ms()
machine.idle()
# When while loop ends!
s.close()
# Turn wifi interface OFF
turn_wifi_off()
# Starts a thread which runs the web server to handle WiFi
def start_web_server(max_run_sec = None):
# start_new_thread(web_server, (max_run_sec))
web_server(max_run_sec)
############# MAIN ##########################
# Initialize two pins to INPUT and OFF by default
init_pin(0)
init_pin(1)
#turn_wifi_off()
# Check if the home wifi network has been setup
# Check if home wifi config is valid, if so, connect to it
# If home wifi is not configured, then use the Web server all the time.
if get_wifi_config() is None:
# Came here means the wifi is not configured
# Start the web server
print("Starting web server")
start_web_server()
EDIT 1:
I'm able to setup WDT and feed it. So no more WDT reboots. However, POST issue is still there:
FYI, the following is the response:
POST /configure.html HTTP/1.1
Host: 192.168.0.1
Connection: keep-alive
Content-Length: 43
Cache-Control: max-age=0
Origin: http://192.168.0.1
Upgrade-Insecure-Requests: 1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Linux; Android 5.1.1; Redmi Note 3 Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.123 Mobile Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: http://192.168.0.1/index.html
Accept-Encoding: g
As could be seen, the packet received is partial, the Content-Length header says 43 byte payload. But its not received. When using "nc", and running server locally, the packet received is follows:
POST /configure.html HTTP/1.1
Host: 192.168.0.1
Connection: keep-alive
Content-Length: 43
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: null
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/65.0.3325.181 Chrome/65.0.3325.181 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
username=admin&password=admin&submit=submit
Here, the 43 byte length payload could be easily seen.
So my question is, is the almost 800 byte payload too much to handle for ESP8266? Is there anything that could be done to remove the unnecessary headers that browser is sending? Any way to get all the data if its fragmented?
I faced a similar problem, my configuration is slightly different though.
I import my html from html_files.py as shown below.
cred_prompt = """
<!DOCTYPE html>
<html>
<head>
<title>ESP8266 connection</title>
</head>
<body>
<h2>
Enter the UID and Password to connect to WiFi
</h2>
<form action="/post">
uid: <input type="text" name="uid">
password: <input type="text" name="password">
<input type="submit" value="Submit">
</form><br>
</body>
</html>
"""
This is the function I use to get the credentials. I later save them into a file (Successive boots will use them)
def get_new_creds():
sta_if.disconnect()
print("Setting up webpage to get new credentials")
addr = socket.getaddrinfo('0.0.0.0', 80)[0][-1]
soc = socket.socket()
soc.bind(addr)
soc.listen(5)
print("Listening on", addr)
while True:
client, addr = soc.accept()
print("Client connected from", addr)
request = client.recv(1024)
request = request.decode().split()
uid, pwd = '', ''
if 'uid' in request[1]:
uid = request[1].split('&')[0].split('=')[1]
pwd = request[1].split('&')[1].split('=')[1]
write_new_creds(uid, pwd)
connect_to_wifi()
print("The UID is", uid, "and the Password is", pwd)
client.send('HTTP/1.1 200 OK\r\nContent-type: text/html\r\n\r\n')
client.send(html_files.connection_response.format(sta_if.ifconfig()[0]))
return uid, pwd
print(request)
client.send('HTTP/1.1 200 OK\r\nContent-type: text/html\r\n\r\n')
client.send(html_files.cred_prompt)
client.close()
This is the whole boot.py code if that helps.
#tornado.web.authenticated
#tornado.web.asynchronous
#tornado.gen.coroutine
def post(self):
try:
files_body = self.request.files['file']
except:
error_msg = u"failed to upload file"
error_msg = self.handle_error(error_msg)
self.finish(dict(is_succ=False, error_msg=error_msg))
return
file_ = files_body[0]
filename = file_['filename']
# asynchronous request， obtain OCR info
files = [('image', filename, file_['body'])]
fields = (('api_key', config.OCR_API_KEY), ('api_secret', config.OCR_API_SECRET))
content_type, body = encode_multipart_formdata(fields, files)
headers = {"Content-Type": content_type, 'content-length': str(len(body))}
request = tornado.httpclient.HTTPRequest(config.OCR_HOST, method="POST", headers=headers, body=body,
validate_cert=False, request_timeout = 30)
try:
response = yield tornado.httpclient.AsyncHTTPClient().fetch(request)
except Exception, e:
logging.error(u'orc timeout {}'.format(e))
error_msg = u"OCR timeout"
error_msg = self.handle_error(error_msg)
self.finish(dict(is_succ=False, error_msg=error_msg))
return
if not response.error and response.body:
data = json.loads(response.body)
self.extra_info(data)
result = dict(is_succ=True, error_msg=u"", data=data)
else:
result = dict(is_succ=False, error_msg=u"request timeout", data={})
self.finish(result)
as the code shown, I want to write an api to handle id-card picture upload, and post a request to third part interface for getting the information of id-card.
This api can run well on my PC，however it timeouts on Testing Server. I cannot figure out where the problem is.