I am trying to put together a simple thermometer that provides the temperature on the OLED display as well as via http requests on an ESP8266 using MicroPython.
A Poller Object has been used to prevent the websocket from blocking the loop (so measurements and OLED display can be updated).
#CREATE SOCKET
serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serverSocket.bind(('', 80))
serverSocket.listen(5)
#REGISTER SOCKET TO THE POLLER
pollerObject = select.poll()
pollerObject.register(serverSocket, select.POLLIN)
#PERFORM FIRST MEASUREMENT AT STARTUP
last_meas_time = startup_time
sensor_readings = read_sensor()
print(sensor_readings)
display.display_measurement(str(temp),str(hum))
#LOOP FOREVER
while True:
#ONE MEASUREMENT UPDATE EVERY 30s
if(time.time() - last_meas_time >= 30):
sensor_readings = read_sensor()
print(sensor_readings)
display.display_measurement(str(temp),str(hum))
last_meas_time = time.time()
#WAIT UP TO 10s FOR INCOMING CONNECTIONS
fdVsEvent = pollerObject.poll(10000)
for descriptor, Event in fdVsEvent:
print()
print("Got an incoming connection request")
print("Start processing")
# Do accept() on server socket or read from a client socket
conn, addr = serverSocket.accept()
print('Got a connection from %s' % str(addr))
request = conn.recv(1024)
print('Content = %s' % str(request))
response = web_page()
conn.send('HTTP/1.1 200 OK\n')
conn.send('Content-Type: text/html\n')
conn.send('Connection: close\n\n')
conn.sendall(response)
conn.close()
It seems to be working fine for some time, but I found two issues with it where I would appreciate your help:
Even though I connect to it only once, 2 or 3 requests are shown as received in shell terminal as you can see below. Why does that happen and how could I address it? Can it be so that the browser waited long enough to send a second or third request?
MPY: soft reboot
Connection successful
('192.168.1.74', '255.255.255.0', '192.168.1.1', '192.168.1.1')
b'29.0,24.0'
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 58581)
Content = b'GET / HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nDNT: 1\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 58582)
Content = b'GET /favicon.ico HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nDNT: 1\r\nAccept: image/webp,image/apng,image/*,*/*;q=0.8\r\nReferer: http://192.168.1.74/\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'
After some long time running I won't be able to connect to it anymore as it will not respond. Is there something obviosly wrong with my approach? This was what I got from the console:
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 59158)
Content = b'GET / HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nDNT: 1\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 59157)
Content = b'GET /favicon.ico HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nDNT: 1\r\nAccept: image/webp,image/apng,image/*,*/*;q=0.8\r\nReferer: http://192.168.1.74/\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 59160)
Content = b''
Traceback (most recent call last):
File "main.py", line 104, in
OSError: [Errno 104] ECONNRESET
MicroPython v1.13 on 2020-09-11; ESP module with ESP8266
Type "help()" for more information.
>>>
Line 104 corresponds to:
conn.sendall(response)
Thanks!
Even though I connect to it only once, 2 or 3 requests are shown as received in shell terminal as you can see below. Why does that happen and how could I address it? Can it be so that the browser waited long enough to send a second or third request?
This depends on how the browser connects to your server. There might be multiple requests the browser is looking for, or the browser has a timeout value for the socket connecting to your server. I don't have any web knowledge, but it looks like two requests for different information. How that information is handled, should be passed onto web_page(). It looks like you are sending the entirety of a web page and not the specific content it is looking for.
After some long time running I won't be able to connect to it anymore as it will not respond. Is there something obviously wrong with my approach?
What might be happening is you have socket.sendall() blocking any new sockets from being created. Also note, even though you have properly closed the socket, the socket may still have data to send. It has been marked closed, but the OS might not have closed it yet.
You are on the right track by using select.poll(). At first glance, it seems that registering your serverSocket with pollerObject (select.poll) would handle future connections. That isn't what is happening. You are registering just the one socket to pollerObject. The severSocket is getting the select.POLLIN event for the incoming connection from the browser. You need a way to add/register new sockets created by serverSocket to pollerObject so you can service other sockets.
Now the best example of what you are trying to do in micropython is to make something similar to the selector example in Python 3 Selectors.
import selectors
import socket
sel = selectors.DefaultSelector()
def accept(sock, mask):
conn, addr = sock.accept() # Should be ready
print('accepted', conn, 'from', addr)
conn.setblocking(False)
sel.register(conn, selectors.EVENT_READ, read)
def read(conn, mask):
data = conn.recv(1000) # Should be ready
if data:
print('echoing', repr(data), 'to', conn)
conn.send(data) # Hope it won't block
else:
print('closing', conn)
sel.unregister(conn)
conn.close()
sock = socket.socket()
sock.bind(('localhost', 1234))
sock.listen(100)
sock.setblocking(False)
sel.register(sock, selectors.EVENT_READ, accept)
while True:
events = sel.select()
for key, mask in events:
callback = key.data
callback(key.fileobj, mask)
Generally, you won't have to worry about filling the socket transmit buffer with socket.send(), but you should handle it. For now, I would put some debug prints before and after the socket.sendall() since that will block/retry until all the data is sent. In the case that not all the data has sent, you will have to register the socket for a write ready event, and pass the remaining data that needs to be sent. This is a bit more complicated.
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 59160)
Content = b''
Traceback (most recent call last):
File "main.py", line 104, in
OSError: [Errno 104] ECONNRESET
MicroPython v1.13 on 2020-09-11; ESP module with ESP8266
Type "help()" for more information.
>>>
The problem you are running into above is you probably have a socket connection that has timed out. TCP is letting you know the connection has expired. You should handle this with a try except else clause.
Related
I am trying to read the contents of a web page using a Groovy script. The page contains the readings from one of my temperature sensors that I want to save regularly. I have tried the simplest variant:
def url = "https://measurements.mobile-alerts.eu/Home/MeasurementDetails?deviceid=021B5594EAB5&vendorid=60122a8b-b343-49cb-918b-ad2cdd6dff16&appbundle=eu.mobile_alerts.mobilealerts&fromepoch=1674432000&toepoch=1674518400&from=23.01.2023%2000:00&to=24.01.2023%2000:00&command=refresh"
def res = url.toURL().getText()
println( res)
The result is:
Caught: java.io.IOException: Server returned HTTP response code: 403 for URL: (my url)
In any browser, this URL works without problems.
I would be very grateful for any tips on how to solve this problem.
HTTP code 403 means that a client is forbidden from accessing a valid URL. In other words, the server knows that you are not making a request via a web browser. To bypass this restriction, you need to specify a User-Agent in the request header.
For example:
def url = 'https://measurements.mobile-alerts.eu/Home/MeasurementDetails?deviceid=021B5594EAB5&vendorid=60122a8b-b343-49cb-918b-ad2cdd6dff16&appbundle=eu.mobile_alerts.mobilealerts&fromepoch=1674432000&toepoch=1674518400&from=23.01.2023%2000:00&to=24.01.2023%2000:00&command=refresh'
def res = url.toURL().getText(requestProperties:
['User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0'])
println res
You can switch to other valid user-agent values.
I have an access.log nginx with cookie:
99.20.231.22 www.carite.com - [01/Dec/2015:03:00:10 -0600] "GET /?mode=_ajax&_imod[]=i159330&make=Mercedes-Benz&_=1448960297171 HTTP/1.1" 200 1182 "http://www.carite.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1" "PHPSESSID=ebg5n89m9pc1iamekii1qra5k0; chooseStoreNotificationShown=1; dfa_visit=1448960180633603603; dfa_visitor=1448960180633796491; mod-compare-box=%7B%22vehicles%22%3A%7B%22v11279294%22%3A%7B%22vuid%22%3A%2211279294%22%2C%22isCompared%22%3Afalse%7D%7D%2C%22compareAll%22%3Atrue%2C%22cookieLifeTime%22%3A30%2C%22cookiePath%22%3A%22%5C%2F%22%7D; _ga=GA1.2.10339867.1448960182; _gali=make; _gat_a1=1; _gat_a2=1; _gat_a3=1; _gat_a4=1; usy46gabsosd=collserve__-2_1448960382693_8786" 80 0.295
Can I specify Yandex-tank get cookie from access log and add it to every yandex-tank request?
Also I need get header "Host:" from access log instead of specify it in load.ini like:
headers = [Host: www.carite.com]
You have two options:
to make stepper read cookies along with uri from access.log (it
should be done around there
https://github.com/yandex/yandex-tank/blob/master/yandextank/stepper/missile.py#L213)
make a separate file from access.log, in https://yandextank.readthedocs.org/en/latest/tutorial.html#uri-style-uris-in-file format. Headers are overriden on the go, so you can redefine headers anywhere
For example it could be like this:
[Host: www.carite.com]
[Cookie: PHPSESSID=ebg5n89m9pc1iamekii1qra5k0; chooseStoreNotificationShown=1; dfa_visit=1448960180633603603; dfa_visitor=1448960180633796491; ...]
/?mode=_ajax&imod[]=i159330&make=Mercedes-Benz&=1448960297171
...
[Host: example.com]
[Cookie: myowncookie=1]
/something
...
I would advice to use the 2nd way as an easiest one
I'm trying to setup a file server inside of my iOS application. I have cocoaHTTPserver setup to do the leg work. It didn't support POST for large-file uploads out of the box so I modifying it to get it working by piping the content body out to a file stream. However, for some strange reason I can never seem to get past around 44MB during an upload. The stream is set for a 30 second timeout and after anywhere from 10-44MB the stream just stop receiving data? The headers look fine coming in (example from Safari but same problem on Chrome):
Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
"Accept-Encoding" = "gzip, deflate";
"Accept-Language" = "en-us";
Connection = "keep-alive";
"Content-Length" = 1256657607;
"Content-Type" = "multipart/form-data; boundary=----WebKitFormBoundary7n0hAe769NCJsl2R";
Host = "192.168.1.102:8088";
Origin = "http://192.168.1.102:8088";
Referer = "http://192.168.1.102:8088/";
"User-Agent" = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Safari/534.45";
Any help would be much appreciated.
Previously i was able to download YouTube videos as mp3 via youtube-mp3.org Using this method:
http://www.youtube-mp3.org/api/pushItem/?item=http%3A//www.youtube.com/watch%3Fv%3D<VIDEOID>&xy=_
Then it returned the video id and they started converting the video on their servers. Then this request would return a JSON string with info about the video and the current conversion status:
http://www.youtube-mp3.org/api/itemInfo/?video_id=<VIDEOID>&adloc=
After repeating the request until the value for status is 'serving' I then started the last request by taking the value for key h from the JSON response from the previous request, and this would download a the mp3 file.
http://www.youtube-mp3.org/get?video_id=<VIDEOID>&h=<JSON string value for h>
Now the first request always returns nothing. The second and third requests only succeed if the requested video is cached on their servers (like popular music videos). If thats not the case then the second request would return nil and so the 3rd request can't be started because of the missing hvalue from the second request. Could anybody help me with getting the website to start a conversion something needs to be wrong with the first URL i just dont know what. Thanks
I just tested it. For the first request, you need to send with it a header of:
Accept-Location: *
Otherwise, it will return a 500 (Internal Server Error). But with that header, it will return a string of the youtube video id, and you can use the 2nd api for checking the progress.
Here's the C# code I used for testing:
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create("FIRST_API_URL");
wr.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7";
wr.Headers.Add("Accept-Location", "*");
string res = (new StreamReader(wr.GetResponse().GetResponseStream())).ReadToEnd();
Btw, you can keep track of the headers in the browser's Network (Chrome) debug tab.
Regards
Is there a way to get the following data from the Application_Error event in the Global.ascx file?
action error came from,
ipaddress error came from,
browser error came from,
browser version error came from,
hostName error came from
??
All that information is contained in the Context.Request property.
Context.Request.Url; // /controller/action?foo=bar so up to you to extract the action
Context.Request.UserHostAddress; // 123.456.789.0123
Context.Request.UserAgent // Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13
And once you are sick of parsing all this crap manually and repeating this code all over again among all your applications you might consider using ELMAH.
hostName error came from
Not sure what you mean here. Isn't that the IP of the client?