I am trying to read the contents of a web page using a Groovy script. The page contains the readings from one of my temperature sensors that I want to save regularly. I have tried the simplest variant:
def url = "https://measurements.mobile-alerts.eu/Home/MeasurementDetails?deviceid=021B5594EAB5&vendorid=60122a8b-b343-49cb-918b-ad2cdd6dff16&appbundle=eu.mobile_alerts.mobilealerts&fromepoch=1674432000&toepoch=1674518400&from=23.01.2023%2000:00&to=24.01.2023%2000:00&command=refresh"
def res = url.toURL().getText()
println( res)
The result is:
Caught: java.io.IOException: Server returned HTTP response code: 403 for URL: (my url)
In any browser, this URL works without problems.
I would be very grateful for any tips on how to solve this problem.
HTTP code 403 means that a client is forbidden from accessing a valid URL. In other words, the server knows that you are not making a request via a web browser. To bypass this restriction, you need to specify a User-Agent in the request header.
For example:
def url = 'https://measurements.mobile-alerts.eu/Home/MeasurementDetails?deviceid=021B5594EAB5&vendorid=60122a8b-b343-49cb-918b-ad2cdd6dff16&appbundle=eu.mobile_alerts.mobilealerts&fromepoch=1674432000&toepoch=1674518400&from=23.01.2023%2000:00&to=24.01.2023%2000:00&command=refresh'
def res = url.toURL().getText(requestProperties:
['User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0'])
println res
You can switch to other valid user-agent values.
I am trying to put together a simple thermometer that provides the temperature on the OLED display as well as via http requests on an ESP8266 using MicroPython.
A Poller Object has been used to prevent the websocket from blocking the loop (so measurements and OLED display can be updated).
#CREATE SOCKET
serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serverSocket.bind(('', 80))
serverSocket.listen(5)
#REGISTER SOCKET TO THE POLLER
pollerObject = select.poll()
pollerObject.register(serverSocket, select.POLLIN)
#PERFORM FIRST MEASUREMENT AT STARTUP
last_meas_time = startup_time
sensor_readings = read_sensor()
print(sensor_readings)
display.display_measurement(str(temp),str(hum))
#LOOP FOREVER
while True:
#ONE MEASUREMENT UPDATE EVERY 30s
if(time.time() - last_meas_time >= 30):
sensor_readings = read_sensor()
print(sensor_readings)
display.display_measurement(str(temp),str(hum))
last_meas_time = time.time()
#WAIT UP TO 10s FOR INCOMING CONNECTIONS
fdVsEvent = pollerObject.poll(10000)
for descriptor, Event in fdVsEvent:
print()
print("Got an incoming connection request")
print("Start processing")
# Do accept() on server socket or read from a client socket
conn, addr = serverSocket.accept()
print('Got a connection from %s' % str(addr))
request = conn.recv(1024)
print('Content = %s' % str(request))
response = web_page()
conn.send('HTTP/1.1 200 OK\n')
conn.send('Content-Type: text/html\n')
conn.send('Connection: close\n\n')
conn.sendall(response)
conn.close()
It seems to be working fine for some time, but I found two issues with it where I would appreciate your help:
Even though I connect to it only once, 2 or 3 requests are shown as received in shell terminal as you can see below. Why does that happen and how could I address it? Can it be so that the browser waited long enough to send a second or third request?
MPY: soft reboot
Connection successful
('192.168.1.74', '255.255.255.0', '192.168.1.1', '192.168.1.1')
b'29.0,24.0'
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 58581)
Content = b'GET / HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nDNT: 1\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 58582)
Content = b'GET /favicon.ico HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nDNT: 1\r\nAccept: image/webp,image/apng,image/*,*/*;q=0.8\r\nReferer: http://192.168.1.74/\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'
After some long time running I won't be able to connect to it anymore as it will not respond. Is there something obviosly wrong with my approach? This was what I got from the console:
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 59158)
Content = b'GET / HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nDNT: 1\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 59157)
Content = b'GET /favicon.ico HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nDNT: 1\r\nAccept: image/webp,image/apng,image/*,*/*;q=0.8\r\nReferer: http://192.168.1.74/\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 59160)
Content = b''
Traceback (most recent call last):
File "main.py", line 104, in
OSError: [Errno 104] ECONNRESET
MicroPython v1.13 on 2020-09-11; ESP module with ESP8266
Type "help()" for more information.
>>>
Line 104 corresponds to:
conn.sendall(response)
Thanks!
Even though I connect to it only once, 2 or 3 requests are shown as received in shell terminal as you can see below. Why does that happen and how could I address it? Can it be so that the browser waited long enough to send a second or third request?
This depends on how the browser connects to your server. There might be multiple requests the browser is looking for, or the browser has a timeout value for the socket connecting to your server. I don't have any web knowledge, but it looks like two requests for different information. How that information is handled, should be passed onto web_page(). It looks like you are sending the entirety of a web page and not the specific content it is looking for.
After some long time running I won't be able to connect to it anymore as it will not respond. Is there something obviously wrong with my approach?
What might be happening is you have socket.sendall() blocking any new sockets from being created. Also note, even though you have properly closed the socket, the socket may still have data to send. It has been marked closed, but the OS might not have closed it yet.
You are on the right track by using select.poll(). At first glance, it seems that registering your serverSocket with pollerObject (select.poll) would handle future connections. That isn't what is happening. You are registering just the one socket to pollerObject. The severSocket is getting the select.POLLIN event for the incoming connection from the browser. You need a way to add/register new sockets created by serverSocket to pollerObject so you can service other sockets.
Now the best example of what you are trying to do in micropython is to make something similar to the selector example in Python 3 Selectors.
import selectors
import socket
sel = selectors.DefaultSelector()
def accept(sock, mask):
conn, addr = sock.accept() # Should be ready
print('accepted', conn, 'from', addr)
conn.setblocking(False)
sel.register(conn, selectors.EVENT_READ, read)
def read(conn, mask):
data = conn.recv(1000) # Should be ready
if data:
print('echoing', repr(data), 'to', conn)
conn.send(data) # Hope it won't block
else:
print('closing', conn)
sel.unregister(conn)
conn.close()
sock = socket.socket()
sock.bind(('localhost', 1234))
sock.listen(100)
sock.setblocking(False)
sel.register(sock, selectors.EVENT_READ, accept)
while True:
events = sel.select()
for key, mask in events:
callback = key.data
callback(key.fileobj, mask)
Generally, you won't have to worry about filling the socket transmit buffer with socket.send(), but you should handle it. For now, I would put some debug prints before and after the socket.sendall() since that will block/retry until all the data is sent. In the case that not all the data has sent, you will have to register the socket for a write ready event, and pass the remaining data that needs to be sent. This is a bit more complicated.
Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 59160)
Content = b''
Traceback (most recent call last):
File "main.py", line 104, in
OSError: [Errno 104] ECONNRESET
MicroPython v1.13 on 2020-09-11; ESP module with ESP8266
Type "help()" for more information.
>>>
The problem you are running into above is you probably have a socket connection that has timed out. TCP is letting you know the connection has expired. You should handle this with a try except else clause.
I followed this blog
https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws-lambda-and-amazon-api-gateway/
and this repo
https://github.com/awslabs/serverless-image-resizing
To be able to resize images on AWS S3 on the fly. The bucket is created successfully. Whenever I upload image to the bucket, I can resize it from the url as indicated in the blog, everything works fine.
The problem is when an image is uploaded from the code (Ruby on Rails using paperclip), then the image can be accessed normally, however when trying with resizing url (e.g. /100x200/photo.png) I receive this in the browser as json response
{
"message": "Internal server error"
}
The response headers are:
Request URL:<url>
Request Method:GET
Status Code:502
Remote Address:<IP>
Referrer Policy:no-referrer-when-downgrade
Response Headers
content-length:36
content-type:application/json
date:Thu, 23 Nov 2017 09:59:24 GMT
status:502
via:1.1 <hash>.cloudfront.net (CloudFront)
x-amz-cf-id:<id>
x-amzn-requestid:<id>
x-cache:Error from cloudfront
Request Headers
:authority:<url>
:method:GET
:path:/prod?key=<path/to/photo>
:scheme:https
accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.9,de;q=0.8,ar;q=0.7
cache-control:max-age=0
upgrade-insecure-requests:1
user-agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36
Query String Parameters
view source
view URL encoded
key:<path/to/photo>
The very strange thing is that, after the error occurs, when I upload the same image manually again under any place in the bucket then I try again using the original resize url, not the new uploaded image url that caused a problem, it works normally and I can resize to any size !
So it seems like uploading manually triggers something related to this very image data ? I have no clue!
The error here that you put the required size before file name directly, you should put it before the whole file path.
Make sure that you follow this pattern:
http://$BucketWebsiteHost/$size/$imagePath
Example:
http://mybucketname.s3-website.mybucketregion.amazonaws.com/60x50/photos/attachments/000/002/002/original/image.jpg
I have an access.log nginx with cookie:
99.20.231.22 www.carite.com - [01/Dec/2015:03:00:10 -0600] "GET /?mode=_ajax&_imod[]=i159330&make=Mercedes-Benz&_=1448960297171 HTTP/1.1" 200 1182 "http://www.carite.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1" "PHPSESSID=ebg5n89m9pc1iamekii1qra5k0; chooseStoreNotificationShown=1; dfa_visit=1448960180633603603; dfa_visitor=1448960180633796491; mod-compare-box=%7B%22vehicles%22%3A%7B%22v11279294%22%3A%7B%22vuid%22%3A%2211279294%22%2C%22isCompared%22%3Afalse%7D%7D%2C%22compareAll%22%3Atrue%2C%22cookieLifeTime%22%3A30%2C%22cookiePath%22%3A%22%5C%2F%22%7D; _ga=GA1.2.10339867.1448960182; _gali=make; _gat_a1=1; _gat_a2=1; _gat_a3=1; _gat_a4=1; usy46gabsosd=collserve__-2_1448960382693_8786" 80 0.295
Can I specify Yandex-tank get cookie from access log and add it to every yandex-tank request?
Also I need get header "Host:" from access log instead of specify it in load.ini like:
headers = [Host: www.carite.com]
You have two options:
to make stepper read cookies along with uri from access.log (it
should be done around there
https://github.com/yandex/yandex-tank/blob/master/yandextank/stepper/missile.py#L213)
make a separate file from access.log, in https://yandextank.readthedocs.org/en/latest/tutorial.html#uri-style-uris-in-file format. Headers are overriden on the go, so you can redefine headers anywhere
For example it could be like this:
[Host: www.carite.com]
[Cookie: PHPSESSID=ebg5n89m9pc1iamekii1qra5k0; chooseStoreNotificationShown=1; dfa_visit=1448960180633603603; dfa_visitor=1448960180633796491; ...]
/?mode=_ajax&imod[]=i159330&make=Mercedes-Benz&=1448960297171
...
[Host: example.com]
[Cookie: myowncookie=1]
/something
...
I would advice to use the 2nd way as an easiest one
I have a backbone app running on top of my Rails app, which is making a number of AJAX requests. Oddly enough, the session ID cookie that my browser stores never seems to change. I can see the next request being made and the reply returning success. The response still hangs on to the old session ID, however, and thus I can't logout or change user state in any way.
Here's an example of the headers being sent:
Request to /login.json:
Accept:*/*
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Cookie:_session_id=BAh7CEkiD3Nlc3Npb25faWQGOgZFRkkiJWY0YTI1Y2I5ZDE5MTgyYjJmN2MzMzZiMmE5ZWE2ZTAyBjsAVEkiEF9jc3JmX3Rva2VuBjsARkkiMTB1ZEozbzJCbUl4ZjluUDdCZVVEOXBhL0ZUeXNVRnFiMlFLblFTVUZPNGM9BjsARkkiDHVzZXJfaWQGOwBGaQg%3D--cf57849d4a4c0cbc5608574d959a772080b3afc5
Host:localhost:3000
If-None-Match:"7363e85fe9edee6f053a4b319588c086"
Referer:http://localhost:3000/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5
X-CSRF-Token:0udJ3o2BmIxf9nP7BeUD9pa/FTysUFqb2QKnQSUFO4c=
X-Requested-With:XMLHttpRequest
Response from /login.json:
Cache-Control:max-age=0, private, must-revalidate
Date:Tue, 15 May 2012 22:44:46 GMT
Etag:"7363e85fe9edee6f053a4b319588c086"
Server:WEBrick/1.3.1 (Ruby/1.9.2/2011-07-09)
Set-Cookie:_session_id=BAh7B0kiD3Nlc3Npb25faWQGOgZFRkkiJWY0YTI1Y2I5ZDE5MTgyYjJmN2MzMzZiMmE5ZWE2ZTAyBjsAVEkiEF9jc3JmX3Rva2VuBjsARkkiMTB1ZEozbzJCbUl4ZjluUDdCZVVEOXBhL0ZUeXNVRnFiMlFLblFTVUZPNGM9BjsARg%3D%3D--b16d1f8e4bba3c86d5e3e9ece1ae50f1ad898d00; path=/; HttpOnly
X-Request-Id:8e278bdd3504257bcb9956e068ca3ca3
X-Runtime:0.014739
X-Ua-Compatible:IE=Edge
Notice it's giving me a new session ID. Yet when I make my next request...
Accept:application/json, text/javascript, */*; q=0.01
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Cookie:_session_id=BAh7CEkiD3Nlc3Npb25faWQGOgZFRkkiJWY0YTI1Y2I5ZDE5MTgyYjJmN2MzMzZiMmE5ZWE2ZTAyBjsAVEkiEF9jc3JmX3Rva2VuBjsARkkiMTB1ZEozbzJCbUl4ZjluUDdCZVVEOXBhL0ZUeXNVRnFiMlFLblFTVUZPNGM9BjsARkkiDHVzZXJfaWQGOwBGaQg%3D--cf57849d4a4c0cbc5608574d959a772080b3afc5
Host:localhost:3000
If-None-Match:"53135bfd970c6b34f39ea3c4780ed240"
Referer:http://localhost:3000/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5
X-CSRF-Token:0udJ3o2BmIxf9nP7BeUD9pa/FTysUFqb2QKnQSUFO4c=
X-Requested-With:XMLHttpRequest
The cookie has never changed.
I figured it out. I had an async. request firing at the same time, so the browser was never getting a chance to get the proper session ID.