I am trying to read the contents of a web page using a Groovy script. The page contains the readings from one of my temperature sensors that I want to save regularly. I have tried the simplest variant:
def url = "https://measurements.mobile-alerts.eu/Home/MeasurementDetails?deviceid=021B5594EAB5&vendorid=60122a8b-b343-49cb-918b-ad2cdd6dff16&appbundle=eu.mobile_alerts.mobilealerts&fromepoch=1674432000&toepoch=1674518400&from=23.01.2023%2000:00&to=24.01.2023%2000:00&command=refresh"
def res = url.toURL().getText()
println( res)
The result is:
Caught: java.io.IOException: Server returned HTTP response code: 403 for URL: (my url)
In any browser, this URL works without problems.
I would be very grateful for any tips on how to solve this problem.
HTTP code 403 means that a client is forbidden from accessing a valid URL. In other words, the server knows that you are not making a request via a web browser. To bypass this restriction, you need to specify a User-Agent in the request header.
For example:
def url = 'https://measurements.mobile-alerts.eu/Home/MeasurementDetails?deviceid=021B5594EAB5&vendorid=60122a8b-b343-49cb-918b-ad2cdd6dff16&appbundle=eu.mobile_alerts.mobilealerts&fromepoch=1674432000&toepoch=1674518400&from=23.01.2023%2000:00&to=24.01.2023%2000:00&command=refresh'
def res = url.toURL().getText(requestProperties:
['User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0'])
println res
You can switch to other valid user-agent values.
Related
I am trying to use python requests to receive my access token for the Amazon Advertising API. The procedure is outlined here: https://advertising.amazon.com/API/docs/v2/guides/authorization Here is what I tried
CLIENT_ID = MyClientID
CLIENT_SECRET = MySecret
RETURN_URL = 'https://myreturn.com/my.php'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.61 Safari/537.36',
}
with requests.Session() as s:
s.headers = headers
r = s.get('https://www.amazon.com/ap/oa?client_id={}&scope=cpc_advertising:campaign_management&error=access_denied&response_type=code&redirect_uri={}'.format(CLIENT_ID,RETURN_URL),headers=headers)
soup = BeautifulSoup(html)
data = {}
form = soup.find('form', {'name': 'signIn'})
for field in form.find_all('input'):
try:
data[field['name']] = field['value']
except:
pass
data[u'email'] = MY_EMAIL
data[u'password'] = MY_PASS
b = s.post('https://www.amazon.com/ap/oa?client_id={}&scope=cpc_advertising:campaign_management&response_type=code&redirect_uri={}',data=data,allow_redirects=True,headers=headers)
i get an error_description=User+not+authenticated&error=access_denied error, what am I doing wrong here?
You DON'T NEED Username and Password in your Python Script to authenticate!
What you need is CLIENT_ID, SCOPE and REDIRECT_URI and three requests:
Get authorization code:
GET https://www.amazon.com/ap/oa?client_id={{CLIENT_ID}}&scope={{SCOPE}}&response_type=code&redirect_uri={{REDIRECT_URI}}
This will open the 'Login with Amazon' Consent Page, where you (or your customer) log into your Amazon Seller Central account and grant access to the Console APP with API access rights.
Request tokens
POST https://api.amazon.com/auth/o2/token
with headers:
Content-Type:application/x-www-form-urlencoded
with body data:
grant_type:authorization_code
code:{{AUTH_CODE}} <----- returned from step 1
client_id:{{CLIENT_ID}}
client_secret:{{CLIENT_SECRET}}
redirect_uri:{{REDIRECT_URI}}
Get/Refresh access token (every time it is outdated):
POST https://api.amazon.com/auth/o2/token
with headers:
Content-Type:application/x-www-form-urlencoded
charset:UTF-8
with body data:
grant_type:refresh_token
refresh_token:{{REFRESH_TOKEN}} <------ returned from step 2
client_id:{{CLIENT_ID}}
client_secret:{{CLIENT_SECRET}}
With the CLIENT_ID and (fresh) access token you can now request every service from the API. For excample listCampaigns:
GET https://advertising-api.amazon.com/v2/sp/campaigns
Headers:
Content-Type:application/json
Amazon-Advertising-API-ClientId:{{CLIENT_ID}}
Amazon-Advertising-API-Scope:{{PROFILE_ID}}
Authorization:Bearer {{ACCESS_TOKEN}} <----- returned from step 3
I've been using Radview's Webload IDE tool for a couple of test simulation projects and it has worked well. But for this one scenario where I have a client web session for a login a screen, it would always fail with a 500 Response error for a particular HTTP post as the page loads.
When I try the scenario to load the page manually with a browser it works fine with no issues.
During the recording I would set clear browser cache and cookies and no luck. And I've also tried out many configuration combinations from the "Recording and Script Generatinon Options: Post Data" settings.
/***** WLIDE - URL : http://192.168.2.2/ - ID:2 *****/
wlGlobals.GetFrames = false
wlGlobals.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko"
wlHttp.Get("http://192.168.2.2/")
// END WLIDE
/***** WLIDE - URL : http://192.168.2.2/Api.ashx?c=Images&action=GetSettings - ID:3 *****/
wlHttp.Header["Referer"] = "http://192.168.2.2/"
wlHttp.FormdataEncodingType = 1
wlHttp.ContentType = "application/x-www-form-urlencoded"
wlHttp.FormData["c"] = "Images"
wlHttp.FormData["action"] = "GetSettings"
wlHttp.Post("http://192.168.2.2/Api.ashx"+"?c=Images&action=GetSettings")
// END WLIDE
Anybody with experience with Radview's Webload can give me some suggestions?
I noticed that commenting out the formdata "c" and "actions" lines works. but later I notice a similar error which requires a sessionID in the URL so I'm not sure if I can comment out the formdata "sessionID" line.
To run the API from Webload you need to specify the authorization if its secured.
Using wlHttp.FormData is not the same as adding a parameter to the URL for a POST request.
FormData will be send as part of the post-data request body, while adding it to the URL will send it as a query string - your sever probably expects one form but not the other.
Contact RadView support if you can't get it to work and they'll help you
I have developed a MVC Application, which, for the purpose of this question only has one controller:
Public Function GetValue()
Return User.Identity.Name
End Function
The application is to be used on an Intranet network, therefore, I have set it to 'Windows Authentication'
The aim is to query this application, through VBA.
Set objHTTP = CreateObject("MSXML2.ServerXMLHTTP")
With objHTTP
.Open "GET", URL, False
.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
.setRequestHeader "Content-type", "application/json"
.setRequestHeader "data-type", "json"
.send
.WaitForResponse
sResult = .ResponseText
End With
Debug.Print (sResult)
If I run the application locally (ie. debug on the computer that is running Excel), it works through Chrome (accessing localhost:xxxxx/api/name returns an xml file with my ActiveDirectory username).
The VBA routine works fine as well, and the Output window displays the XML I get in Chrome.
Now, if I publish the project to the IIS server, it still works through Chrome (accessing myserver/api/name returns an xml file with my ActiveDirectory username).
However, when I run the VBA module, it returns an Error 401:
Error:401 - Unauthorized: Access is denied due to invalid credentials.
The fact that it works in browsers leads me to believe that server-side configuration is OK, and that I need to tweak something in my VBA.
I have to admit that I am a bit clueless at this point...
Thank you for any leads you may give me :)
Thanks to #SWa comment, I solved this with a minor tweak to the function: Switching to WinHttpRequest and using setAutoLogonPolicy 0
Set objHTTP = CreateObject("WinHttp.WinHttpRequest.5.1")
With objHTTP
.Open "GET", URL, False
.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
.setRequestHeader "Content-type", "application/json"
.setRequestHeader "data-type", "json"
.setAutoLogonPolicy 0
.send
.WaitForResponse
sResult = .ResponseText
End With
Debug.Print (sResult)
Previously i was able to download YouTube videos as mp3 via youtube-mp3.org Using this method:
http://www.youtube-mp3.org/api/pushItem/?item=http%3A//www.youtube.com/watch%3Fv%3D<VIDEOID>&xy=_
Then it returned the video id and they started converting the video on their servers. Then this request would return a JSON string with info about the video and the current conversion status:
http://www.youtube-mp3.org/api/itemInfo/?video_id=<VIDEOID>&adloc=
After repeating the request until the value for status is 'serving' I then started the last request by taking the value for key h from the JSON response from the previous request, and this would download a the mp3 file.
http://www.youtube-mp3.org/get?video_id=<VIDEOID>&h=<JSON string value for h>
Now the first request always returns nothing. The second and third requests only succeed if the requested video is cached on their servers (like popular music videos). If thats not the case then the second request would return nil and so the 3rd request can't be started because of the missing hvalue from the second request. Could anybody help me with getting the website to start a conversion something needs to be wrong with the first URL i just dont know what. Thanks
I just tested it. For the first request, you need to send with it a header of:
Accept-Location: *
Otherwise, it will return a 500 (Internal Server Error). But with that header, it will return a string of the youtube video id, and you can use the 2nd api for checking the progress.
Here's the C# code I used for testing:
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create("FIRST_API_URL");
wr.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7";
wr.Headers.Add("Accept-Location", "*");
string res = (new StreamReader(wr.GetResponse().GetResponseStream())).ReadToEnd();
Btw, you can keep track of the headers in the browser's Network (Chrome) debug tab.
Regards
Is there a way to get the following data from the Application_Error event in the Global.ascx file?
action error came from,
ipaddress error came from,
browser error came from,
browser version error came from,
hostName error came from
??
All that information is contained in the Context.Request property.
Context.Request.Url; // /controller/action?foo=bar so up to you to extract the action
Context.Request.UserHostAddress; // 123.456.789.0123
Context.Request.UserAgent // Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13
And once you are sick of parsing all this crap manually and repeating this code all over again among all your applications you might consider using ELMAH.
hostName error came from
Not sure what you mean here. Isn't that the IP of the client?