I am trying to read the contents of a web page using a Groovy script. The page contains the readings from one of my temperature sensors that I want to save regularly. I have tried the simplest variant:
def url = "https://measurements.mobile-alerts.eu/Home/MeasurementDetails?deviceid=021B5594EAB5&vendorid=60122a8b-b343-49cb-918b-ad2cdd6dff16&appbundle=eu.mobile_alerts.mobilealerts&fromepoch=1674432000&toepoch=1674518400&from=23.01.2023%2000:00&to=24.01.2023%2000:00&command=refresh"
def res = url.toURL().getText()
println( res)
The result is:
Caught: java.io.IOException: Server returned HTTP response code: 403 for URL: (my url)
In any browser, this URL works without problems.
I would be very grateful for any tips on how to solve this problem.
HTTP code 403 means that a client is forbidden from accessing a valid URL. In other words, the server knows that you are not making a request via a web browser. To bypass this restriction, you need to specify a User-Agent in the request header.
For example:
def url = 'https://measurements.mobile-alerts.eu/Home/MeasurementDetails?deviceid=021B5594EAB5&vendorid=60122a8b-b343-49cb-918b-ad2cdd6dff16&appbundle=eu.mobile_alerts.mobilealerts&fromepoch=1674432000&toepoch=1674518400&from=23.01.2023%2000:00&to=24.01.2023%2000:00&command=refresh'
def res = url.toURL().getText(requestProperties:
['User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0'])
println res
You can switch to other valid user-agent values.
I've been using Radview's Webload IDE tool for a couple of test simulation projects and it has worked well. But for this one scenario where I have a client web session for a login a screen, it would always fail with a 500 Response error for a particular HTTP post as the page loads.
When I try the scenario to load the page manually with a browser it works fine with no issues.
During the recording I would set clear browser cache and cookies and no luck. And I've also tried out many configuration combinations from the "Recording and Script Generatinon Options: Post Data" settings.
/***** WLIDE - URL : http://192.168.2.2/ - ID:2 *****/
wlGlobals.GetFrames = false
wlGlobals.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko"
wlHttp.Get("http://192.168.2.2/")
// END WLIDE
/***** WLIDE - URL : http://192.168.2.2/Api.ashx?c=Images&action=GetSettings - ID:3 *****/
wlHttp.Header["Referer"] = "http://192.168.2.2/"
wlHttp.FormdataEncodingType = 1
wlHttp.ContentType = "application/x-www-form-urlencoded"
wlHttp.FormData["c"] = "Images"
wlHttp.FormData["action"] = "GetSettings"
wlHttp.Post("http://192.168.2.2/Api.ashx"+"?c=Images&action=GetSettings")
// END WLIDE
Anybody with experience with Radview's Webload can give me some suggestions?
I noticed that commenting out the formdata "c" and "actions" lines works. but later I notice a similar error which requires a sessionID in the URL so I'm not sure if I can comment out the formdata "sessionID" line.
To run the API from Webload you need to specify the authorization if its secured.
Using wlHttp.FormData is not the same as adding a parameter to the URL for a POST request.
FormData will be send as part of the post-data request body, while adding it to the URL will send it as a query string - your sever probably expects one form but not the other.
Contact RadView support if you can't get it to work and they'll help you
I have a MVC3 web application, I followed the instruction is below link to make my site mobile friendly,
http://www.hanselman.com/blog/ABetterASPNETMVCMobileDeviceCapabilitiesViewEngine.aspx
everything is fine on my local IIS and it works when I connect to my local site via Mobile device,
I uploaded the site to the production server but it doesn't work, I traced the site and I figured out that on production server someController.HttpContext.Request.Browser.IsMobileDevice return false when I brows even with the mobile device. The mobile device is the same as in local tests.
Can everyone give me a clue?
Edit 1
someController.HttpContext.Request.UserAgent is User-Agent:Mozilla/5.0 (Linux; U; Android 4.1.2; en-us; GT-N7000 Build/JZO54K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
Edit 2
I am using .net4 and I can access to production server via WebSitePanel I don't have direct access to server.
I found the solution which doesn't need to change the server
var isMobile= false;
string u = Request.UserAgent;//ServerVariables["HTTP_USER_AGENT"];
Regex b = new Regex(#"(android|bb\d+|meego).+mobile|avantgo|bada\/|blackberry|blazer|compal|elaine|fennec|hiptop|iemobile|ip(hone|od)|iris|kindle|lge |maemo|midp|mmp|mobile.+firefox|netfront|opera m(ob|in)i|palm( os)?|phone|p(ixi|re)\/|plucker|pocket|psp|series(4|6)0|symbian|treo|up\.(browser|link)|vodafone|wap|windows ce|xda|xiino", RegexOptions.IgnoreCase | RegexOptions.Multiline);
Regex v = new Regex(#"1207|6310|6590|3gso|4thp|50[1-6]i|770s|802s|a wa|abac|ac(er|oo|s\-)|ai(ko|rn)|al(av|ca|co)|amoi|an(ex|ny|yw)|aptu|ar(ch|go)|as(te|us)|attw|au(di|\-m|r |s )|avan|be(ck|ll|nq)|bi(lb|rd)|bl(ac|az)|br(e|v)w|bumb|bw\-(n|u)|c55\/|capi|ccwa|cdm\-|cell|chtm|cldc|cmd\-|co(mp|nd)|craw|da(it|ll|ng)|dbte|dc\-s|devi|dica|dmob|do(c|p)o|ds(12|\-d)|el(49|ai)|em(l2|ul)|er(ic|k0)|esl8|ez([4-7]0|os|wa|ze)|fetc|fly(\-|_)|g1 u|g560|gene|gf\-5|g\-mo|go(\.w|od)|gr(ad|un)|haie|hcit|hd\-(m|p|t)|hei\-|hi(pt|ta)|hp( i|ip)|hs\-c|ht(c(\-| |_|a|g|p|s|t)|tp)|hu(aw|tc)|i\-(20|go|ma)|i230|iac( |\-|\/)|ibro|idea|ig01|ikom|im1k|inno|ipaq|iris|ja(t|v)a|jbro|jemu|jigs|kddi|keji|kgt( |\/)|klon|kpt |kwc\-|kyo(c|k)|le(no|xi)|lg( g|\/(k|l|u)|50|54|\-[a-w])|libw|lynx|m1\-w|m3ga|m50\/|ma(te|ui|xo)|mc(01|21|ca)|m\-cr|me(rc|ri)|mi(o8|oa|ts)|mmef|mo(01|02|bi|de|do|t(\-| |o|v)|zz)|mt(50|p1|v )|mwbp|mywa|n10[0-2]|n20[2-3]|n30(0|2)|n50(0|2|5)|n7(0(0|1)|10)|ne((c|m)\-|on|tf|wf|wg|wt)|nok(6|i)|nzph|o2im|op(ti|wv)|oran|owg1|p800|pan(a|d|t)|pdxg|pg(13|\-([1-8]|c))|phil|pire|pl(ay|uc)|pn\-2|po(ck|rt|se)|prox|psio|pt\-g|qa\-a|qc(07|12|21|32|60|\-[2-7]|i\-)|qtek|r380|r600|raks|rim9|ro(ve|zo)|s55\/|sa(ge|ma|mm|ms|ny|va)|sc(01|h\-|oo|p\-)|sdk\/|se(c(\-|0|1)|47|mc|nd|ri)|sgh\-|shar|sie(\-|m)|sk\-0|sl(45|id)|sm(al|ar|b3|it|t5)|so(ft|ny)|sp(01|h\-|v\-|v )|sy(01|mb)|t2(18|50)|t6(00|10|18)|ta(gt|lk)|tcl\-|tdg\-|tel(i|m)|tim\-|t\-mo|to(pl|sh)|ts(70|m\-|m3|m5)|tx\-9|up(\.b|g1|si)|utst|v400|v750|veri|vi(rg|te)|vk(40|5[0-3]|\-v)|vm40|voda|vulc|vx(52|53|60|61|70|80|81|83|85|98)|w3c(\-| )|webc|whit|wi(g |nc|nw)|wmlb|wonu|x700|yas\-|your|zeto|zte\-", RegexOptions.IgnoreCase | RegexOptions.Multiline);
if ((b.IsMatch(u) || v.IsMatch(u.Substring(0, 4)))) {
isMobile = true;
}
This solution is described here
dear collegues! I need your help.
Fisrt of all, this is NOT ad. I try to make a POST request on website of railway tickets http://booking.uz.gov.ua/en/ to know if there is tickets on current date. BUT... have a problem. I'm using a VBScript to make a request. To know what HTTP Header and POST request to send on website I used Chrome in-build development tools.
Here's my script:
Dim URL
Dim URL2
Dim URL3
Dim sRequest
Dim sCookies
'This is web page where I need to enter information.
URL = "http://booking.uz.gov.ua/en/"
'This is path that Chrome shows to send POST request.
URL2 = "http://booking.uz.gov.ua/en/purchase/search/"
'Optional URL, Chrome shows this link near of URL2. I think this is .js that works on info I enter on web site (URL).
'URL3 = "http://booking.uz.gov.ua/i/js/common.138.js"
'POST request that Chrome shows to send.
sRequest ="station_id_from=2200001&station_id_till=2208001&station_from=Kyiv&station_till=Odesa&date_ dep=09.19.2013&time_dep=00%3A00&search="
'Here I'm using GET request to retrieve Set-Cookie Header (SessionID first of all) to reuse in my second POST request.
sCookies = GetSetHeader(URL)
'Here I'm calling function to make POST request.
Result = HTTPPost(URL2, sRequest)
Function GetSetHeader(URL)
Set objhttp = CreateObject("Microsoft.XmlHttp")
objhttp.open "GET", URL, FALSE
objhttp.Send
'I'm getting only SessionID + other cookies that Chrome shows.
GetSetHeader = Left (objhttp.getResponseHeader("Set-Cookie"), 38) & " " & "HTTPSERVERID=server1; _gv_lang=en; __utma=31515437.675496133.1376934004.1376934004.1376934004.1; __utmb=31515437.2.10.1376934004; __utmc=31515437; __utmz=31515437.1376934004.1.1.utmcsr= (direct)|utmccn=(direct)|utmcmd=(none)"
End Function
Function HTTPPost(URL2, sRequest)
'Header I just took from Chrome.
Set objhttp = CreateObject("Microsoft.XmlHttp")
objHTTP.open "POST", URL2, false
objHTTP.setRequestHeader "Connection", "keep-alive"
objHTTP.setRequestHeader "Host", "booking.uz.gov.ua"
objHTTP.setRequestHeader "Connection", "keep-alive"
objHTTP.setRequestHeader "Content-Length", "Len(Request)"
objHTTP.setRequestHeader "GV-Token", "64214392f178b9f91e3b61a069915cd1"
objHTTP.setRequestHeader "Origin", "http://booking.uz.gov.ua"
objHTTP.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36"
objHTTP.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
objHTTP.setRequestHeader "GV-Unique-Host", "1"
objHTTP.setRequestHeader "GV-Ajax", "1"
objHTTP.setRequestHeader "GV-Screen", "1366x768"
objHTTP.setRequestHeader "GV-Referer", "http://booking.uz.gov.ua/en/"
objHTTP.setRequestHeader "Accept", "*/*"
objHTTP.setRequestHeader "Referer", "http://booking.uz.gov.ua/en/"
objHTTP.setRequestHeader "Accept-Encoding", "gzip,deflate,sdch"
objHTTP.setRequestHeader "Accept-Language", "ru-RU,ru;q=0.8,en- US;q=0.6,en;q=0.4"
'Here I use cookies retrieved with first GET request.
objHTTP.setRequestHeader "Cookie", "sCookies"
objHTTP.send sRequest
'I use this msg to check that right cookies send with POST request.
WScript.Echo sCookies
HTTPPost = objHttp.responseText
'Write answer to TXT file.
Set FSO = CreateObject("Scripting.FileSystemObject")
Set oFile = FSO.OpenTextFile("D:\Results.txt", 2, True)
oFile.Write(objHttp.responseText)
oFile.Close
Set oFile = Nothing
Set FSO = Nothing
end Function
I can't make me script work. I get empty TXT file if I use URL2 to send a request. If I use URL3 - path to script that Chrome show me - to send a request, I just receive a contents of common.138.js in my TXT file. But I expect to receive info in JSON type like shown in Chrome response.
What I noticed,
First, if refresh the website, and try to resend old request, I give me an error:
NetworkError: 400 Bad Request - http://booking.uz.gov.ua/en/purchase/search/"
Maybe because SessionID changed.
Second, I can't to simply write name of station, I need to chose it from drop-down list (When work with this site in UI mode). OR I get an error - Select a departure point from a drop down list.
Third, if try to send request by simply clicking the button on site to search, I get error Status Code:400 Bad Request. I think time of SessionID expired.
There is a working script using InternetExplorer.Application system object, but it is no decision. I want to make it work by sending requests. In future want to try do it on php (as a peart of learning process).
Maybe it's some kind of defence from people like me??? There is a way to make my script work???? Maybe SessionID changes between GET and POST requests?? Or maybe VBScript can't resolve it and I need PHP, for example???
I don't know how to solve this problem. Help me please. Can't sleep. Can't eat. Thanks very very much.
You are getting "400" because you are sending wrong GV-Token header to UZ site.
Eventually GV-token is an md5 of some session-dependent variable (session is identified via _gv_sessid cookie).
This token is obfuscated in JavaScript and resides in page body like,
...
$$_.$_=($$_.$_=$$_+"")[$$_.$_$]+($$_._$=$$_.$_[$$_.__$])+($$_.$$=($$_.$+"")[$$_.__$])+((!$$_)+"")[$$_._$$]+($$_.__=$$_.$_[$$_.$$_])
...
Which evaluates to something like
localStorage.setItem('gv-token',4619709a341b4ffdacce3dafd2f85af3)
and then conducted to all UZ Ajax requests.
So I wish you happy deobfuscating :)) (not for the weak)
PS Also make sure to turn on .NET useUnsafeHeaderParsing via app configuration or reflection.
UPD: As i see, this topic is still alive, so i made up deobfuscation code - it seems like basic regular expressions & string search-and-replace are enough.
Suppose you have UZ start page HTML in pageHTML, then to make things work you need something like (sans validitiy checking), in C#:
Obfuscated code contains some tokens, each evaluates to a hex number from 0 to F, they can be directly replaced. Here is a correspondence dictionary:
var subsitutes = new Dictionary<string, string>
{
{"$$_.$$$", "7"},
{"$$_.$$$$", "f"},
{"$$_.$$$_", "e"},
{"$$_.$$_", "6"},
{"$$_.$$_$", "d"},
{"$$_.$$__", "c"},
{"$$_.$_$", "5"},
{"$$_.$_$$", "b"},
{"$$_.$_$_", "a"},
{"$$_.$__", "4"},
{"$$_.$__$", "9"},
{"$$_.$___", "8"},
{"$$_._$$", "3"},
{"$$_._$_", "2"},
{"$$_.__$" ,"1"},
{"$$_.___", "0"},
};
Then by using regex we get part of obfuscated the code we're interesed in
var scramble = Regex.Match(pageHTML, #"\$\$_\.\$\(\$\$_\.\$\((.*)\)\(\)\)\(\);");
And replace the mentioned above tokens with their real meaning
var keysSorted = subsitutes.Keys.OrderByDescending(key => key.Length);
var halfBakedDeobfuscated = keysSorted.Aggregate(scramble.Groups[1].Value, (current, key) => current.Replace(key, subsitutes[key]));
Almost done, cut out some garbage
var start = Regex.Escape(new string(new[] { '"', '\\', '\\', '\\', '"', ',', '\\', '\\', '"', '+' }) + "4+0+" + new string(new[] { '\"', '\\', '\\', '\\', '\"', '\"', '+' }));
var end = Regex.Escape(new string(new[] { '+', '"', '\\', '\\', '\\', '"', ')' }));
var core = Regex.Match(halfBakedDeobfuscated, start + "(.*)" + end).Groups[1].Value;
Now core contains almost clean version of gvToken, something like 7+0+f+a+7+7+9+8+5+7+e+b+3+3+a+8+3+c+7+8+3+b+d+d+e+f+4+8+7+7+f+7
so last step is to remove these + symbols
var gvToken = string.Join(string.Empty, core.Split('+'));
At last, gvToken contains what you need to give to UZ site - a string like 70fa779857eb33a83c783bddef4877f7.
No JS library and of course no InternetExplorer needed.
You made mistake here:
objHTTP.setRequestHeader "Content-Length", "Len(Request)"
Should be:
objHTTP.setRequestHeader "Content-Length", Len(Request)
Interesting, need to use InternetExplorer.Application, for example list powershell code:
$erroractionpreference = "Continue"
$ie = New-Object -ComObject "InternetExplorer.Application"
$ie.navigate("http://booking.uz.gov.ua/en/")
$ie.visible = $true
sleep 5
while($ie.ReadyState -ne 4) {start-sleep -m 100}
$ie.document.getElementByID("station_id_from").Value = "2200001"
$ie.document.getElementByID("station_id_till").Value = "2208001"
$ie.document.getElementsByName("station_from").Item(1).Value = "Kyiv"
$ie.document.getElementsByName("station_till").Item(1).Value = "Odesa"
$ie.document.getElementByID("date_dep").Value = "12.26.2014"
$ie.document.getElementByID("time_dep").Value = "00:00"
$ie.document.getElementByID("search").Click()
Cookies, including GV-Token, in such case need no to be transferred. I think, there is a way to write without InternetExplorer.Application but emulate browser with your code. Need to explore it.
Is there a way to get the following data from the Application_Error event in the Global.ascx file?
action error came from,
ipaddress error came from,
browser error came from,
browser version error came from,
hostName error came from
??
All that information is contained in the Context.Request property.
Context.Request.Url; // /controller/action?foo=bar so up to you to extract the action
Context.Request.UserHostAddress; // 123.456.789.0123
Context.Request.UserAgent // Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13
And once you are sick of parsing all this crap manually and repeating this code all over again among all your applications you might consider using ELMAH.
hostName error came from
Not sure what you mean here. Isn't that the IP of the client?