This may be a very unusual question, but basically there's a page on another domain (that I can view, but can't edit/change) that has a button. When that button is clicked it generates some unique keys.
I need to pull those unique keys with my web service (using ASP .NET MVC3) I can get the initial HTML of the page, but how can I force the page to "click" the button so that I can get the values after the POST?
Normally, I'd reuse the code to generate keys myself, but I don't have access to the logic.
I hope this makes sense.
Use e.g. firebug to see what POST parameters are sent with form and then make the same POST from your code.
For this you can use WebRequest or WebClient.
See this SO questions that will help you how to do it:
HTTP request with post
Send POST request in C# like a web page does?
How to simulate browser HTTP POST request and capture result in C#
Then just parse the response with technology of your choice (I would use regular expressions - Regex, or LinqToXml if the response is well formed XML).
Note: Keep in mind that your code will be dependent on some service you are not maintaining. So you can get in problems when the service is unavailable, discontinued or if the format of POSTed form or response will be changed.
This really depends on the technology on the targeted site.
If the page is a simple HTML form then you can easily send a POST. You will need to send the expected data to the POST. Then you can parse the data.
If its not so straight forward you will need to look into ways to automate the click. Check Selenium. Also you might need to employ scrapping if the results page is a mess.
Related
I have an ingredient form and I want the user to type only the name and then the carb, protein and fat get automatically loaded. I know I need an ajax request but I don't know the path to learn how to accomplish this.
Can anyone gives an example or tell me where I can find it?
Learn Jquery too. Jquery is a widely used Javascript library and you will find a lot of help too.
Coming back to the question, $ajax method will be used to send an ajax request from your .js file. You will find a lot of documentation on it.
If you want to send an ajax request using rails helpers and views use the
remote : true
option. it will process your request as a XmlHttpRequest one. You can check the type of request by naigating to your browsers console and see the type of request. In case of Ajax request it will be XmlHttpRequest type.
This is not a thorough answer since you will find a lot of documentation on internet. But just to get you started.
Im trying to send some data to a form on a site were im a member using cURL, but when i look at the headers being sent, they seem to have been encrypted.
Is there a way i can get around this by making the computer / server visit the site and actual add the data to the inputs on the form and then hit submit, so that it would generate the correct data and post the form ?
You have got a few options:
reverse engineer the JavaScript that does the encryption (or possibly just encoding) process
get a browser engine (e.g. the Gecko engine), and add some scripting to it to fill in the forms and push the submit button - of course you would need JavaScript support within the page itself
parse the HTML using an HTML parser, feed the JavaScript in it to a JavaScript runtime with the correct libraries, fill in the "form" and hit the submit button
It's probably easiest to go for the first option. The JavaScript must be in the open to be able to be executed in the browser. But it may take some time to reverse-engineer as it is likely obfuscated.
You can use a framework to automate user interaction on the web pages, like Selenium.
This would enable you to not bother reverse engineering anything.
Selenium has binding in various languages, including Python and java.
Provided the javascript is visible on the website in question, you should be able to simply copy and paste their encryption routines to prepare the headers exactly as they do
A hacky fix if you can isolate the function that encodes the data you type in the form - is to use something like PyV8 to execute the JS inside python.
Use AutoHotKeyIt and actually have it use the Browser Normally. It can read from files, and do repetitive tasks infinitely. Also you can push a flag to make it only happen within that application, which means you can have it minimized and yet still preform the action.
You seem to be having issues with the problem of them encrypting the headers and such, so why not simply use that too your advantage? Your still pushing the same data in, but now your working around their system. With little to no side effect too you.
I want to get reponse data send by a Jetty Server via POST within a Greasemonkey script.
New data is send every few seconds.
How can I directly access the data, that is displayed in Firebug "Network" tab as response data?
(I tried to get the data with solutions provided here that make use of jQuery and ajaxSuccess, but that did not work. And I am not sure if the POST data is returned from an jQuery ajax request. So is there a way to directly access the POST "stream").
The best way depends highly on details that you've not (yet) provided:
Exactly how does the page send and receive the information. Is it really form posts, or is it AJAX? If AJAX, is jQuery involved? Pages can post without AJAX; are iframes involved?
Link to the target page.
What does the page do with the information? Does it update displays?
What is your ultimate reason for wanting the information? Most of the time, it's easier not to try and intercept -- and parse -- AJAX. Usually, monitoring the DOM tree is much easier and more robust way of getting what you want.
If the page displays the data, you want, to the user, the best way to get it is shown in "How can I detect AJAX node insertion...".
If you really want to intercept the data:
Intercepting jQuery AJAX from Greasemonkey.
Intercepting any target-page AJAX from Greasemonkey.
Intercepting AJAX, bare bones.
If the page is submitting forms, creating iframes, etc., POST DETAILS. But in general, you would replace the javascript function that is receiving (and/or sending) the data.
I think I am mostly struggling with this problem because I do not know what to search for.
I want to make an app that allows the user to enter their gift card number and use that number to login to this website:
https://www.getmybalance.com
I have no idea how to do this without control over the website. Is it even possible to do so?
I don't want to use a UIWebView to show the page.
You should read up on NSURLConnection, you're going to have to execute a post request to login. Then you're going to have to determine whether or not you logged in successfully probably by parsing the returned page. NSURLConnection will handle storing the login cookie the site returns. After you've logged in you're probably going to need to execute another post request to query their system. Once again you will have to probably parse the result out of the HTML page that is returned.
NSURLConnection:
https://developer.apple.com/library/mac/#documentation/Cocoa/Reference/Foundation/Classes/nsurlconnection_Class/Reference/Reference.html
NSURLConnection Delegate Protocol:
https://developer.apple.com/library/mac/#documentation/Foundation/Reference/NSURLConnectionDelegate_Protocol/Reference/Reference.html#//apple_ref/occ/intf/NSURLConnectionDelegate
This all of course assumes that this website doesn't have an API you can use.
Looks like you need to programatically POST in https to the server, then you will get back some DOM document, or JSON, or some weird thing, which you then parse.
POSTing with iOS is pretty easy, look at something like LRResty https://github.com/lukeredpath/LRResty or similar.
When you get the data back, first thing to do is look at it with NSLog. Then if the data is HTML, you will need to wade into the HTML to get the result.
The problem with that approach is that the company hosting the page may change their API at any time. You should ask them to either not ever change anything (if they want to change, then make a new page and leave the old one working, or better, support a simple REST API - which would also help them build nice AJAX/html5 web sites in the future.).
I understand what progressive enhancement is, I'm just fuzzy on some of the details in actually pulling it off. Of course, that could be because I'm looking at it in the wrong way. Let me try to explain my difficulty with a hypothetical:
ASP.NET MVC site. I have a view that has tabbed navigation. Each tab is for a movie category/genre which displays 5-10 links to movies in that category. The movie data is obtained through Netflix's Odata.
My initial thought is to use Ajax to pull and parse the JSON from the proper OData GET requests when each tab is clicked. How would I provide a non-JavaScript version of that? Is it even possible?
For simpler requests where JSON isn't necessary - like, say, having a user log into the system - I see how I could simply set a cookie and dynamically change the page based on it to reflect the change. But what if I need to return and parse JSON? How do I provide an alternative?
The deal with progressive enhancement is that your server side must be fully capable of generating every last bit of HTML that appears in all of your pages. This is obvious, since otherwise (if JS is turned off) there will be no part of your application capable of doing said rendering.
Since the server side must know how to render everything, it doesn't make much sense to generate things (DOM elements/HTML) on the client side from JSON responses the server gives you. Why repeat yourself?
This brings us to the logical conclusion that when doing dynamic updates on the client, you need to get ready-made HTML from the server (since the rendering logic is over there) and insert it into the DOM as appropriate. You are then free to work on the newly inserted elements with jQuery and enhance them all you want.
So -- forget about parsing JSON on the client, otherwise you 're locking yourself out of progressive enhancement. If you want to call a third party, have the server be your intermediary: call the server with all the necessary information for it to call the third party and get ready-made HTML back.
If you do this, then the server can of course provide non-JS versions of everything on your site with no problem. Total non-reliance on JS achieved.
There is no JSON without JS, by definition (JavaScript Object Notation). Without JS you won't make AJAX calls. Your pages will render as is, just like oldschool sites.
If you need to do this progressively, you will have to call the odata service server-side, and provide .net objects to the site in viewdata, or your viewmodel, and have your views/partials render it.
In ASP.Net MVC actions, the httpcontext available via the controller will have a property on this path: this.HttpContext.Request.IsAjaxRequest() and can be used to test whether you want to return a view or just json data, or whatever type of ActionResult you want. This can be an excellent timesaver for building progressive enhancement style sites.