Using a cookie with Indy - delphi

I'm trying to get data from a site using the Indy components. (This is in Delphi 7 but happy to use anything that works.)
If you go into a normal browser and put in the path:
http://inventory.data.xyz.com/provide_data.aspx?ID=41100&Mixed=no?fc=true&lang=en
it makes you tick a disclaimer before redirecting you to the actual site. This creates a cookie, which if I look at it in Firefox is like this:
http://inventory.data.xyz.com
Name: ASP.NET_SessionId
Content: vm4l0w033cdng5mevz5bkzzq
Path: /
Send For: Any type of connection
Expires: At end of session
I can't get through the disclaimer part using programming but I thought if I manually sign the disclaimer, I can then enter the details of the cookie into my code and connect directly to the data page. I have tried to do this with the code below but it only returns the html for the disclaimer page which tends to imply it's not using the cookie data I've given it. What am I doing wrong?
procedure TfmMain.GetWebpageData;
var
http: TIdHTTP;
cookie: TIdCookieManager;
sResponse: String;
begin
try
http := TIdHTTP.Create(nil);
http.AllowCookies := True;
http.HandleRedirects := True;
cookie := TIdCookieManager.Create(nil);
cookie.AddCookie('ASP.NET_SessionId=vm4l0w033cdng5mevz5bkzzq', 'inventory.data.xyz.com');
http.CookieManager := cookie;
sResponse := http.Get('http://inventory.data.xyz.com/provide_data.aspx?ID=41100&Mixed=no?fc=true&lang=en');
ShowMessage(sResponse); // returns text of disclaimer
except
end;
end;

Since you have not provided a real URL, I can only speculate, but chances are that either the cookie value you are providing to TIdCookieManager is wrong or outdated by the time TIdHTTP.Get() tries to use it, or more likely TIdCookieManager.AddCookie() is rejecting the cookie outright (if the TIdCookieManager.OnNewCookie event is not triggered, then the cookie was not accepted).

Related

Why does Indy Project HttpClient Get() give code 500 on some URLs which work fine in web browsers?

I have several URLs which work just fine in all browsers, but if I try to get the page content using Get() of the Indy Http client, it returns error code 500, internal server error. This is with the latest Indy SVN build (4981).
Here is my example code. All that is needed for this is Delphi with Indy components and a form with a button and a memo.
procedure TForm1.Button1Click(Sender: TObject);
var HTTPCLIENT1: TIdHTTP;
begin
try
try
HTTPCLIENT1 := TIdHTTP.Create(nil);
Memo1.Clear;
with HTTPCLIENT1 do
begin
HandleRedirects := True;
Request.UserAgent := 'Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.7.3) Gecko/20040924 Epiphany/1.4.4 (Ubuntu)';
Memo1.Text := Get('http://www.laredoute.fr/vente-machine-a-coudre-bernette-20-kit-couture--garantie-2-ans.aspx?productid=401225048&documentid=999999&categoryid=22918417&customertarget=0&offertype=0&prodcolor=1#pos=33_n_n_n_n_n_n&numberpage=2');
Caption := ResponseText;
end;
except
On e: Exception do
begin
Memo1.Lines.Add('Exception: '+e.Message);
end;
end;
finally
HTTPCLIENT1.Free;
end;
end;
It's not a connection problem on my side, since 99% of URLs return 200 or 404, only few return 500, but every browser opens them fine in a second.
That kind of failure usually suggests the GET request is malformed in some way, causing the server code to fail on its end. But without seeing what the webbrowser requests actually look like for comparison to TIdHTTP's requests, there is no way to know for sure what the server is not liking.
Update: what I see happening is that when a webbrowser requests the URL, the server sends back a 200 response immediately, however when TIdHTTP requests the URL, the server sends a 301 redirect to a new URL, which then sends a 302 redirect to an error page when TIdHTTP requests that URL, which then sends the 500 response when TIdHTTP requests that URL.
The two differences between a webbrowser request and the initial TIdHTTP request that would have an effect on a webserver are:
the URL you are requesting with TIdHTTP includes an anchor tag at the end (everything after the # character - #pos=33_n_n_n_n_n_n&numberpage=2) which webbrowsers would normally strip out. Anchors are not actually part of URLs. They are meant for webbrowsers to use when locating spots within data that is retrieved from a URL.
the user agent. Some web servers are sensitive to different user agents, and can send different responses to different types of user agents.
When I remove the anchor from the URL, TIdHTTP.Get() no longer crashes:
Memo1.Text := Get('http://www.laredoute.fr/vente-machine-a-coudre-bernette-20-kit-couture--garantie-2-ans.aspx?productid=401225048&documentid=999999&categoryid=22918417&customertarget=0&offertype=0&prodcolor=1');

IdCookieManager with IdHTTP not sending cookies

I'm logging onto a website, trying to get the cookies running. From what I understand, assigning a IdCookieManager to an IdHTTP and setting AllowCookies:=true should be all that I need to do, right? I succesfully receive a cookie after logging in, but when I try to navigate further, the cookie is not being sent.
Here is the code I have:
procedure TForm1.Login;
var data: TStringList;
begin
data:=TStringList.Create;
try
IdHTTP.Get('http://navratdoreality.cz/'); // Here I receive Set-Cookie
data.Add('ACTION=check_adult');
data.Add('check=18plus');
Memo1.text:=IdHTTP.Post('http://navratdoreality.cz/',data); // Here the
// request contains the cookie and I get a succesfully-logged page in
// response, no more set-cookie
except
ShowMessage('err');
end;
data.Free;
end;
procedure TForm1.Navigate;
var ms:TMemoryStream;
begin
ms:=TMemoryStream.Create;
try
IdHTTP.Get('http://www.navratdoreality.cz/?p=view&id='+Edit1.Text, ms);
// the request doesn't contain any cookies, even though the cookie from
// logging is saved in IdCookieManager
ms.Position:=0;
Memo1.Lines.LoadFromStream(ms, TEncoding.UTF8);
except
ShowMessage('err');
end;
ms.Free;
end;
I don't know what might be the problem. My Indy is 10.5.8.0. If you are going to look at the site, beware, some of it is nsfw.
Thanks
Okay, this was a very stupid question. The problem was that the login part had http://navratdoreality.cz, whereas the next part had http://www.navratdoreality.cz. Both URLs displayed the same, but they were apparently different for the IdCookieManager, so that's why the cookie wasn't sent.

Delphi, WebBrowser, Google Login, FusionTable

I check a possibility to integrate fusiontables into my Delphi TWebBrowser based application.
But I cannot continue my project because I don't understand many things.
I have a public table, I want to access this, upload some rows, update some rows, and show it with fusiontablelayer. I have only "free" account now.
The problems:
1.)
I need to authenticate.
var
posts, s, url : string;
authToken : string;
postdata,
header : OleVariant;
params : TStringList;
i : integer;
begin
header := 'Content-type: application/x-www-form-urlencoded'#13#10;
params := TStringList.Create;
try
params.Values['accountType'] := 'GOOGLE';
params.Values['Email'] := 'any';
params.Values['Passwd'] := 'any';
params.Values['service'] := 'fusiontables';
params.Values['source'] := '?'; // WHAT IS THIS?
posts := EncodeParamsToURL(params);
finally
params.Free;
end;
postdata := VarArrayCreate([0, Length(posts) - 1], varByte);
// Put Post in array
for i := 1 to Length(posts) do
postdata[I - 1] := Ord(posts[I]);
url := 'https://www.google.com/accounts/ClientLogin';
wb.Navigate(url, emptyparam, emptyparam, postdata, header);
while wb.ReadyState <> READYSTATE_COMPLETE do
Application.ProcessMessages;
s := (wb.Document as iHTMLDocument2).body.innerText;
This is only a demo, but it is working.
I don't know what is "SOURCE" parameter, but I got three lines as result, and the last is "Auth=...." that containing the token.
http://code.google.com/intl/hu-HU/apis/fusiontables/docs/samples/apps_script.html
2.)
I need to push this token into header.
When I do this, I got 401 error.
params := TStringList.Create;
try
params.Text := s;
authToken := params.Values['Auth'];
finally
params.Free;
end;
header := 'Authorization : GoogleLogin auth="' + authToken + '"'#13#10;
url := 'http://www.google.com/fusiontables/api/query?select * from 1236944';
wb.Navigate(url, emptyparam, emptyparam, emptyparam, header);
So I'm totally confused now.
First:
Because JavaScript layer don't have authentication interface, I think I need to authenticate the "browser". May this is is wrong idea, but my thinking based on common web login logic, where the login creates a Session, and the Session is identified as a hidden cookie what is valid in this browser.
But may Google login is uses an identifier what passed on every request... I don't know.
So because this I must do an automatic "login" in the browser. (If that is not true then I can use WinInet, or IdHTTP for login, and use only the token in the browser).
Now I don't have idea how to do this login automatically without show the login name/pwd in the html, or show the token result in the TWebBrowser...
Second:
I must modify the data. This may realizable in a transparent component, like idHTTP, and I can show only the changes in the WebBrowser...
Third:
I can show the fusion table with a layer. This is not too hard if I has been authenticated once...
So: I'm confused now, because Google supports only Python/Java as client library, and not Delphi. I need to integrate the authentication and visualization into my TWebBrowser component very transparently.
But there is no good example or source in the net what demonstrate the login + fusiontable manipulation...
Can anybody help me in this question?
Question 1 is answered in section "The ClientLogin interface":
Source:
Short string identifying your application, for logging purposes. This
string should take the form: "companyName-applicationName-versionID".
Question 2:
Your URL is wrong, it has to be:
url := 'http://www.google.com/fusiontables/api/query?sql=select * from 1236944';
See the "sql=" -part? That's important. Have a look here for an example.
Regarding your other questions: they are a bit confusing. I think you don't have to use the TWebBrowser and can use anything that can issue GET and POST requests. For the login part: this information should be provided by your user, because your application should empower your users to work with their data, right?

delphi post 'illegal access' error

I'm making some simple Delphi software using the IdHttp module in Indy. I want to access this page by using IdHttp's post function to open this webpage.
Firstly I have to log in, (at http://user.buddybuddy.co.kr/Login/Login.asp), so I logged in, but after I logged in to the webpage I can see another login page. But when I try to login (at http://user.buddybuddy.co.kr/usercheck/UserCheckPWExec.asp), I encountered an error message, "Illegal access."
If anyone can help me I would appreciate it!
begin
sl.Clear;
sl.Add('ID=ph896011');
sl.add('PWD=pk1089');
sl.add('SECCHK=0');
IdHTTP1.HandleRedirects := True;
IdHTTP1.Request.ContentType := 'application/x-www-form-urlencoded';
memo1.Text:=idhttp1.Post('http://user.buddybuddy.co.kr/Login/Login.asp',sl);
if pos('top.location =',Memo1.Text)> 0 then
begin
application.ProcessMessages;
sleep(1000);
Memo1.Text:= '';
sleep(300);
sl2.Clear;
sl2.Add('PASSWD=pk1089' );
Memo2.Text := IdHTTP1.Post('http://user.buddybuddy.co.kr/usercheck/UserCheckPWExec.asp', sl2);
sleep(300);
Memo1.Text := '';
//memo1.Text := IdHTTP1.Get('https://user.buddybuddy.co.kr/Login/Logout.asp');
//Sleep(1000);
end;
The server almost certainly keeps track of whether you're logged in by returning a cookie with the response from the request for Login.asp. You need to send that cookie back with any subsequent requests. To keep track of cookies, create a TIdCookieManager object and assign it to your HTTP object's CookieManager property. The first request will store the response's cookies there, and the next request on that same HTTP object will send them back, indicating to the server that you have already logged in.

How to Prevent dialog (Basic Authentication prompt) during call of Webservice

In a delphi program (running as a service) i need to call some webservices.
The calls works fine if basic Authentications is not requerired. The calls also works fine if Basic Authentication is requerired and username/password is provided (in BeforePost) using:
InternetSetOption(Data, INTERNET_OPTION_USERNAME,...
InternetSetOption(Data, INTERNET_OPTION_PASSWORD,...
But if Basic Authentication is Requeried, and username/password is not provided, the program brings up af prompt for the username/password (thats a NO-GO in a servcice).
So how can I signal that i DON'T want a prompt, but instead an error?
The problem is, as i can se it, in the SOAPHTTPTrans function THTTPReqResp.Send(const ASrc: TStream): Integer; (line 762 (second call to InternetErrorDlg i that method)).
EDIT1:
if i change the Flags in the beginning of the send method (in SOAPHTTPTRANS) to include INTERNET_FLAG_NO_AUTH, it works as i wanted.
But how do i do that without changing the SAOPHTTPTrans (if possible)?
EDIT2:
ws := THTTPRIO.Create(Self);
ws.URL := 'http://excample.com/ws.asmx';
ws.HTTPWebNode.InvokeOptions := [soIgnoreInvalidCerts];
ws.HTTPWebNode.OnBeforePost := WebServiceCallBeforePost;
AvailabilityWebservice := (ws as AvailabilityServiceSoap);
sTemp := AvailabilityWebservice.GetVersion;
Where AvailabilityServiceSoap is the interface generated using the WSDL importer.
I had this problem when trying to let Windows Live Messenger work through a web filter.
I ended up writing a small program that auto-authenticates every so often.
Hope this helps you too.
uses
... IdHTTP ...;
...
var
httpGetter: TIdHTTP;
...
httpGetter.Request.Username := username;
httpGetter.Request.Password := password;
httpGetter.HandleRedirects := True;
httpGetter.Request.BasicAuthentication := True;
//custom useragent required to let live messenger work
//this part is probably not necessary for your situation
httpGetter.Request.UserAgent := 'MSN Explorer/9.0 (MSN 8.0; TmstmpExt)';
httpGetter.Get(url,MS);
...
You could create a new class which Inherits from THTTPReqResp and override the send method so that you can include your own flags. You should be able to set ws.HTTPWebNode to a new node using the new class.
Something Like
ws := THTTPRIO.Create(Self);
MyNewNode := MyNewClass.Create;
ws.HTTPWebNode := MyNewNode;
ws.URL := 'http://excample.com/ws.asmx';
ws.HTTPWebNode.InvokeOptions := [soIgnoreInvalidCerts];
ws.HTTPWebNode.OnBeforePost := WebServiceCallBeforePost;
AvailabilityWebservice := (ws as AvailabilityServiceSoap);
sTemp := AvailabilityWebservice.GetVersion;
How about checking the servers authentication mode first?
http://en.wikipedia.org/wiki/Basic_access_authentication
The client asks for a page that
requires authentication but does not
provide a user name and password.
Typically this is because the user
simply entered the address or
followed a link to the page.
The server responds with the 401
response code and provides the
authentication realm.
So the client service application could send a Get and see if the response has a header like
WW-Authenticate: Basic realm="Secure Area"

Resources