I do a post for authentication using TIdHttp normally, but when the username has a special character it does not work, analyzing the POST by the browser I see the conversion, for example the username "aforça" in the browser stays "afor%E7a", even sending me what the browser sends, ie "afor% E7a" also does not work.
I've tried the following:
var
Params2: TStringList;
begin
Params2 := TStringList.Create;
Params2.Add('username=' + UTF8String('aforça'));
Params2.Add('password=' + UTF8String('123456'));
xHtmlRetorno := IdHttp.Post('urlsite.com', Params2);
end;
does not work.
searching how to do, I found the following:
var
Params3 : TIdMultipartFormDataStream;
begin
Params3 := TIdMultipartFormDataStream.Create;
Params3.AddFormField('username', UTF8Encode('aforça'), 'utf-8').ContentTransfer := '8bit';
Params3.AddFormField('password', UTF8Encode('123456'), 'utf-8').ContentTransfer := '8bit';
xHtmlRetorno := IdHttp.Post('urlsite.com', Params3);
end;
which also did not work ...
Even though I send as the browser sends, it does not work.
var
Params2: TStringList;
begin
Params2 := TStringList.Create;
Params2.Add('username=' + UTF8String('afor%E7a'));
Params2.Add('password=' + UTF8String('123456'));
xHtmlRetorno := IdHttp.Post('urlsite.com', Params2);
end;
how can I do this, since I can only authenticate with user that can not special characters.
Thanks!
Related
I subscribe to a secure https web page containing a button that downloads some data as csv. I am trying to automate the download without the 'save as' dialog appearing but always seem to get an empty file downloaded. I suspect it has something to do with file type I'm using with IdHttp as most of my code works correctly.
Please can anyone help with my use of IdHttp or see where else I am going wrong?
The download button on the site calls some javascript to perform the download as follows
<a class="dlCSV" href="javascript:void(0);" onclick="dl_module.DownloadCsv();return false;">Download in CSV format…</a>
In Delphi I use a TWeb browser to log on securely and navigate to the page.
Clicking the download button in the TwebBrowser by hand shows the 'save as' dialog and then correctly downloads the csv data, defaulting to the filename 'data.csv'.
Automating clicking the button using execScript (below) also works, again showing the 'save as' dialog and correctly downloading the data with the same default filename.
procedure TForm1.BtnClickDownloadbuttonClick(Sender: TObject);
var TheDocument : IHTMLDocument2; // current HTML document
HTMLWindow: IHTMLWindow2; // parent window of current HTML document
begin
TheDocument := WebBrowser1.Document as IHTMLDocument2; // Get reference to current document
if not Assigned(TheDocument) then
Exit;
HTMLWindow := TheDocument.parentWindow; // Get parent window of current document
if Assigned(HTMLWindow) then
try
HTMLWindow.execScript('dl_module.DownloadCsv()', 'JavaScript'); // execute JS function to do download
except
on E : Exception do
begin
showmessage ('Exception class name = '+E.ClassName+ slinebreak
+ 'Exception message = '+E.Message);
end //on E
end;
end;
Then I added TLama's code from here How do I keep an embedded browser from prompting where to save a downloaded file? to use IDownloadManager to intercept the download and prevent the 'save as' dialog. This is where it seems to go wrong as I then get an empty file downloaded, and not with the name data.csv.
My code for function TWebBrowser.Download, TWebBrowser.InvokeEvent, function TWebBrowser.QueryService and TForm1.FormCreate are identical to that provided by TLama in the link above.
My procedure TForm1.Button1Click is the same except that I changed the download function being called to the one on my page by changing the line
HTMLWindow.execScript('SRT_stocFund.Export()', 'JavaScript');
to
HTMLWindow.execScript('dl_module.DownloadCsv()', 'JavaScript');
and my procedure TForm1.BeforeFileDownload is identical except that because I'm on a secure site I added the variable
var
LHandler: TIdSSLIOHandlerSocketOpenSSL; //<< on a https site
and after creating the Filestream I added the lines
LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
IdHTTP.IOHandler := LHandler;
The issue seems to be in procedure TForm1.BeforeFileDownload where I note that the value of FileSource is
https://www.the_web_site_name/Ashx/GenericCSV.ashx.
There is a short delay while IdHTTP.Get(FileSource, FileStream); executes and then a file is created on my hard disc but called 'GenericCSV.ashx' (not data.csv) and the file is zero bytes long and completely empty.
Any ideas why its not downloading the file called data.csv (Do I somehow have to execute GenericCSV.ashx as well? if so how?)
For info here is my version of procedure TForm1.BeforeFileDownload
procedure TForm1.BeforeFileDownload(Sender: TObject; const FileSource: WideString; var Allowed: Boolean);
var
IdHTTP: TIdHTTP;
FileTarget: string;
FileStream: TMemoryStream;
LHandler: TIdSSLIOHandlerSocketOpenSSL; // added as its a https site
begin
FileSourceEdit.Text := FileSource;
Allowed := ShowDialogCheckBox.Checked;
if not Allowed then
try
IdHTTP := TIdHTTP.Create(nil);
try
FileStream := TMemoryStream.Create;
LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil); //<<< added as its a https site
IdHTTP.IOHandler := LHandler; //<<< added as its a https site
try
IdHTTP.HandleRedirects := True;
IdHTTP.Get(FileSource, FileStream);
FileTarget := IdHTTP.URL.Document;
if FileTarget = '' then
FileTarget := 'File';
FileTarget := ExtractFilePath(ParamStr(0)) + FileTarget;
FileStream.SaveToFile(FileTarget);
finally
FileStream.Free;
end;
finally
IdHTTP.Free;
end;
ShowMessage('Downloading finished! File has been saved as:' + sLineBreak +
FileTarget);
except
on E: Exception do
ShowMessage(E.Message);
end;
end;
After you login, you can use this code to retrieve cookies from TWebBrowser
procedure GetHttpOnlyCookie(const AUrl: string; var ACookies: string);
const
INTERNET_COOKIE_HTTPONLY = 8192;
var
i: Integer;
hModule: THandle;
InternetGetCookieEx: function(lpszUrl, lpszCookieName, lpszCookieData
: PAnsiChar; var lpdwSize: DWORD; dwFlags: DWORD; lpReserved: pointer)
: BOOL; stdCall;
CookieSize: DWORD;
CookieData: PAnsiChar;
begin
LoadLibrary('wininet.dll');
hModule := GetModuleHandle('wininet.dll');
if (hModule <> 0) then
begin
#InternetGetCookieEx := GetProcAddress(hModule, 'InternetGetCookieExA');
if (#InternetGetCookieEx <> nil) then
begin
CookieSize := 1024;
Cookiedata := AllocMem(CookieSize);
try
if InternetGetCookieEx(PAnsiChar(AUrl), nil, Cookiedata, CookieSize, INTERNET_COOKIE_HTTPONLY, nil) then
begin
ACookies:=CookieData;
end;
finally
FreeMem(Cookiedata);
end;
end;
end;
end;
Then you just parse your cookies and add them (you have to create CookieManager in IdHTTP first)
IdHTTP1.CookieManager.AddServerCookie();
Then you start your download and it should work if you passed all parameters correctly (unfortunately, it is not possible to find out what your site requires).
Thank you smooty86 but I think its time I gave up trying to doing it this way and simply parse the page I can see.
I don't mind trying to understand code and adapting it to my needs but its so much harder trying to follow hints and suggestions when I'm working in the dark and especially don't know what parameters are needed everywhere. (I'm not daft, I've been programming for nearly 30 years and have spent over 4 years developing this particular data processing application but rarely touch web stuff)
However, the progress so far is...
Running your GetHttpOnlyCookie code after a successful login using automated filling in of the fields and clicking the login button returned an empty string so I used this code instead that at least seemed to return something that looked a little similar to your cookie string, ie seveveral strings separated by semicolons, most being name=value. (IdCookieManager1 is connected to IdHttp)
CookieList := Tstringlist.Create ;
try
CookieList.Delimiter := ';' ;
document := WebBrowser1.Document as IHTMLDocument2;
CookieList.DelimitedText := document.cookie;
for i := 0 to CookieList.Count-1 do
IdCookieManager1.AddCookie(CookieList[i],LOGIN_URL)
finally
CookieList.Free;
end;
Then in my original procedure BeforeFileDownload I try to log IdHttp into the site as well using code I adapted from here Log in to website from Delphi and the the cookies held in the cookie manager.
Displaying the string returned showed lots of HTML that appeared to represent the oringinal log in page and not the page you see after log in
procedure TFrmInportGrades.BeforeFileDownload(Sender: TObject; const FileSource: WideString; var Allowed: Boolean);
var
FileTarget: string;
FileStream: TMemoryStream;
request : Tstringlist;
s : string;
begin
FileSourceEdit.Text := FileSource;
Allowed := ShowDialogCheckBox.Checked;
if not Allowed then
begin
try
FileStream := TMemoryStream.Create;
IdHTTP.CookieManager := IdCookieManager1;
s := LogInIdHttp; //<<<< log in the IdHttp
showmessage(s); //<<<< debug
IdHTTP.Get(FileSource, FileStream);
FileTarget := IdHTTP.URL.Document;
if FileTarget = '' then
FileTarget := 'File';
FileTarget := ExtractFilePath(ParamStr(0)) + FileTarget;
FileStream.SaveToFile(FileTarget);
finally
FileStream.Free;
end;
ShowMessage('Downloading finished! File has been saved as:' + sLineBreak +
FileTarget);
end;
end;
The login code I used is below but I don't really know what I am doing here or what needs to be put into the Request.Add() parameters. I used 'Inspect element' from firefox to get the name of the user and password boxes and put the correct users name and password after the '=' sign in lines {3} and {4}. In lines {2},{6} and {7} I put the url of the log in site. I've no idea what lines {1}, {2}, {5} do or even if they are correct or necessary
function TFrmInportGrades.LogInIdHttp: string;
var
Request: TStringList;
Response: TMemoryStream;
LHandler: TIdSSLIOHandlerSocketOpenSSL; // added as its a https site
begin
Result := '';
try
Response := TMemoryStream.Create;
try
Request := TStringList.Create;
try
{1} Request.Add('op=login');
{2} Request.Add('redirect=https://www.thewebsite.com/Login.aspx' );
{3} Request.Add('ctl00$ctl00$Body$Body$loginManager$ctl00$loginEmailInput=usernme');
{4} Request.Add('ctl00$ctl00$Body$Body$loginManager$ctl01$passwordInput=password'});
LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil); //<<< added as its a https site
IdHTTP.IOHandler := LHandler; //<<< added as its a https site
IdHTTP.AllowCookies := True;
IdHTTP.HandleRedirects := True;
{5} IdHTTP.Request.ContentType := 'application/x-www-form-urlencoded';
{6} IdHTTP.Post('https://www.thewebsite.com/Login.aspx', Request, Response);
{7} Result := IdHTTP.Get('https://www.thewebsite.com/Login.aspx');
finally
Request.Free;
end;
finally
Response.Free;
end;
except
on E: Exception do
ShowMessage(E.Message);
end;
end;
The net result of all this is that I don't get a file created at all now, not even a zero byte one. This all seems very overcomplicated simply to avoid or automate the 'Save As' dialog and is requiring lots of code that I won't be able to maintan afterwards. Unless somebody has a simpler solution I'll just parse what I can see (BTW I tried TEmbeddedWebBrowser but there is so little documentation for it I couldn't see how to make it download correctly. Might try again later.) Thank you for trying to help!
I'm trying to log in to Gmail (not the email) through Indy component using Delphi XE5,
Using this function:
procedure TForm1.Button1Click(Sender: TObject);
var
http : TIdHTTP;
S, GALX, Email, Pass : String;
lParam : TStringList;
begin
try
lParam := TStringList.Create;
try
http := TIdHTTP.Create(nil);
http.IOHandler := IOHandler;
http.CookieManager := Cookie;
http.AllowCookies := true;
http.HandleRedirects := true;
http.Request.UserAgent := 'Mozilla/5.0 (Windows NT 6.2; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0';
http.Request.Host := 'accounts.google.com';
http.Request.Accept := 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
http.Request.ContentType := 'application/x-www-form-urlencoded';
S := http.Get('https://accounts.google.com/ServiceLogin');
Delete(S, 1, Pos('GALX', S));
S := Copy(S, 1, Pos('">', S) - 1);
Delete(S, 1, Pos('value=', S) + length('value='));
GALX := S;
lParam.Add('GALX='+GALX);
lParam.Add('Email='+Email);
lParam.Add('Passwd='+Pass);
Memo1.Lines.Add(http.Post('http://accounts.google.com/ServiceLoginAuth', lParam));
finally
http.Free;
end;
finally
lParam.Free;
end;
end;
Now whenever i try to execute that i get:HTTP/1.0.405 Method Not Allowed.
and i only get this error when the email/pass are right, when the email/pass is wrong i get the usual error page, so i'm guessing it's not the POST Method that is not allowed.
What am i doing wrong here?
You are not submitting all of the input fields that /ServiceLoginAuth looks for. If you look at the HTML for /ServiceLogin, there are 8 other fields posted to /ServiceLoginAuth besides the 3 that you are already sending. When submitting data from an HTML form, you have to submit everything the HTML form wants to submit, you can't just pick and choose what you want. Try adding the other fields and see what happens.
You need to provide the /ServiceLogin URL in the TIdHTTP.Request.Referer property when posting to /ServiceLoginAuth so it thinks that the request is coming from /ServiceLogin.
You are retrieving /ServiceLogin using HTTPS, but you are posting to /ServiceLoginAuth using HTTP instead. You need to use HTTPS.
When the user has multiple Google accounts, /ServiceLogin posts to /AccountChooser, which then redirects back to /ServiceLogin with additional input parameters, so you might need to take that into account as well.
Posting to /ServiceLoginAuth redirects to /CheckCookie, which then redirects to /ManageAccount, so make sure those requests are complete and accurate at each step.
Here is my Issue. I'm using Delphi 5 and Indy 9. I have no option to upgrade either at the moment. I am trying to send an email via gmail, and i hard code the string as 'smtp.google.com' it works just fine. however, if smtp.host is getting the host name from a variable it fails with error 11001, and i havent been able to figure out why. I'm new using indy so i'm probably missing something silly, but i don't understand why it could accept a string as the host, but not a variable holding the string. (It's got to be a variable because i need to pass the procedure different SMTP hosts based on the user signed in. Here is the code:
procedure TFormEmail.SendSimpleMail(ToEmail, MyAttachment: string);
var
Msg: TIdMessage;
DestAddr: TIdEmailAddressItem;
SMTP: TIdSMTP;
SSLHandler : TidSSLIOHandlerSocket;
Attachment: TIdAttachment;
SMTPHost1 :string;
begin
Msg := idMessage1;
Msg.From.Text := EmailName(My_User);
Msg.From.Address := EmailAddress(My_User);
msg.Subject := 'Test';//email subject
DestAddr := Msg.Recipients.Add;
DestAddr.Text := '';//receiver's name (optional)
DestAddr.Address := ToEmail;//where its going
Msg.Body.Add(edtEmailBody.text); //email body
SMTP := IdSMTP1;
SMTP.IOHandler := idSSLIOHandlerSocket1;
SMTPhost1 := SMTPHost(My_User);
SMTPhost1 := 'smtp.google.com';
//SMTP.Host := SMTPhost1; //<--FAILS
SMTP.Host := 'smtp.google.com'; //<--SUCCEEDS
SMTP.Port := SMTPPort(My_User);
SMTP.AuthenticationType := atLogin; //error here (2 error)
SMTP.Username := EmailAddress(My_User);
SMTP.Password := SMTPPassword(My_User);
If not empty(MyAttachment) then
Attachment := TIdAttachment.Create(Msg.MessageParts, MyAttachment);//loads Att
Try
SMTP.Connect;
except
SMTP.Connect;//refire if exception (issue with INDY)
end;
if useSSL(My_User) then
SMTP.SendCmd('STARTTLS');//load TLS
SMTP.Authenticate;
SMTP.Send(Msg);
SMTP.Disconnect;//disconnect from server
end;
I marked the one that fails and the one that succeeds, but i don't understand what i'm doing wrong. Any help would be appreciated
Seems you have more issues in one question, I can only help you with one.
I had the same issue with Connect, I simply called the Load method from IdSSLOpenSSLHeaders.
Try the following:
SMTP := IdSMTP1;
IdSSLOpenSSLHeaders.Load;
SMTP.IOHandler := idSSLIOHandlerSocket1;
I know there's alot of Indy threads but I can't get one to match my case.
I have been given a URL with a username and password form. this then actions to a URL/reports.php on which there are multiple hyperlinks.
Each of these links will direct to a page with URL variables e.g. reports.php?report=variablename where a download will immediately start.
My thinking so far:
procedure TForm1.PostData(Sender: TObject);
var
paramList:TStringList;
url,text:string;
// IdHTTP1: TIdHTTP;
IdSSLIOHandlerSocket1: TIdSSLIOHandlerSocket;
idLogFile1 : TidLogFile;
begin
idLogFile1 := TidLogFile.Create(nil);
with idLogFile1 do
begin
idLogFile1.Filename := 'C:\HTTPSlogfile.txt';
idLogFile1.active := True;
end;
IdHTTP1 := TIdHTTP.Create(nil);
IdSSLIOHandlerSocket1 := TIdSSLIOHandlerSocket.Create(nil);
IdSSLIOHandlerSocket1.SSLOptions.Method := sslvSSLv23;
IdHTTP1.IOHandler := IdSSLIOHandlerSocket1;
IdHTTP1.HandleRedirects := true;
IdHTTP1.ReadTimeout := 5000;
IdHTTP1.Intercept := idLogFile1;
paramList:=TStringList.create;
paramList.Clear;
paramList.Add('loguser=testuser');
paramList.Add('logpass=duke7aunt');
paramList.Add('logclub=8005');
url := 'https://www.dfcdata.co.uk/integration/reports.php?report=live';
try
IdHTTP1.Post(url,paramList);
except
on E:Exception do
begin
showMessage('failed to post to: '+url);
ShowMessage('Exception message = '+E.Message);
end;
end;
end;
procedure TForm1.Button1Click(Sender: TObject);
var
reportType : String;
begin
PostData(Self);
reportType := 'live';
GetUrlToFile('',reportType+'.csv');
end;
procedure TForm1.GetUrlToFile(AURL, AFile : String);
var
Output : TMemoryStream;
success : Boolean;
begin
success := True;
Output := TMemoryStream.Create;
try
try
IdHTTP1.Get(AURL, Output);
IdHTTP1.Disconnect;
except
on E : Exception do
begin
ShowMessage('Get failed to GET from '+IdHTTP1.GetNamePath +'. Exception message = '+E.Message);
success := False;
end;
end;
if success = True then
begin
showMessage('Filed saved');
Output.SaveToFile(AFile);
end;
finally
Output.Free;
end;
end;
On each try I get "IOHandler is not valid" error. Obviously I'm not posting correctly to the initial page but can anyone advise me on what I'm missing? Also can I simply then hit the download URL after login or will I have to use cookies?
Thanks
There are several bugs in your code:
1) PostData() is requesting an HTTPS URL, but it is not assigning an SSL-enabled IOHandler to the TIdHTTP.IOHandler property. You need to do so.
2) Button1Click() is passing a URL to GetUrlToFile() that does not specify any protocol, so TIdHTTP will end up treating that URL as relative to its existing URL, and thus try to download from https://www.testurl.com/test/testurl.com/test/reports.phpinstead of https://testurl.com/test/reports.php. If you want to request a relative URL, don't include the hostname (or even the path in this case, since you are sending multiple requests to the same path, just different documents).
3) you are leaking the TIdHTTP object.
Issue 1) has now been resolved in another post:
Delphi 5 Indy/ics SSL workaround?
However I would greatly appreciate help on the rest, as follows.
Would I need to make a GET call with the same IdHTTP object and additional URL variable? or should I create a new IdHTTP object?
Would I need to record the session using cookies or can all of this be done with the same call?
Is the GET call above actually what I need to save a csv to file? I may also choose to handle it directly as the data will need importing anyway.
Currently the code gets the error: EIdHTTPProtocolException
This is the continue of my previous question:
Delphi, WebBrowser, Google Login, FusionTable
But the test with WinHTTP also failed as TWebBrowser based test...
And this is one question as you wish... :-)
I have one table what is NOW PUBLIC, but when we will buy non-free account it will be changed to private kind.
I created a simple WinHTTP test, but this also failed.
I can login, I got the "Auth" tag, but the next "private" request returns 401 error.
procedure TForm1.BitBtn1Click(Sender: TObject);
var
WinHttpReq, temp : variant;
URL, s : String;
params : TStringList;
authtoken, query, posts : string;
begin
URL := 'https://www.google.com/accounts/ClientLogin';
WinHttpReq := CreateOleObject('WinHttp.WinHttpRequest.5.1');
params := TStringList.Create;
try
params.Values['accountType'] := 'GOOGLE';
params.Values['Email'] := csEmail;
params.Values['Passwd'] := csPwd;
params.Values['service'] := 'fusiontables';
params.Values['source'] := csSource;
posts := EncodeParamsToURL(params);
finally
params.Free;
end;
URL := URL + '?' + posts;
WinHttpReq.Open('POST', URL, false);
WinHttpReq.Send();
s := WinHttpReq.ResponseText;
Memo1.Lines.Text := s;
params := TStringList.Create;
try
params.Text := s;
authtoken := params.Values['Auth'];
Edit1.Text := authtoken;
finally
params.Free;
end;
//query := URLEncode('SHOW TABLES');
query := URLEncode('select * from 1236965');
url := 'http://www.google.com/fusiontables/api/query?sql=' + query;
WinHttpReq.Open('POST', URL, false);
WinHttpReq.setRequestHeader('Authorization', 'GoogleLogin auth="' + authToken + '"');
WinHttpReq.Send();
s := WinHttpReq.ResponseText;
Memo1.Lines.Text := s;
end;
When I made "select", I got the rows.
But when I want to see the tablenames, I get 401 error...
I'm not sure what cause this error.
a. The free account don't have enough rights to access it privately
b. I set the header wrong
c. I set the csSource wrong (I set it "MyCompanyName-Test-1.0")
d. Other thing I don't know what...
Can anybody help me how to login and access the data successfully?
Change your method to GET:
WinHttpReq.Open('GET', URL, false);
And remove the " around your auth token:
WinHttpReq.setRequestHeader('Authorization', 'GoogleLogin auth=' + authToken);
The first change is to comply with the documentation (although POST also works). The second change fixes the error.