I know there's alot of Indy threads but I can't get one to match my case.
I have been given a URL with a username and password form. this then actions to a URL/reports.php on which there are multiple hyperlinks.
Each of these links will direct to a page with URL variables e.g. reports.php?report=variablename where a download will immediately start.
My thinking so far:
procedure TForm1.PostData(Sender: TObject);
var
paramList:TStringList;
url,text:string;
// IdHTTP1: TIdHTTP;
IdSSLIOHandlerSocket1: TIdSSLIOHandlerSocket;
idLogFile1 : TidLogFile;
begin
idLogFile1 := TidLogFile.Create(nil);
with idLogFile1 do
begin
idLogFile1.Filename := 'C:\HTTPSlogfile.txt';
idLogFile1.active := True;
end;
IdHTTP1 := TIdHTTP.Create(nil);
IdSSLIOHandlerSocket1 := TIdSSLIOHandlerSocket.Create(nil);
IdSSLIOHandlerSocket1.SSLOptions.Method := sslvSSLv23;
IdHTTP1.IOHandler := IdSSLIOHandlerSocket1;
IdHTTP1.HandleRedirects := true;
IdHTTP1.ReadTimeout := 5000;
IdHTTP1.Intercept := idLogFile1;
paramList:=TStringList.create;
paramList.Clear;
paramList.Add('loguser=testuser');
paramList.Add('logpass=duke7aunt');
paramList.Add('logclub=8005');
url := 'https://www.dfcdata.co.uk/integration/reports.php?report=live';
try
IdHTTP1.Post(url,paramList);
except
on E:Exception do
begin
showMessage('failed to post to: '+url);
ShowMessage('Exception message = '+E.Message);
end;
end;
end;
procedure TForm1.Button1Click(Sender: TObject);
var
reportType : String;
begin
PostData(Self);
reportType := 'live';
GetUrlToFile('',reportType+'.csv');
end;
procedure TForm1.GetUrlToFile(AURL, AFile : String);
var
Output : TMemoryStream;
success : Boolean;
begin
success := True;
Output := TMemoryStream.Create;
try
try
IdHTTP1.Get(AURL, Output);
IdHTTP1.Disconnect;
except
on E : Exception do
begin
ShowMessage('Get failed to GET from '+IdHTTP1.GetNamePath +'. Exception message = '+E.Message);
success := False;
end;
end;
if success = True then
begin
showMessage('Filed saved');
Output.SaveToFile(AFile);
end;
finally
Output.Free;
end;
end;
On each try I get "IOHandler is not valid" error. Obviously I'm not posting correctly to the initial page but can anyone advise me on what I'm missing? Also can I simply then hit the download URL after login or will I have to use cookies?
Thanks
There are several bugs in your code:
1) PostData() is requesting an HTTPS URL, but it is not assigning an SSL-enabled IOHandler to the TIdHTTP.IOHandler property. You need to do so.
2) Button1Click() is passing a URL to GetUrlToFile() that does not specify any protocol, so TIdHTTP will end up treating that URL as relative to its existing URL, and thus try to download from https://www.testurl.com/test/testurl.com/test/reports.phpinstead of https://testurl.com/test/reports.php. If you want to request a relative URL, don't include the hostname (or even the path in this case, since you are sending multiple requests to the same path, just different documents).
3) you are leaking the TIdHTTP object.
Issue 1) has now been resolved in another post:
Delphi 5 Indy/ics SSL workaround?
However I would greatly appreciate help on the rest, as follows.
Would I need to make a GET call with the same IdHTTP object and additional URL variable? or should I create a new IdHTTP object?
Would I need to record the session using cookies or can all of this be done with the same call?
Is the GET call above actually what I need to save a csv to file? I may also choose to handle it directly as the data will need importing anyway.
Currently the code gets the error: EIdHTTPProtocolException
Related
I subscribe to a secure https web page containing a button that downloads some data as csv. I am trying to automate the download without the 'save as' dialog appearing but always seem to get an empty file downloaded. I suspect it has something to do with file type I'm using with IdHttp as most of my code works correctly.
Please can anyone help with my use of IdHttp or see where else I am going wrong?
The download button on the site calls some javascript to perform the download as follows
<a class="dlCSV" href="javascript:void(0);" onclick="dl_module.DownloadCsv();return false;">Download in CSV format…</a>
In Delphi I use a TWeb browser to log on securely and navigate to the page.
Clicking the download button in the TwebBrowser by hand shows the 'save as' dialog and then correctly downloads the csv data, defaulting to the filename 'data.csv'.
Automating clicking the button using execScript (below) also works, again showing the 'save as' dialog and correctly downloading the data with the same default filename.
procedure TForm1.BtnClickDownloadbuttonClick(Sender: TObject);
var TheDocument : IHTMLDocument2; // current HTML document
HTMLWindow: IHTMLWindow2; // parent window of current HTML document
begin
TheDocument := WebBrowser1.Document as IHTMLDocument2; // Get reference to current document
if not Assigned(TheDocument) then
Exit;
HTMLWindow := TheDocument.parentWindow; // Get parent window of current document
if Assigned(HTMLWindow) then
try
HTMLWindow.execScript('dl_module.DownloadCsv()', 'JavaScript'); // execute JS function to do download
except
on E : Exception do
begin
showmessage ('Exception class name = '+E.ClassName+ slinebreak
+ 'Exception message = '+E.Message);
end //on E
end;
end;
Then I added TLama's code from here How do I keep an embedded browser from prompting where to save a downloaded file? to use IDownloadManager to intercept the download and prevent the 'save as' dialog. This is where it seems to go wrong as I then get an empty file downloaded, and not with the name data.csv.
My code for function TWebBrowser.Download, TWebBrowser.InvokeEvent, function TWebBrowser.QueryService and TForm1.FormCreate are identical to that provided by TLama in the link above.
My procedure TForm1.Button1Click is the same except that I changed the download function being called to the one on my page by changing the line
HTMLWindow.execScript('SRT_stocFund.Export()', 'JavaScript');
to
HTMLWindow.execScript('dl_module.DownloadCsv()', 'JavaScript');
and my procedure TForm1.BeforeFileDownload is identical except that because I'm on a secure site I added the variable
var
LHandler: TIdSSLIOHandlerSocketOpenSSL; //<< on a https site
and after creating the Filestream I added the lines
LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
IdHTTP.IOHandler := LHandler;
The issue seems to be in procedure TForm1.BeforeFileDownload where I note that the value of FileSource is
https://www.the_web_site_name/Ashx/GenericCSV.ashx.
There is a short delay while IdHTTP.Get(FileSource, FileStream); executes and then a file is created on my hard disc but called 'GenericCSV.ashx' (not data.csv) and the file is zero bytes long and completely empty.
Any ideas why its not downloading the file called data.csv (Do I somehow have to execute GenericCSV.ashx as well? if so how?)
For info here is my version of procedure TForm1.BeforeFileDownload
procedure TForm1.BeforeFileDownload(Sender: TObject; const FileSource: WideString; var Allowed: Boolean);
var
IdHTTP: TIdHTTP;
FileTarget: string;
FileStream: TMemoryStream;
LHandler: TIdSSLIOHandlerSocketOpenSSL; // added as its a https site
begin
FileSourceEdit.Text := FileSource;
Allowed := ShowDialogCheckBox.Checked;
if not Allowed then
try
IdHTTP := TIdHTTP.Create(nil);
try
FileStream := TMemoryStream.Create;
LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil); //<<< added as its a https site
IdHTTP.IOHandler := LHandler; //<<< added as its a https site
try
IdHTTP.HandleRedirects := True;
IdHTTP.Get(FileSource, FileStream);
FileTarget := IdHTTP.URL.Document;
if FileTarget = '' then
FileTarget := 'File';
FileTarget := ExtractFilePath(ParamStr(0)) + FileTarget;
FileStream.SaveToFile(FileTarget);
finally
FileStream.Free;
end;
finally
IdHTTP.Free;
end;
ShowMessage('Downloading finished! File has been saved as:' + sLineBreak +
FileTarget);
except
on E: Exception do
ShowMessage(E.Message);
end;
end;
After you login, you can use this code to retrieve cookies from TWebBrowser
procedure GetHttpOnlyCookie(const AUrl: string; var ACookies: string);
const
INTERNET_COOKIE_HTTPONLY = 8192;
var
i: Integer;
hModule: THandle;
InternetGetCookieEx: function(lpszUrl, lpszCookieName, lpszCookieData
: PAnsiChar; var lpdwSize: DWORD; dwFlags: DWORD; lpReserved: pointer)
: BOOL; stdCall;
CookieSize: DWORD;
CookieData: PAnsiChar;
begin
LoadLibrary('wininet.dll');
hModule := GetModuleHandle('wininet.dll');
if (hModule <> 0) then
begin
#InternetGetCookieEx := GetProcAddress(hModule, 'InternetGetCookieExA');
if (#InternetGetCookieEx <> nil) then
begin
CookieSize := 1024;
Cookiedata := AllocMem(CookieSize);
try
if InternetGetCookieEx(PAnsiChar(AUrl), nil, Cookiedata, CookieSize, INTERNET_COOKIE_HTTPONLY, nil) then
begin
ACookies:=CookieData;
end;
finally
FreeMem(Cookiedata);
end;
end;
end;
end;
Then you just parse your cookies and add them (you have to create CookieManager in IdHTTP first)
IdHTTP1.CookieManager.AddServerCookie();
Then you start your download and it should work if you passed all parameters correctly (unfortunately, it is not possible to find out what your site requires).
Thank you smooty86 but I think its time I gave up trying to doing it this way and simply parse the page I can see.
I don't mind trying to understand code and adapting it to my needs but its so much harder trying to follow hints and suggestions when I'm working in the dark and especially don't know what parameters are needed everywhere. (I'm not daft, I've been programming for nearly 30 years and have spent over 4 years developing this particular data processing application but rarely touch web stuff)
However, the progress so far is...
Running your GetHttpOnlyCookie code after a successful login using automated filling in of the fields and clicking the login button returned an empty string so I used this code instead that at least seemed to return something that looked a little similar to your cookie string, ie seveveral strings separated by semicolons, most being name=value. (IdCookieManager1 is connected to IdHttp)
CookieList := Tstringlist.Create ;
try
CookieList.Delimiter := ';' ;
document := WebBrowser1.Document as IHTMLDocument2;
CookieList.DelimitedText := document.cookie;
for i := 0 to CookieList.Count-1 do
IdCookieManager1.AddCookie(CookieList[i],LOGIN_URL)
finally
CookieList.Free;
end;
Then in my original procedure BeforeFileDownload I try to log IdHttp into the site as well using code I adapted from here Log in to website from Delphi and the the cookies held in the cookie manager.
Displaying the string returned showed lots of HTML that appeared to represent the oringinal log in page and not the page you see after log in
procedure TFrmInportGrades.BeforeFileDownload(Sender: TObject; const FileSource: WideString; var Allowed: Boolean);
var
FileTarget: string;
FileStream: TMemoryStream;
request : Tstringlist;
s : string;
begin
FileSourceEdit.Text := FileSource;
Allowed := ShowDialogCheckBox.Checked;
if not Allowed then
begin
try
FileStream := TMemoryStream.Create;
IdHTTP.CookieManager := IdCookieManager1;
s := LogInIdHttp; //<<<< log in the IdHttp
showmessage(s); //<<<< debug
IdHTTP.Get(FileSource, FileStream);
FileTarget := IdHTTP.URL.Document;
if FileTarget = '' then
FileTarget := 'File';
FileTarget := ExtractFilePath(ParamStr(0)) + FileTarget;
FileStream.SaveToFile(FileTarget);
finally
FileStream.Free;
end;
ShowMessage('Downloading finished! File has been saved as:' + sLineBreak +
FileTarget);
end;
end;
The login code I used is below but I don't really know what I am doing here or what needs to be put into the Request.Add() parameters. I used 'Inspect element' from firefox to get the name of the user and password boxes and put the correct users name and password after the '=' sign in lines {3} and {4}. In lines {2},{6} and {7} I put the url of the log in site. I've no idea what lines {1}, {2}, {5} do or even if they are correct or necessary
function TFrmInportGrades.LogInIdHttp: string;
var
Request: TStringList;
Response: TMemoryStream;
LHandler: TIdSSLIOHandlerSocketOpenSSL; // added as its a https site
begin
Result := '';
try
Response := TMemoryStream.Create;
try
Request := TStringList.Create;
try
{1} Request.Add('op=login');
{2} Request.Add('redirect=https://www.thewebsite.com/Login.aspx' );
{3} Request.Add('ctl00$ctl00$Body$Body$loginManager$ctl00$loginEmailInput=usernme');
{4} Request.Add('ctl00$ctl00$Body$Body$loginManager$ctl01$passwordInput=password'});
LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil); //<<< added as its a https site
IdHTTP.IOHandler := LHandler; //<<< added as its a https site
IdHTTP.AllowCookies := True;
IdHTTP.HandleRedirects := True;
{5} IdHTTP.Request.ContentType := 'application/x-www-form-urlencoded';
{6} IdHTTP.Post('https://www.thewebsite.com/Login.aspx', Request, Response);
{7} Result := IdHTTP.Get('https://www.thewebsite.com/Login.aspx');
finally
Request.Free;
end;
finally
Response.Free;
end;
except
on E: Exception do
ShowMessage(E.Message);
end;
end;
The net result of all this is that I don't get a file created at all now, not even a zero byte one. This all seems very overcomplicated simply to avoid or automate the 'Save As' dialog and is requiring lots of code that I won't be able to maintan afterwards. Unless somebody has a simpler solution I'll just parse what I can see (BTW I tried TEmbeddedWebBrowser but there is so little documentation for it I couldn't see how to make it download correctly. Might try again later.) Thank you for trying to help!
I want to put data from Memo1 directly to my FTP server, I've got code:
procedure TForm5.SendClick(Sender: TObject);
var K: TStream;
begin
K := TStream.Create;
Memo1.Lines.SaveToStream(K);
FTP.Host := 'localhost';
FTP.Username := 'login';
FTP.Password := 'haslo';
FTP.Connect;
if FTP.Connected then FTP.Put(K,'');
end;
But when I click "Send" button I've got two errors:
when Memo is empty
when I try send data
TStream is an abstract class. You must never instantiate it. Use a concrete class instead like, for instance, TMemoryStream.
You'll also want to destroy the stream when you are finished with it, or it will leak. Do yourself a favour and set ReportMemoryLeaksOnShutdown to True, for instance in your .dpr file. That will allow you to get a report of all the memory you are leaking when your program terminates.
Your code might run like this:
var
Stream: TMemoryStream;
....
Stream := TMemoryStream.Create;
try
// .... initialize the Indy object
if FTP.Connected then begin
// .... populate stream
Stream.Position := 0;
FTP.Put(Stream, '');
end;
finally
Stream.Free;
end;
I am trying to log into craigslist using Delphi, and retrieve my account page (in order to gather a listing of all my posts)
However, I can't seem to get the login to work, what Am I doing wrong?
function TfrmMain.Login: string;
var
IdHTTP: TIdHTTP;
Request: TStringList;
Response: TMemoryStream;
begin
Result := '';
try
Response := TMemoryStream.Create;
try
Request := TStringList.Create;
try
Request.Add('op=login');
Request.Add('redirect=http://newyork.craigslist.org/');
Request.Add('login=' + myEmail);
Request.Add('password=' + myPassword);
IdHTTP := TIdHTTP.Create;
try
IdHTTP.AllowCookies := True;
IdHTTP.HandleRedirects := True;
IdHTTP.Request.ContentType := 'application/x-www-form-urlencoded';
IdHTTP.Post('https://accounts.craigslist.org/login', Request, Response);
Result := IdHTTP.Get('https://accounts.craigslist.org/');
finally
IdHTTP.Free;
end;
finally
Request.Free;
end;
finally
Response.Free;
end;
except
on E: Exception do
ShowMessage(E.Message);
end;
end;
I get a exception class EIdIOHandlerPropInvalid with message 'IOHandler value is not valid' on the line:
IdHTTP.Post('https://accounts.craigslist.org/login', Request, Response);
thanks
See (Indy) TIdHTTP EIdSocketError Socket Error # 0 exceptions when downloading files and the suggestions in the comments. It looks like you should upgrade to a more recent version of Indy.
How to get body texts of all e-mail messages from a certain IMAP mailbox in Delphi ? For example, from the INBOX mailbox ?
There are many ways to retrieve all body texts of all messages from the selected mailbox. I've used the one, where you iterate the mailbox and Retrieve every single message from the mailbox one by one. This way allows you to modify the code, so that you'll be able to break the loop when you need or e.g. replace Retrieve by RetrievePeek which won't mark the message as read on server like the first mentioned does. When the message is retrieved from server, all its parts are iterated and when it's the text part, its body is appended to a local S variable. After the iteration the S variable is added to the output BodyTexts string list. So, as the result you'll get string list collection where each item consists from the concatenated message's text part bodies and where each item means one message.
uses
IdIMAP4, IdSSLOpenSSL, IdText, IdMessage, IdExplicitTLSClientServerBase;
procedure GetGmailBodyTextParts(const UserName, Password: string;
BodyTexts: TStrings);
var
S: string;
MsgIndex: Integer;
MsgObject: TIdMessage;
PartIndex: Integer;
IMAPClient: TIdIMAP4;
OpenSSLHandler: TIdSSLIOHandlerSocketOpenSSL;
begin
BodyTexts.Clear;
IMAPClient := TIdIMAP4.Create(nil);
try
OpenSSLHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
try
OpenSSLHandler.SSLOptions.Method := sslvSSLv3;
IMAPClient.IOHandler := OpenSSLHandler;
IMAPClient.Host := 'imap.gmail.com';
IMAPClient.Port := 993;
IMAPClient.UseTLS := utUseImplicitTLS;
IMAPClient.Username := UserName;
IMAPClient.Password := Password;
IMAPClient.Connect;
try
if IMAPClient.SelectMailBox('INBOX') then
begin
BodyTexts.BeginUpdate;
try
for MsgIndex := 1 to IMAPClient.MailBox.TotalMsgs do
begin
MsgObject := TIdMessage.Create(nil);
try
S := '';
IMAPClient.Retrieve(MsgIndex, MsgObject);
MsgObject.MessageParts.CountParts;
if MsgObject.MessageParts.TextPartCount > 0 then
begin
for PartIndex := 0 to MsgObject.MessageParts.Count - 1 do
if MsgObject.MessageParts[PartIndex] is TIdText then
S := S + TIdText(MsgObject.MessageParts[PartIndex]).Body.Text;
BodyTexts.Add(S);
end
else
BodyTexts.Add(MsgObject.Body.Text);
finally
MsgObject.Free;
end;
end;
finally
BodyTexts.EndUpdate;
end;
end;
finally
IMAPClient.Disconnect;
end;
finally
OpenSSLHandler.Free;
end;
finally
IMAPClient.Free;
end;
end;
This code requires OpenSSL, so don't forget to put the libeay32.dll and ssleay32.dll libraries to a path visible to your project; you can download OpenSSL libraries for Indy in different versions and platforms from here.
I want to make the registration process a user name and password. I use Delphi 7. Component which can be done? (I know very little english, sorry.)
http request is easiest method (with GET, or POST), but preferred you'll want to use SSL if you dont want passwords/usernames passed along to your webserver un-encrypted.
Example of using POST request:
uses
IdHTTP;
function PostData(const URL: string; Params: TStrings): string;
var
IdHTTP: TIdHTTP;
begin
Result := '';
IdHTTP := TIDHttp.Create(nil);
try
IdHTTP.HandleRedirects := True;
IdHTTP.ReadTimeout := 5000;
Result := IdHTTP.Post(URL, Params);
finally
IdHTTP.Free;
end;
end;
Optionally you can write your own socket, but that will be more difficult because you'll have to write your own listener. (Which is usually not allowed on most shared hosting plans.)