I have the following task: download a file using HTTPS and authentication. Indy seems the way to go but for some reason it doesn't work so far. I have the following in place:
a TIdHTTP component which I use for downloading
a TIdURI component used to create the URL
a TIdSSLIOHandlerSocketOpenSSL component which should provide the secure connection. The required DLLs are in the binary folder.
The site also requires authentication and I included the user/pass in the URL as in the example below. In short this is the code:
URI := TIdURI.Create('https://test.example.com/');
URI.Username := ParamUserName;
URI.Password := ParamPassword;
HTTP := TIdHTTP.Create(nil);
if URI.Protocol = 'https' then
begin
IOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
IOHandler.SSLOptions.Method := sslvSSLv3;
HTTP.IOHandler := IOHandler;
end;
HTTP.Get(URI.GetFullURI([ofAuthInfo]), FileStream);
Using this code I get a "Read Timeout" EIdReadTimeout exception very fast. Testing the URL in a browser works without problem. Any ideas on what's missing or what I did wrong?
I finally abandoned Indy and OpenSSL and used WinInet for downloading. This is the code that worked for me:
function Download(URL, User, Pass, FileName: string): Boolean;
const
BufferSize = 1024;
var
hSession, hURL: HInternet;
Buffer: array[1..BufferSize] of Byte;
BufferLen: DWORD;
F: File;
begin
Result := False;
hSession := InternetOpen('', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0) ;
// Establish the secure connection
InternetConnect (
hSession,
PChar(FullURL),
INTERNET_DEFAULT_HTTPS_PORT,
PChar(User),
PChar(Pass),
INTERNET_SERVICE_HTTP,
0,
0
);
try
hURL := InternetOpenURL(hSession, PChar(URL), nil, 0, 0, 0) ;
try
AssignFile(f, FileName);
Rewrite(f,1);
try
repeat
InternetReadFile(hURL, #Buffer, SizeOf(Buffer), BufferLen) ;
BlockWrite(f, Buffer, BufferLen)
until BufferLen = 0;
finally
CloseFile(f) ;
Result := True;
end;
finally
InternetCloseHandle(hURL)
end
finally
InternetCloseHandle(hSession)
end;
end;
I have seen the same thing. Setting the TIdHTTP.ReadTimeout to zero fixes the problem for me.
...
HTTP.IOHandler := IOHandler;
HTTP.ReadTimeout := 0;
All SSL related functions for secure connections will fail unless some additional libraries are installed properly.
1.) Download libraries
2.) unzip and copy both DLLs to your project folder (or somewhere in the PATH of your system)
with that your code from the question works fine for me.
--reinhard
Related
I tried using TRESTClient to connect to an HTTPS web service using TLS 1.2. But NO LUCK with sending multipart/form-data.
So now I am trying with Indy. I got this "Wrong Version Number" error.
I think there is nothing wrong with the code since it worked with HTTP.
Probably my Delphi is missing something. What should I install and how?
procedure TForm10.Button2Click(Sender: TObject);
var
HTTP: TIdHTTP;
RequestBody: TStream;
ResponseBody: string;
myssl: TIdSSLIOHandlerSocketOpenSSL;
Input: TIdMultipartFormDataStream;
begin
ResponseBody := '';
try
try
HTTP := TIdHTTP.Create;
try
Input := TIdMultipartFormDataStream.Create;
try
Input.Clear;
Input.AddFormField('Email', 'xx#xx.com.tr');
Input.AddFormField('Password', 'xx');
myssl := TIdSSLIOHandlerSocketOpenSSL.Create(HTTP);
HTTP.IOHandler := myssl;
myssl.SSLOptions.Mode := sslmUnassigned;
myssl.SSLOptions.Method := sslvTLSv1_2;
myssl.SSLOptions.SSLVersions := [sslvTLSv1_2];
HTTP.HTTPOptions := [hoForceEncodeParams];
HTTP.Request.CustomHeaders.FoldLines := False;
ResponseBody := HTTP.Post('https://xxx.com.tr/api/Mobile/MobileLoginControl', Input);
finally
Input.Free;
end;
finally
HTTP.Free;
end;
finally
end;
except
ResponseBody := '"-20"';
end;
end;
The code is fine, though you are enabling only TLS 1.2 on the SSLIOHandler. Maybe the website in question doesn't support TLS 1.2? Try enabling TLS 1.0 and 1.1 as well:
myssl.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
And don't set the SSLOptions.Method at all. Setting the SSLVersions updates the Method and vice versa. So set one or the other, not both.
I have an AWS S3 account and got SecretAccessKey, SessionToken, Expiration, AccessKeyId items. I would like to upload some files to the cloud, in the simplest way.
Have read a some docs regarding authorization headers (http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html) but still do not understand how to build them))
also, saw another example with Indy, so, please help to build the authorization header with these items that I have: SecretAccessKey, SessionToken, Expiration, AccessKeyId. It's ok to be with a "Transferring Payload in a Single Chunk" mode + "Signed payload option".
FS := TFileStream.Create('c:\myfile.txt', fmOpenRead or fmShareDenyWrite);
try
IdHTTP1.Request.CustomHeaders.Values['Authorization'] := ...; // please help
IdHTTP1.Request.BasicAuthentication := False;
IdHTTP1.Request.Date := ...; //what should I enter here?
IdHTTP1.Request.Expect := '100-continue';
IdHTTP1.Request.ProtocolVersion := pv1_1;
...
IdHTTP1.Put('http://'+BucketName+'.s3.amazonaws.com/myfile.txt', FS);
finally
FS.Free;
end;
Thank you!
Here's my routine to upload files to Amazon using the Cloud Components:
function UploadFile(File: TBytes; FileName: string; Bucket: string): boolean;
var Service: TAmazonStorageService;
ConAmazon: TAmazonConnectionInfo;
begin
try
ConAmazon := TAmazonConnectionInfo.Create(nil);
ConAmazon.AccountKey := 'Dih71bG09****************';
ConAmazon.AccountName := 'AKIA***********';
ConAmazon.QueueEndpoint := 'queue.amazonaws.com';
ConAmazon.StorageEndpoint := 's3-eu-west-1.amazonaws.com';
ConAmazon.TableEndpoint := 'sdb.amazonaws.com';
ConAmazon.UseDefaultEndpoints := False;
Service := TAmazonStorageService.Create(ConAmazon);
Result := Service.UploadObject(Bucket, FileName, File, TRUE, nil, nil, amzbaPrivate, nil);
finally
ConAmazon.Free;
Service.Free;
end;
end;
Ok, so, finally we've solved the issue:
1.Data.Cloud from Delphi10.1 Berlin should be used. It supports Amazon AWS4 security standard.
2.TAmazonStorageService.InitHeaders should be patched by adding the following code to enable temporary session tokens usage in the header:
...
Result.Values['x-amz-security-token'] := //your session_token string;
...
tested it from many sides, works fine now:)
I have to use Delphi 2006. I have to use Indy 10.1.5 - comes with Delphi 2006 or not, but I have to use these versions! I found an example how to use indy SSL https get but now I completely lost my head and close to to do another 'bad day' video!
Finally, the SSL library loaded without any problem.
But... Why I get always 'EidReadTimeout with message 'Read Timeout'
here is my code:
var
IdHTTP1: TIdHTTP;
ParamStringList: TStringList;
s1: String;
IdSSLIOHandlerSocket1: TIdSSLIOHandlerSocketOpenSSL;
begin
IdHTTP1 := TIdHTTP.Create(nil);
IdSSLIOHandlerSocket1 := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
IdSSLIOHandlerSocket1.ReadTimeout := 10000;
IdHTTP1.IOHandler := IdSSLIOHandlerSocket1;
IdHTTP1.ConnectTimeout := 10000;
IdSSLIOHandlerSocket1.SSLOptions.Method := sslvSSLv23; // Which one is the good for...
IdSSLIOHandlerSocket1.SSLOptions.Mode := sslmClient;
IdSSLIOHandlerSocket1.SSLOptions.VerifyMode := [];
IdSSLIOHandlerSocket1.SSLOptions.VerifyDepth := 0;
ParamStringList := TStringList.Create;
ParamStringList.Text := '';
s1 := IdHTTP1.Post('https://msp.f-secure.com/web-test/common/test.html', ParamStringList);
Memo1.Text := s1;
ParamStringList.Free;
IdSSLIOHandlerSocket1.Free;
IdHTTP1.Free;
end;
Any idea? What can I missed?
I changed the timeout between 3 and 100 seconds, but no changes when I tried to ran my code.
Thanks in advance!
There's a web services I want to call in my application, I can use it with importing the WSDL or by just use "HTTP GET" with the URL and parameters, so I prefer the later because it's simple thing.
I know I can use indy idhttp.get, to do the job, but this is very simple thing and I don't want to add complex indy code to my application.
UPDATE: sorry if I was not clear, I meant by "not to add complex indy code", that I don't want add indy components for just this simple task, and prefer more lighter way for that.
Calling a RESTful web service using Indy is pretty straight forward.
Add IdHTTP to your uses clause. Remember that IdHTTP needs the "HTTP://" prefix on your URLs.
function GetURLAsString(const aURL: string): string;
var
lHTTP: TIdHTTP;
begin
lHTTP := TIdHTTP.Create;
try
Result := lHTTP.Get(aURL);
finally
lHTTP.Free;
end;
end;
You could use the WinINet API like this:
uses WinInet;
function GetUrlContent(const Url: string): string;
var
NetHandle: HINTERNET;
UrlHandle: HINTERNET;
Buffer: array[0..1024] of Char;
BytesRead: dWord;
begin
Result := '';
NetHandle := InternetOpen('Delphi 5.x', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
if Assigned(NetHandle) then
begin
UrlHandle := InternetOpenUrl(NetHandle, PChar(Url), nil, 0, INTERNET_FLAG_RELOAD, 0);
if Assigned(UrlHandle) then
{ UrlHandle valid? Proceed with download }
begin
FillChar(Buffer, SizeOf(Buffer), 0);
repeat
Result := Result + Buffer;
FillChar(Buffer, SizeOf(Buffer), 0);
InternetReadFile(UrlHandle, #Buffer, SizeOf(Buffer), BytesRead);
until BytesRead = 0;
InternetCloseHandle(UrlHandle);
end
else
{ UrlHandle is not valid. Raise an exception. }
raise Exception.CreateFmt('Cannot open URL %s', [Url]);
InternetCloseHandle(NetHandle);
end
else
{ NetHandle is not valid. Raise an exception }
raise Exception.Create('Unable to initialize Wininet');
end;
source: http://www.scalabium.com/faq/dct0080.htm
The WinINet API uses the same stuff InternetExplorer is using so you also get any connection and proxy settings set by InternetExplorer for free.
Actually code in accepted answer did't work for me. So I modified it a little bit so it actually returns String and gracefully closes everything after execution. Example returns retrieved data as UTF8String so it will work well for ASCII as well as for UTF8 pages.
uses WinInet;
function GetUrlContent(const Url: string): UTF8String;
var
NetHandle: HINTERNET;
UrlHandle: HINTERNET;
Buffer: array[0..1023] of byte;
BytesRead: dWord;
StrBuffer: UTF8String;
begin
Result := '';
NetHandle := InternetOpen('Delphi 2009', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
if Assigned(NetHandle) then
try
UrlHandle := InternetOpenUrl(NetHandle, PChar(Url), nil, 0, INTERNET_FLAG_RELOAD, 0);
if Assigned(UrlHandle) then
try
repeat
InternetReadFile(UrlHandle, #Buffer, SizeOf(Buffer), BytesRead);
SetString(StrBuffer, PAnsiChar(#Buffer[0]), BytesRead);
Result := Result + StrBuffer;
until BytesRead = 0;
finally
InternetCloseHandle(UrlHandle);
end
else
raise Exception.CreateFmt('Cannot open URL %s', [Url]);
finally
InternetCloseHandle(NetHandle);
end
else
raise Exception.Create('Unable to initialize Wininet');
end;
Hope it helps for somebody like me who was looking for easy code how to retrieve page content in Delphi.
Cheers, Aldis :)
In newer Delphi versions it is better to use THTTPClient from System.Net.HttpClient unit, since it is standard and cross-platform. Simple example is
function GetURL(const AURL: string): string;
var
HttpClient: THttpClient;
HttpResponse: IHttpResponse;
begin
HttpClient := THTTPClient.Create;
try
HttpResponse := HttpClient.Get(AURL);
Result := HttpResponse.ContentAsString();
finally
HttpClient.Free;
end;
end;
If it's okay to download to a file, you can use TDownloadURL from the ExtActns unit. Much simpler than using WinInet directly.
procedure TMainForm.DownloadFile(URL: string; Dest: string);
var
dl: TDownloadURL;
begin
dl := TDownloadURL.Create(self);
try
dl.URL := URL;
dl.FileName := Dest;
dl.ExecuteTarget(nil); //this downloads the file
finally
dl.Free;
end;
end;
It's also possible to get progress notifications when using this. Simply assign an event handler to TDownloadURL's OnDownloadProgress event.
Using Windows HTTP API might be easy too.
procedure TForm1.Button1Click(Sender: TObject);
var http: variant;
begin
http:=createoleobject('WinHttp.WinHttpRequest.5.1');
http.open('GET', 'http://lazarus.freepascal.org', false);
http.send;
showmessage(http.responsetext);
end;
https://msdn.microsoft.com/ru-ru/library/windows/desktop/aa382925.aspx
http://forum.lazarus.freepascal.org/index.php?topic=14609.0
Save a file downloaded via WinHTTP to disk, using Delphi XE
How to use "WinHttp.WinHttpRequest.5.1" asynchronously?
http://www.delphigroups.info/2/4/217167.html
https://github.com/fabriciocolombo/delphi-rest-client-api/blob/master/src/HttpConnectionWinHttp.pas
In the code above I imply that COM was already initialized for the main VCL thread. Reportedly it might not be always the case for simplistic apps or for LCL apps. Also it would definitely not be the case for async (multithread) work.
Below is the snippet from a real code running. Note - the functionality is bonus. It is not required to work. So while I do issue requests, I do not care about their results, that result is ignored and dumped.
procedure TfmHaspList.YieldBlinkHTTP(const LED: boolean; const Key_Hardware_ID: cardinal);
var URL: WideString;
begin
URL := 'http://127.0.0.1:1947/action.html?blink' +
IfThen( LED, 'on', 'off') + '=' + IntToStr(Key_Hardware_ID);
TThread.CreateAnonymousThread(
procedure
var Request: OleVariant;
begin
// COM library initialization for the current thread
CoInitialize(nil);
try
// create the WinHttpRequest object instance
Request := CreateOleObject('WinHttp.WinHttpRequest.5.1');
// open HTTP connection with GET method in synchronous mode
Request.Open('GET', URL, False);
// set the User-Agent header value
// Request.SetRequestHeader('User-Agent', 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0');
// sends the HTTP request to the server, the Send method does not return
// until WinHTTP completely receives the response (synchronous mode)
Request.Send;
// // store the response into the field for synchronization
// FResponseText := Request.ResponseText;
// // execute the SynchronizeResult method within the main thread context
// Synchronize(SynchronizeResult);
finally
// release the WinHttpRequest object instance
Request := Unassigned;
// uninitialize COM library with all resources
CoUninitialize;
end;
end
).Start;
end;
Use the Synapse TCP/IP function in the HTTPSEND unit (HTTPGetText, HTTPGetBinary). It will do the HTTP pull for you and doesn't require any external DLL's other than Winsock. The latest SVN release works perfectly well in Delphi 2009. This uses blocking function calls, so no events to program.
Update: The units are very light, and are not component based. The latest version from SVN runs perfectly well in Delphi XE4 also.
If your application is Windows-only, I would suggest using WinSock. It's simple enough, allows to execute any HTTP request, can work both synchronously and asynchronously (using non-blocking WSASend/WSARecv with callbacks or good old send/recv in a dedicated thread).
We are trying to write an update server for our software using the TIdHTTPServer component. Currently we are serving an XML file that lists the available updates and their file versions etc.., when the client program finds a updated version it should start to download it using BITS.
Now this is where we have a problem, our programs are requesting the XML file and seeing there is an update available. It then creates a BITS job to download it, however BITS keeps reporting that the download failed. We can download the file using the same URL and IE/Firefox/Chrome.
so my question:
Is TIdHTTPServer compatible with BITS?
I ask this as I have discovered that there are these download requirements for bits to work.
HTTP Requirements for BITS Downloads
BITS supports HTTP and HTTPS downloads and uploads and requires that the server supports the HTTP/1.1 protocol. For downloads, the HTTP server's Head method must return the file size and its Get method must support the Content-Range and Content-Length headers. As a result, BITS only transfers static file content and generates an error if you try to transfer dynamic content, unless the ASP, ISAPI, or CGI script supports the Content-Range and Content-Length headers.
BITS can use an HTTP/1.0 server as long as it meets the Head and Get method requirements.
To support downloading ranges of a file, the server must support the following requirements:
Allow MIME headers to include the standard Content-Range and Content-Type headers, plus a maximum of 180 bytes of other headers.
Allow a maximum of two CR/LFs between the HTTP headers and the first boundary string.
Just found a bug in indy that prevents transfer of files over 2.1GB when using range requests.
here it is
IdHTTPHeaderInfo.pas aprox line 770
procedure TIdEntityRange.SetText(const AValue: String);
var
LValue, S: String;
begin
LValue := Trim(AValue);
if LValue <> '' then
begin
S := Fetch(LValue, '-'); {do not localize}
if S <> '' then begin
FStartPos := StrToIntDef(S, -1);
FEndPos := StrToIntDef(Fetch(LValue), -1);
FSuffixLength := -1;
end else begin
FStartPos := -1;
FEndPos := -1;
FSuffixLength := StrToIntDef(Fetch(LValue), -1);
end;
end else begin
FStartPos := -1;
FEndPos := -1;
FSuffixLength := -1;
end;
end;
This should be
procedure TIdEntityRange.SetText(const AValue: String);
var
LValue, S: String;
begin
LValue := Trim(AValue);
if LValue <> '' then
begin
S := Fetch(LValue, '-'); {do not localize}
if S <> '' then begin
FStartPos := StrToInt64Def(S, -1);
FEndPos := StrToInt64Def(Fetch(LValue), -1);
FSuffixLength := -1;
end else begin
FStartPos := -1;
FEndPos := -1;
FSuffixLength := StrToInt64Def(Fetch(LValue), -1);
end;
end else begin
FStartPos := -1;
FEndPos := -1;
FSuffixLength := -1;
end;
end;
One for Remy to fix
When you handle the OnCommandGet event, you are given a TIdRequestHeaderInfo, which descends from TIdEntityHeaderInfo; that contains all the headers the request contained, and it even parses out some header values to read as properties, including ContentRangeStart, ContentRangeEnd, and ContentLength.
You can use those properties to populate the stream that you assign to the TIdHTTPResponseInfo.ContentStream property. The entire stream will get sent.
It's your job to differentiate between GET and HEAD requests; OnCommandGet will get triggered either way. Check the IdHTTPRequestInfo.CommandType property.
So, although Indy may not support BITS, it provides all the tools you need to write a program that does support BITS.
So the answer to this question is:
Yes TIdHTTPServer is Bits Compatible.
But only if you are prepared to do the work yourself.
As suggested by #Rob Kennedy and Myself it is possible to read the headers and send the data back using the requested ranges, one chunk at a time.
Here is an example of what I am doing in the OnCommandGet event
procedure TForm3.IdHTTPServer1CommandGet(AContext: TIdContext;
ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
var
Ranges : TIdEntityRanges;
DataChunk: TMemoryStream;
ReqFile: TFileStream;
ChunkLength: Int64;
Directory, FileName: string;
begin
Directory := 'H:';
case ARequestInfo.Ranges.Count of
0:
begin
//serve file normally
end;
1:
begin
//serve range of bytes specified for file
filename := Directory + ARequestInfo.Document;
if FileExists(FileName) then
begin
ReqFile := TFileStream.Create(FileName, fmOpenRead);
try
ChunkLength := Succ(ARequestInfo.Ranges.Ranges[0].EndPos - ARequestInfo.Ranges.Ranges[0].StartPos);
if ChunkLength > ReqFile.Size then
ChunkLength := ReqFile.Size;
DataChunk := TMemoryStream.Create;
DataChunk.Posistion := ARequestInfo.Ranges.Ranges[0].StartPos;
DataChunk.CopyFrom(ReqFile, ChunkLength);
AResponseInfo.ContentStream := DataChunk;
AResponseInfo.ContentType := IdHTTPServer1.MIMETable.GetFileMIMEType(FileName);
AResponseInfo.ContentRangeUnits := ARequestInfo.Ranges.Units;
AResponseInfo.ContentRangeStart := ARequestInfo.Ranges.Ranges[0].StartPos;
AResponseInfo.ContentRangeEnd := ARequestInfo.Ranges.Ranges[0].StartPos + Pred(ChunkLength);
AResponseInfo.ContentRangeInstanceLength := ReqFile.Size;
AResponseInfo.ResponseNo := 206;
finally
ReqFile.Free;
end;
end
else
AResponseInfo.ResponseNo := 404;
end
else
begin
//serve the file as multipart/byteranges
end;
end;
end;
This is by no means finished but it shows the basics of responding to the range requests from BITS. Most importantly it works.
Any comments on the code would be appreciated, constructive criticism always welcome.