Download file progressively using TIdHttp - delphi

Question

I want to implement a simple http downloader using TIdHttp (Indy10). I found two kind of code examples from the internet. Unfortunately none of them satisfy me 100%. Here is the code and I want some advise.
Variant 1
var
Buffer: TFileStream;
HttpClient: TIdHttp;
begin
Buffer := TFileStream.Create('somefile.exe', fmCreate or fmShareDenyWrite);
try
HttpClient := TIdHttp.Create(nil);
try
HttpClient.Get('http://somewhere.com/somefile.exe', Buffer); // wait until it is done
finally
HttpClient.Free;
end;
finally
Buffer.Free;
end;
end;
The code is compact and very easy to understand. The problem is that it allocates disk space when downloading begins. Another problem is that we cannot show the download progress in GUI directly, unless the code is executed in a background thread (alternatively we can bind HttpClient.OnWork event).
Variant 2:
const
RECV_BUFFER_SIZE = 32768;
var
HttpClient: TIdHttp;
FileSize: Int64;
Buffer: TMemoryStream;
begin
HttpClient := TIdHttp.Create(nil);
try
HttpClient.Head('http://somewhere.com/somefile.exe');
FileSize := HttpClient.Response.ContentLength;
Buffer := TMemoryStream.Create;
try
while Buffer.Size < FileSize do
begin
HttpClient.Request.ContentRangeStart := Buffer.Size;
if Buffer.Size + RECV_BUFFER_SIZE < FileSize then
HttpClient.Request.ContentRangeEnd := Buffer.Size + RECV_BUFFER_SIZE - 1
else
HttpClient.Request.ContentRangeEnd := FileSize;
HttpClient.Get(HttpClient.URL.URI, Buffer); // wait until it is done
Buffer.SaveToFile('somefile.exe');
end;
finally
Buffer.Free;
end;
finally
HttpClient.Free;
end;
end;
First we query the file size from the server and then we download file contents in pieces. Retrieved file contents will be save to disk when they are received completely. The potential problem is we have to send multiple GET requests to the server. I am not sure if some servers (such as megaupload) might limit the number of requests within particular time period.
My expectations
The downloader should send only one GET-request to the server.
The disk space must not be allocated when the download begins.
Any hints are appreciated.

Answer 1

Variant #1 is the simpliest, and is how Indy is meant to be used.
Regarding the disk allocation issue, you can derive a new class from TFileStream and override its SetSize() method to do nothing. TIdHTTP will still attempt to pre-allocate the file when appropriate, but it will not actually allocate any disk space. Writing to TFileStream will grow the file as needed.
Regarding status reporting, TIdHTTP has OnWork... events for that purpose. The AWorkCountMax parameter of the OnWorkBegin will be the actual file size if known (the response is not chunked), or 0 if not known. The AWorkCount parameter of the OnWork event will be the cumulative number of bytes that have been transferred so far. If the file size is known, you can display the total percentage by simply dividing the AWorkCount by the AWorkCountMax and multiplying by 100, otherwise just display the AWorkCount value by itself. If you want to display the speed of the transfer, you can calculate that from the difference of AWorkCount values and the time intervals between multiple OnWork events.
Try this:
type
TNoPresizeFileStream = class(TFileStream)
procedure
procedure SetSize(const NewSize: Int64); override;
end;
procedure TNoPresizeFileStream.SetSize(const NewSize: Int64);
begin
end;
.
type
TSomeClass = class(TSomething)
...
TotalBytes: In64;
LastWorkCount: Int64;
LastTicks: LongWord;
procedure Download;
procedure HttpWorkBegin(ASender: TObject; AWorkMode: TWorkMode; AWorkCountMax: Int64);
procedure HttpWork(ASender: TObject; AWorkMode: TWorkMode; AWorkCount: Int64);
procedure HttpWorkEnd(ASender: TObject; AWorkMode: TWorkMode);
...
end;
procedure TSomeClass.Download;
var
Buffer: TNoPresizeFileStream;
HttpClient: TIdHttp;
begin
Buffer := TNoPresizeFileStream.Create('somefile.exe', fmCreate or fmShareDenyWrite);
try
HttpClient := TIdHttp.Create(nil);
try
HttpClient.OnWorkBegin := HttpWorkBegin;
HttpClient.OnWork := HttpWork;
HttpClient.OnWorkEnd := HttpWorkEnd;
HttpClient.Get('http://somewhere.com/somefile.exe', Buffer); // wait until it is done
finally
HttpClient.Free;
end;
finally
Buffer.Free;
end;
end;
procedure TSomeClass.HttpWorkBegin(ASender: TObject; AWorkMode: TWorkMode; AWorkCountMax: Int64);
begin
if AWorkMode <> wmRead then Exit;
// initialize the status UI as needed...
//
// If TIdHTTP is running in the main thread, update your UI
// components directly as needed and then call the Form's
// Update() method to perform a repaint, or Application.ProcessMessages()
// to process other UI operations, like button presses (for
// cancelling the download, for instance).
//
// If TIdHTTP is running in a worker thread, use the TIdNotify
// or TIdSync class to update the UI components as needed, and
// let the OS dispatch repaints and other messages normally...
TotalBytes := AWorkCountMax;
LastWorkCount := 0;
LastTicks := Ticks;
end;
procedure TSomeClass.HttpWork(ASender: TObject; AWorkMode: TWorkMode; AWorkCount: Int64);
var
PercentDone: Integer;
ElapsedMS: LongWord;
BytesTransferred: Int64;
BytesPerSec: Int64;
begin
if AWorkMode <> wmRead then Exit;
ElapsedMS := GetTickDiff(LastTicks, Ticks);
if ElapsedMS = 0 then ElapsedMS := 1; // avoid EDivByZero error
if TotalBytes > 0 then
PercentDone := (Double(AWorkCount) / TotalBytes) * 100.0;
else
PercentDone := 0.0;
BytesTransferred := AWorkCount - LastWorkCount;
// using just BytesTransferred and ElapsedMS, you can calculate
// all kinds of speed stats - b/kb/mb/gm per sec/min/hr/day ...
BytesPerSec := (Double(BytesTransferred) * 1000) / ElapsedMS;
// update the status UI as needed...
LastWorkCount := AWorkCount;
LastTicks := Ticks;
end;
procedure TSomeClass.HttpWorkEnd(ASender: TObject; AWorkMode: TWorkMode);
begin
if AWorkMode <> wmRead then Exit;
// finalize the status UI as needed...
end;

Answer 2

Here is an example that shows how to use the components OnWork to show a progress bar:
Download a File from internet programatically with an Progress event using Delphi and Indy
You should not worry about the disk allocation. Disk space that is allocated is not actually written to, so it won't damage your disks. Be happy that it is allocated so that it is not possible that another process claims the disk space and let you run out of space!

Answer 3

Do not forget to add this for the Variant 2
: Else HttpClient.Request.ContentRangeEnd := FileSize;
Replace
if Buffer.Size + RECV_BUFFER_SIZE < FileSize then
HttpClient.Request.ContentRangeEnd := Buffer.Size + RECV_BUFFER_SIZE - 1;
By
if Buffer.Size + RECV_BUFFER_SIZE < FileSize then
HttpClient.Request.ContentRangeEnd := Buffer.Size + RECV_BUFFER_SIZE - 1;
Else HttpClient.Request.ContentRangeEnd := FileSize;