I use this code to download small files:
Var
ms:TMemoryStream;
begin
ms:=TMemoryStream.Create;
Idhttp1.get('http://mydomain.com/myfile.zip',ms);
ms.SaveToFile('myfile.zip');
ms.Free;
end;
But file is saved in RAM before storing to disk, so it may be difficult to download files >1Gb, for example. Is there a way to download a file by its parts? Or do I need to use the WinInet? Thanks in advance!
TMemoryStream provides an in-memory buffer, so if you download into one, you need to have enough memory to hold everything you receive. It's not the only kind of stream, though. You can pass the Get method any kind of stream you want, including one that writes its contents to disk as it receives it. Use TFileStream, for example.
var
s: TStream;
s := TFileStream.Create('myfile.zip', fmCreate);
try
IdHttp1.Get(..., s);
finally
s.Free;
end;
Anywhere you call LoadFromFile or SaveToFile on a TMemoryStream, it's possible that TFileStream is a better choice.
Related
Using Delphi XE2 and the native TZipFile I attempt to extract the contents of a downloaded zip file (which contains 2 zipped XML files) and it always extracts zero byte files.
The file is being compressed by C# code like this:
var zipFile = new ZipFile();
foreach (Tuple<string, string> t in filesMeta) {
zipFile.AddFile(string.Format("{0}{1}", StaticVariables.WebServerFileStorage, t.Item2), "").FileName = t.Item1 + ".xml";
}
response.Clear();
response.ContentType = "application/zip";
zipFile.Save(response.OutputStream);
response.End();
The Delphi extraction code is this:
zipFile := TZipFile.Create;
try
filename := 'C:\test\57f52480-ec87-4169-a820-0a65bc4ad952.zip';
if zipFile.IsValid(filename) then begin
zipFile.Open(filename, zmRead);
zipFile.ExtractAll('C:\test\');
end;
finally
zipFile.Free;
end;
I even tried using a TStream as the source instead of a file on disk. That's actually what I want to do since the zip file is downloaded from a web server into a TStream. I tried to extract the data using the overloaded open method of TZipFile to open the stream.
That got me zero byte files so I saved the zip file to disk and tried to open the file from disk and extract. Same zero byte files are extracted.
I even tried using the class method to extract the files from the zip file on disk:
System.Zip.TZipFile.ExtractZipFile(filename, 'C:\Test\');
Same zero byte files extracted.
The zip file is valid and the 2 zipped XML files can be extracted properly by both Windows 7 native file handling and 7-Zip.
Now here is something nutty...
In desperation I tried to see what the ExtractToFile() procedure
David Heffernan came up with in this question about extracting a zip to a stream would do so I tried using it like this:
var x : integer;
var fileCount : integer;
var fileNames : TArray<string>;
begin
zipFile := TZipFile.Create;
try
filename := 'C:\test\57f52480-ec87-4169-a820-0a65bc4ad952.zip';
if zipFile.IsValid(filename) then begin
zipFile.Open(filename, zmRead);
fileCount := zipFile.FileCount;
fileNames := copy(zipFile.FileNames, 0, MaxInt);
zipFile.Close;
for x := 0 to fileCount-1 do begin
// Use David Heffernan's stream procedure
ExtractToFile(filename, x, 'C:\test\' + fileNames[x]);
end;
end;
finally
zipFile.Free;
end;
end;
And David's procedure extracts the files to disk as expected! WTF???
I am really confused why a convoluted method of extraction would work and the simple extraction method would not work. I'll use David's example if I have to but I'd prefer to get the normal extract working if possible.
Any ideas are appreciated.
Cheers!
TJ
I had this same problem.
The source code to TZipFile shows that the TStream passed into the Read function returns the entire zip file with position set to the start of the filename you're wanting. So don't rewind. Just copyfrom or do what you want with the TStream for the uncompressed length given in the TZipHeader.
ZipStream := TStream.Create;
ZipFile.Read(MyFileName, ZipStream, ZipHeader);
//leave ZipStream pointer where it is!!!
SomethingElse.LoadFromStream(ZipStream, ZipHeader.UncompressedSize);
ZipStream.Free;
In my opinion, TZipFile should really load the ZipStream with only what is requested. The way this is implemented is not intuitive without first going through the TZipFile source code.
TL;DR: The solution to my problem was an external component. Zip Forge (or Abbrevia)
Read on for details.
Nothing I tried except for the roundabout way of saving the file and re-opening it using David's function worked. While that would have worked, it was not optimal as it required me to save the downloaded file to disk first and reopen it for extract and then delete the zip file. My goal was to just open the downloaded stream and extract the files directly to disk. One write to disk and no temporary file.
We even tried two different C# libraries to zip the files and both gave the same results on the streamed data. The Delphi TZipFile component could not handle it.
It turns out we have a license to ZipForge which I had forgotten about since I had not used it in ages it and that handles the download stream from the C# web server and extracts the files successfully.
For reference, I also tried the Abbrevia component version 5.2 and that also successfully extracted the files from the stream.
Hopefully this will help someone else.
All the suggestions by David and Uwe were appreciated.
Cheers!
TJ
Is it possible in Delphi to determine the size of a file as it is being copied? I get a notification when a file is first copied to a folder, but need to wait until the copy is complete before I can process the file.
I've used JclFileUtils.GetSizeOfFile(Filename) but that gives me the 'expected' file size, not the current filesize.
Regards, Pieter
Prompted by the first answer I decided to give up on trying to determine when a file copy has completed. Instead I found that using TFileStream gave me a reliable indication whether a file is in use or not.
function IsFileInUse(Filename: string; var ResultMessage: string): boolean;
var
Stream: TFileStream;
begin
Result := True;
ResultMessage := '';
try
Stream := TFileStream.Create(Filename, fmOpenRead or fmShareDenyWrite);
try
Result := False;
finally
FreeAndNil(Stream);
end;
Except on E: Exception do
ResultMessage := 'IsFileInUse: ' + E.Message
end;
end;
In this way I can keep on checking until the file is not in use anymore before attempting to process it.
It depends on the technique that is used by the copying function. Most copy-methods will allocate the disk space first before they start to copy a file. Thus, if you want to copy a file of 4 GB, the system starts by creating a file with random data for 4 GB in total. (Which is done lightning-fast, btw.) It then copies the data itself, but the file size is already what you expect.
This has as advantage that the sysmen can check if there's enough disk space available to actually copy the data.
If you write your own file copy function then you can have total control over how it does this. Else, you're limited to whatever the chosen copy-method offers you. So, how do you copy a file?
If you have control over the file copy process, it is easiest to have the copy routine create the file using a temporary filename, and when done, rename it to correct filename.
That way, you can use Windows folder monitoring to watch for the renaming (JCL contains a component to help with this, not sure about the name from here). When your code gets triggered you are sure the other side has finished writing the file.
A simple trick I used was to have the copying process create new files with a '$$$' extension. My code still got triggered for those but I ignored them until they were renamed to their proper filename.
Hope this helps.
I have a "Find Files" function in my program that will find text files with the .ged suffix that my program reads. I display the found results in an explorer-like window that looks like this:
I use the standard FindFirst / FindNext methods, and this works very quickly. The 584 files shown above are found and displayed within a couple of seconds.
What I'd now like to do is add two columns to the display that shows the "Source" and "Version" that are contained in each of these files. This information is found usually within the first 10 lines of each file, on lines that look like:
1 SOUR FTM
2 VERS Family Tree Maker (20.0.0.368)
Now I have no problem parsing this very quickly myself, and that is not what I'm asking.
What I need help with is simply how to most quickly load the first 10 or so lines from these files so that I can parse them.
I have tried to do a StringList.LoadFromFile, but it takes too much time loading the large files, such at those above 1 MB.
Since I only need the first 10 lines or so, how would I best get them?
I'm using Delphi 2009, and my input files might or might not be Unicode, so this needs to work for any encoding.
Followup: Thanks Antonio,
I ended up doing this which works fine:
var
CurFileStream: TStream;
Buffer: TBytes;
Value: string;
Encoding: TEncoding;
try
CurFileStream := TFileStream.Create(folder + FileName, fmOpenRead);
SetLength(Buffer, 256);
CurFileStream.Read(Buffer[0], 256);
TEncoding.GetBufferEncoding(Buffer, Encoding);
Value := Encoding.GetString(Buffer);
...
(parse through Value to get what I want)
...
finally
CurFileStream.Free;
end;
Use TFileStream and with Read method read number of bytes needed. Here is the example of reading bitmap info that is also stored on begining of the file.
http://www.delphidabbler.com/tips/19
Just open the file yourself for block reading (not using TStringList builtin functionality), and read the first block of the file, and then you can for example load that block to a stringlist with strings.SetText() (if you are using block functions) or simply strings.LoadFromStream() if you are loading your blocks using streams.
I would personally just go with FileRead/FileWrite block functions, and load the block into a buffer. You could also use similair winapi functions, but that's just more code for no reason.
OS reads files in blocks, which are at least 512bytes big on almost any platform/filesystem, so you can read 512 bytes first (and hope that you got all 10 lines, which will be true if your lines are generally short enough). This will be (practically) as fast as reading 100 or 200 bytes.
Then if you notice that your strings objects has only less than 10 lines, just read next 512 byte block and try to parse again. (Or just go with 1024, 2048 and so on blocks, on many systems it will probably be as fast as 512 blocks, as filesystem cluster sizes are generally larger than 512 bytes).
PS. Also, using threads or asynchronous functionality in winapi file functions (CreateFile and such), you could load that data from files asynchronously, while the rest of your application works. Specifically, the interface will not freeze during reading of large directories.
This will make the loading of your information appear faster, (since the file list will load directly, and then some milliseconds later the rest of the information will come up), while not actually increasing the real reading speed.
Do this only if you have tried the other methods and you feel like you need the extra boost.
You can use a TStreamReader to read individual lines from any TStream object, such as a TFileStream. For even faster file I/O, you could use Memory-Mapped Views with TCustomMemoryStream.
Okay, I deleted my first answer. Using Remy's first suggestion above, I tried again with built-in stuff. What I don't like here is that you have to create and free two objects. I think I would make my own class to wrap this up:
var
fs:TFileStream;
tr:TTextReader;
filename:String;
begin
filename := 'c:\temp\textFileUtf8.txt';
fs := TFileStream.Create(filename, fmOpenRead);
tr := TStreamReader.Create(fs);
try
Memo1.Lines.Add( tr.ReadLine );
finally
tr.Free;
fs.Free;
end;
end;
If anybody is interested in what I had here before, it had the problem of not working with unicode files.
Sometimes oldschool pascal stylee is not that bad.
Even though non-oo file access doesn't seem to be very popular anymore, ReadLn(F,xxx) still works pretty ok in situations like yours.
The code below loads information (filename, source and version) into a TDictionary so that you can look it up easily, or you can use a listview in virtual mode, and look stuff up in this list when the ondata even fires.
Warning: code below does not work with unicode.
program Project101;
{$APPTYPE CONSOLE}
uses
IoUtils, Generics.Collections, SysUtils;
type
TFileInfo=record
FileName,
Source,
Version:String;
end;
function LoadFileInfo(var aFileInfo:TFileInfo):Boolean;
var
F:TextFile;
begin
Result := False;
AssignFile(F,aFileInfo.FileName);
{$I-}
Reset(F);
{$I+}
if IOResult = 0 then
begin
ReadLn(F,aFileInfo.Source);
ReadLn(F,aFileInfo.Version);
CloseFile(F);
Exit(True)
end
else
WriteLn('Could not open ', aFileInfo.FileName);
end;
var
FileInfo:TFileInfo;
Files:TDictionary<string,TFileInfo>;
S:String;
begin
Files := TDictionary<string,TFileInfo>.Create;
try
for S in TDirectory.GetFiles('h:\WINDOWS\system32','*.xml') do
begin
WriteLn(S);
FileInfo.FileName := S;
if LoadFileInfo(FileInfo) then
Files.Add(S,FileInfo);
end;
// showing file information...
for FileInfo in Files.Values do
WriteLn(FileInfo.Source, ' ',FileInfo.Version);
finally
Files.Free
end;
WriteLn;
WriteLn('Done. Press any key to quit . . .');
ReadLn;
end.
I want to distribute only a single .exe, however, at runtime I would like it to extract some embedded image resources to the users hard disk drive.
Can I, and if so, how?
Use Delphi's TResourceStream. It's constructor will find and load the resource into memory, and it's SaveToFile method will do the disk write.
Something similar to this should work:
var
ResStream: TResourceStream;
begin
ResStream := TResourceStream.Create(HInstance, 'YOURRESOURCENAME', RT_RCDATA);
try
ResStream.Position := 0;
ResStream.SaveToFile('C:\YourDir\YourFileName.jpg');
finally
ResStream.Free;
end;
end;
If you can use the resource ID instead of name, it's a little less memory. In that case, you'd resplace Create with CreateFromID, and supply the numeric ID rather than the string name.
Create a TResourceStream. You'll need the module instance handle (usually SysInit.HInstance for the current EXE file, or else whatever you get from LoadLibrary or LoadPackage), the resource type (such as rt_Bitmap or rt_RCData), and either the resource name or numeric ID. Then call the stream's SaveToFile method.
try
if not Assigned(Bitmap)
then
Bitmap := TBitmap.Create();
Bitmap.LoadFromResourceName(HInstance,SRC);
except
on E:Exception do
ShowMessage(e.Message);
end;
And then save the Bitmap to disk.
Maybe this might come in handy too if you need to work with the resources itself.
Delphidabbler / ResourceFiles
I want to check a file has a valid IMAGE_DOS_SIGNATURE (MZ)
function isMZ(FileName : String) : boolean;
var
Signature: Word;
fexe: TFileStream;
begin
result:=false;
try
fexe := TFileStream.Create(FileName, fmOpenRead or fmShareDenyNone);
fexe.ReadBuffer(Signature, SizeOf(Signature));
if Signature = $5A4D { 'MZ' } then
result:=true;
finally
fexe.free;
end;
end;
I know I can use some code in Windows unit to check the IMAGE_DOS_SIGNATURE. The problem is I want the fastest way to check IMAGE_DOS_SIGNATURE (for a big file). I need your some suggestion about my code or maybe a new code?
Thanks
The size of the file doesn't matter because your code only reads the first two bytes.
Any overhead from allocating and using a TFileStream, which goes through SysUtils.FileRead before reaching Win32 ReadFile, ought to be all but invisible noise compared to the cost of seeking in the only situation where it should matter, where you're scanning through hundreds of executables.
There might possibly be some benefit in tweaking Windows' caching by using the raw WinAPI, but I would expect it to be very marginal.