Is it possible in Delphi to determine the size of a file as it is being copied? I get a notification when a file is first copied to a folder, but need to wait until the copy is complete before I can process the file.
I've used JclFileUtils.GetSizeOfFile(Filename) but that gives me the 'expected' file size, not the current filesize.
Regards, Pieter
Prompted by the first answer I decided to give up on trying to determine when a file copy has completed. Instead I found that using TFileStream gave me a reliable indication whether a file is in use or not.
function IsFileInUse(Filename: string; var ResultMessage: string): boolean;
var
Stream: TFileStream;
begin
Result := True;
ResultMessage := '';
try
Stream := TFileStream.Create(Filename, fmOpenRead or fmShareDenyWrite);
try
Result := False;
finally
FreeAndNil(Stream);
end;
Except on E: Exception do
ResultMessage := 'IsFileInUse: ' + E.Message
end;
end;
In this way I can keep on checking until the file is not in use anymore before attempting to process it.
It depends on the technique that is used by the copying function. Most copy-methods will allocate the disk space first before they start to copy a file. Thus, if you want to copy a file of 4 GB, the system starts by creating a file with random data for 4 GB in total. (Which is done lightning-fast, btw.) It then copies the data itself, but the file size is already what you expect.
This has as advantage that the sysmen can check if there's enough disk space available to actually copy the data.
If you write your own file copy function then you can have total control over how it does this. Else, you're limited to whatever the chosen copy-method offers you. So, how do you copy a file?
If you have control over the file copy process, it is easiest to have the copy routine create the file using a temporary filename, and when done, rename it to correct filename.
That way, you can use Windows folder monitoring to watch for the renaming (JCL contains a component to help with this, not sure about the name from here). When your code gets triggered you are sure the other side has finished writing the file.
A simple trick I used was to have the copying process create new files with a '$$$' extension. My code still got triggered for those but I ignored them until they were renamed to their proper filename.
Hope this helps.
Related
Using Delphi XE2 and the native TZipFile I attempt to extract the contents of a downloaded zip file (which contains 2 zipped XML files) and it always extracts zero byte files.
The file is being compressed by C# code like this:
var zipFile = new ZipFile();
foreach (Tuple<string, string> t in filesMeta) {
zipFile.AddFile(string.Format("{0}{1}", StaticVariables.WebServerFileStorage, t.Item2), "").FileName = t.Item1 + ".xml";
}
response.Clear();
response.ContentType = "application/zip";
zipFile.Save(response.OutputStream);
response.End();
The Delphi extraction code is this:
zipFile := TZipFile.Create;
try
filename := 'C:\test\57f52480-ec87-4169-a820-0a65bc4ad952.zip';
if zipFile.IsValid(filename) then begin
zipFile.Open(filename, zmRead);
zipFile.ExtractAll('C:\test\');
end;
finally
zipFile.Free;
end;
I even tried using a TStream as the source instead of a file on disk. That's actually what I want to do since the zip file is downloaded from a web server into a TStream. I tried to extract the data using the overloaded open method of TZipFile to open the stream.
That got me zero byte files so I saved the zip file to disk and tried to open the file from disk and extract. Same zero byte files are extracted.
I even tried using the class method to extract the files from the zip file on disk:
System.Zip.TZipFile.ExtractZipFile(filename, 'C:\Test\');
Same zero byte files extracted.
The zip file is valid and the 2 zipped XML files can be extracted properly by both Windows 7 native file handling and 7-Zip.
Now here is something nutty...
In desperation I tried to see what the ExtractToFile() procedure
David Heffernan came up with in this question about extracting a zip to a stream would do so I tried using it like this:
var x : integer;
var fileCount : integer;
var fileNames : TArray<string>;
begin
zipFile := TZipFile.Create;
try
filename := 'C:\test\57f52480-ec87-4169-a820-0a65bc4ad952.zip';
if zipFile.IsValid(filename) then begin
zipFile.Open(filename, zmRead);
fileCount := zipFile.FileCount;
fileNames := copy(zipFile.FileNames, 0, MaxInt);
zipFile.Close;
for x := 0 to fileCount-1 do begin
// Use David Heffernan's stream procedure
ExtractToFile(filename, x, 'C:\test\' + fileNames[x]);
end;
end;
finally
zipFile.Free;
end;
end;
And David's procedure extracts the files to disk as expected! WTF???
I am really confused why a convoluted method of extraction would work and the simple extraction method would not work. I'll use David's example if I have to but I'd prefer to get the normal extract working if possible.
Any ideas are appreciated.
Cheers!
TJ
I had this same problem.
The source code to TZipFile shows that the TStream passed into the Read function returns the entire zip file with position set to the start of the filename you're wanting. So don't rewind. Just copyfrom or do what you want with the TStream for the uncompressed length given in the TZipHeader.
ZipStream := TStream.Create;
ZipFile.Read(MyFileName, ZipStream, ZipHeader);
//leave ZipStream pointer where it is!!!
SomethingElse.LoadFromStream(ZipStream, ZipHeader.UncompressedSize);
ZipStream.Free;
In my opinion, TZipFile should really load the ZipStream with only what is requested. The way this is implemented is not intuitive without first going through the TZipFile source code.
TL;DR: The solution to my problem was an external component. Zip Forge (or Abbrevia)
Read on for details.
Nothing I tried except for the roundabout way of saving the file and re-opening it using David's function worked. While that would have worked, it was not optimal as it required me to save the downloaded file to disk first and reopen it for extract and then delete the zip file. My goal was to just open the downloaded stream and extract the files directly to disk. One write to disk and no temporary file.
We even tried two different C# libraries to zip the files and both gave the same results on the streamed data. The Delphi TZipFile component could not handle it.
It turns out we have a license to ZipForge which I had forgotten about since I had not used it in ages it and that handles the download stream from the C# web server and extracts the files successfully.
For reference, I also tried the Abbrevia component version 5.2 and that also successfully extracted the files from the stream.
Hopefully this will help someone else.
All the suggestions by David and Uwe were appreciated.
Cheers!
TJ
I'm doing an application(XE6 , Firemonkey) to synchronize files between a shared folder and a computer/s. This application checks every x hours if there are new files to be synchronized, and it starts on windows start-up.
I can do everything, my application starts on start-up, and it does the synchronization, as long as i'm the one starting it. Whem the application auto starts on start up it gives me an exception "EINOUTERROR" - File Access Denied.
On starting the application reads a small .txt file to set up it self (shared folder location, rate of synchronization etc), my guess is that since its the windows starting the app runs it without privileges to read the .txt, but even after changing the .txt permissions to full control on everyone it gives the same error.
File open code:
AssignFile(myFile,'Dados.txt');
if FileExists('Dados.txt') then
Append(myFile)
else
Rewrite(myFile);
FileMode := fmOpenRead;
Reset(myFile);
Code of placing the app on startup programs :
procedure TSyncM.RunOnStartup(const sCmdLine: string; bRunOnce: boolean; Remove: Boolean) ;
var sKey: string;
Section: string;
const ApplicationTitle = 'GEN4Sync';
begin
if (bRunOnce) then
sKey := 'Once'
else
sKey := '';
Section := 'Software\Microsoft\Windows\CurrentVersion\Run' + sKey + #0;
with TRegIniFile.Create('') do
try
RootKey := HKEY_CURRENT_USER;
if Remove then
DeleteKey(Section, ApplicationTitle)
else
WriteString(Section, ApplicationTitle, sCmdLine) ;
finally
Free;
end;
end;
If i comment the piece of code that calls the reading of that .txt my app starts and executes well, but i don't want to set it up everytime.
Thanks in advance
I think that the issue is related to your use of relative paths. You have written the code under the assumption that the working directory is the same directory as contains the executable. That is not necessarily so.
When you start the application by double clicking on the executable file, for instance, the shell ensures that the initial working directory is the directory containing the executable file. However, when Windows starts your program at startup I suspect that the working directory is the system directory. And of course your file is not found there, and you don't have rights to write there.
Instead of using relative paths, use the full path to the file.
FileName := ExtractFilePath(ParamStr(0)) + 'Dados.txt';
Or perhaps
FileName := TPath.Combine(ExtractFilePath(ParamStr(0)), 'Dados.txt');
Note that this does also assume that your executable file is located in a folder which you can write to. That is often not the case so you may need to find a different location.
I do have to comment that I find it somewhat incongruous that you are mixing the very modern (FireMonkey) with the ancient (Pascal I/O). Perhaps it is time to move to a more modern I/O technique.
I have a function which works ok to check if a local file is in use.
However if I map a network drive and try to check if a file from the mapped drive is in use then the result of the function is always false.
I need to wait before a large file is being copied to the mapped drive and after completion I rename the file.
If the file in not in use then i start performing various actions else i wait another minute and check again.
How can I modify the function below in order to work with mapped drive files that are constantly copied?
Thank you
function IsFileInUse(FileName: TFileName): Boolean;
var
HFileRes: HFILE;
begin
Result := False;
if not FileExists(FileName) then
begin
showmessage('Fisierul "'+Filename+'" nu exista!');
Exit;
end
else
begin
HFileRes := CreateFile(PChar(FileName),
GENERIC_READ or GENERIC_WRITE,
0,
nil,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
0);
Result := (HFileRes = INVALID_HANDLE_VALUE);
if not Result then
CloseHandle(HFileRes);
end;
end;
What you are claiming is that CreateFile succeeds in opening a file in exclusive mode whilst another party is writing to the file. Possible explanations:
You have made a mistake with the file name and the file you are opening is not the one in use.
The other party is writing to the file without having locked it. In other words it opened the file with a share mode that allowed other parties to read and write. This would be quite unusual. If this is the case then you need to fix the other process.
The remote file server is broken and fails to respect locks. I'd regard this as quite unlikely.
I think the final option can be rejected immediately. Option 2 seems rather unlikely. Which leaves option 1. You are able to lock the file because it is not locked.
I'd also comment that the function is spurious. You can remove it. Simply attempt whatever operation you need to perform. If that operation fails due to a sharing violation you know that the file was locked. Consider also the race condition in any code using that function. The fact that a file is unlocked now does not prevent another party locking the file before you can do anything with it.
HI
We have a large number of remote computers that capture video onto disk drives. Each camera has it's own unique directory and there can be up to 16 directories on any one disk.
I'm trying to locate the oldest video file on the disk but using FindFirst/FindNext to compare the File Creation DateTime takes forever.
Does anybody know of a more efficient way of finding the oldest file in a directory? We remotely connect to the pc's from a central HO location.
Regards, Pieter
-- Update
Thank you all for the answers. In the end I used the following.
Map a drive ('w:') to the remote computer using windows.WNetAddConnection2
//Execute dir on the remote computer using cmd.exe /c dir
//NOTE: Drive letters are relative to the remote computer. (psexec -w parameter)
psexec \\<IPAddress> -i /accepteula -w "c:\windows\system32" cmd.exe "/c dir q:\video /OD /TC /B > q:\dir.txt"
//Read the first line of "w:\dir.txt" to get the oldest file in that directory.
//Disconnect from the remote computer using windows.WNetCancelConnection2
You could also try FindFirstFileEx with FindExInfoBasic parameter, and on Windows 7 or Server 2008 R2 or later, FIND_FIRST_EX_LARGE_FETCH which should improve performance.
First, grab the RunDosAppPipedToTStrings routine from this page on how to run a DOS program and pipe its output to a TStrings. The example uses a TMemo's Lines property, but you can pass any TStrings in, such as TStringList. Note that this will fail silently if CreateProcess returns false. You might want to add an else case to the "if CreateProcess" block that raises an exception.
Then create a simple batch file in the same folder as your EXE. Call it getdir.bat. All it should say is:
dir %1
This produces a directory listing of whatever folder you pass to it. Unfortunately, "dir" is a DOS keyword command, not a program, so you can't invoke it directly. Wrapping it in a batch file gets around that. This is a bit of a hack, but it works. If you can find a better way to run DIR, so much the better.
You'll want to invoke RunDosAppPipedToTStrings with code that looks something like this:
procedure GetDirListing(dirname: string; list: TStringList);
const
CMDNAME = '%s\getdir.bat "%s"';
var
path: string;
begin
list.Clear;
path := ExcludeTrailingPathDelimiter(ExtractFilePath(ParamStr(0)));
RunDosAppPipedToTStrings(format(CMDNAME, [path, dirname]), list, false);
end;
Then all that's left to do is parse the output, extract date and time and filenames, sort by date and time, and grab the filename of the file with the lowest date. I'll leave that much to you.
If you can run something on the remote computer that can iterate over the directories, that will be the fastest approach. If you wanted to use Mason's example, try launching it with PsExec from SysInternals.
If you can only run an application locally then no, there's no faster way than FindFirst/FindNext, and anything else you do will boil down to that eventually. If your local computer is running Windows 7 you can use FindFirstFileEx instead, which has flags to indicate it should use larger buffers for the transfers and that it shouldn't read the 8.3 alias, which can help the speed a bit.
I had almost the same problem on the fax server software I developed. I had to send the faxes in the order they were received from thousands (all stored in a directory). The solution I adopted (which is slow to start but fast to run) is to make a sorted list of all the files using the
SearchRec.Time
as the key. After the file is in the list, I'm setting the attributes of the file as a faSysFile:
NewAttributes := Attributes or faSysFile;
Now when I do a new search with
FileAttrs := (faAnyFile and not faDirectory);
only the files that are not faSysFile are shown, so I can add to the list the files that are coming in new.
Now you have a list with all the files sorted by time.
Don't forget, when you start your application, first step is to remove the faSysFile attribute from the files in the folder so they can be processed again.
procedure FileSetSysAttr(AFileName: string);
var
Attributes, NewAttributes: Word;
begin
Attributes := FileGetAttr(AFileName);
NewAttributes := Attributes or faSysFile;
FileSetAttr(AFileName, NewAttributes);
end;
procedure FileUnSetSysAttr(AFileName: string);
var
Attributes, NewAttributes: Word;
begin
Attributes := FileGetAttr(AFileName);
NewAttributes := Attributes and not faSysFile;
FileSetAttr(AFileName, NewAttributes);
end;
procedure PathUnSetSysAttr(APathName: string);
var
sr: TSearchRec;
FileAttrs: Integer;
begin
FileAttrs := (faAnyFile and not faDirectory) and (faAnyFile or faSysFile);
APathName := IncludeTrailingBackslash(APathName);
if SysUtils.FindFirst(APathName + '*.*', FileAttrs, sr) = 0 then
try
repeat
if (sr.Attr and faDirectory) = 0 then
FileUnSetSysAttr(APathName + sr.Name);
until SysUtils.FindNext(sr) <> 0;
finally
SysUtils.FindClose(sr);
end;
end;
I know this is not the best solution, but works for me.
I have a problem adding strings to a TStringList. I've searched other posts but couldn't find an answer to this.
What I'm trying to do is to add a big amount of strings to a TStringList (more than 14000) but somewhere in the process I get an EAccessViolation. Here's the code I'm using:
procedure TForm1.FormCreate(Sender: TObject);
begin
List := TStringList.Create;
List.Duplicates := dupAccept;
end;
procedure TForm1.ButtonStartClick(Sender: TObject);
begin
List.Clear;
List.Add('125-AMPLE');
List.Add('TCUMSON');
List.Add('ATLV 4300');
List.Add('150T-15');
List.Add('TDL-08ZE');
List.Add('RT20L');
List.Add('SIN LINEA');
List.Add('TIARA');
List.Add('FL200ZK1');
List.Add('FL250ZK1');
List.Add('SIN LINEA');
List.Add('CENTAURO-70 S.P.');
List.Add('CORSADO');
{ This list continues to about 14000 strings...}
List.Add('VOSJOD 2');
List.Add('Z 125');
List.Add('ZUMY');
List.Add('NEW AGE 125');
List.Add('SIN LINEA');
end;
procedure TForm1.FormClose(Sender: TObject; var Action: TCloseAction);
begin
FreeAndNil(List);
end;
¿What's wrong with this code? The list contains duplicate strings so I set the Duplicates property to dupAccept. I was able to load the list using LoadFromFile, but I don't want to have a text file outside my application.
I hope you can help me!!! Please tell me if you need any further information.
Thank you very much. I really appreciate your help.
The suggestions for using an external file are on the mark here. However, your post indicates your desire for not having an external file. I would then suggest you link the file to the executable as a resource. You can easily do this following these steps:
Place all the strings into a text file called stringdata.txt (or whatever name you choose). Then create a .rc file of whatever name you choose and put the following into it (STRING_DATA can be any identifier you choose):
STRING_DATA RCDATA "stringdata.txt"
Create a .res file from the .rc:
BRCC32 <name of rc>.rc
Now reference this file from the source code. Place the following someplace in the unit:
{$R <name of res>.res}
Instead of loading from a file stream, load from a resource stream:
StringData := TResourceStream.Create(HInstance, 'STRING_DATA', RT_RCDATA);
try
List.LoadFromStream(StringData);
finally
StringData.Free;
end;
If you do command-line automated builds, I would suggest you keep the .rc file under source control and build the .res during the build process. This way you can also keep the stringdata.txt file under source control and any edits are automatically caught on the next build without having to explicitly build the .res file each time the .txt file changes.
What Delphi version are you using? Some older versions had a bug in the memory manager that can cause an access violation when trying to reallocate an array to a size that's too large.
Try adding FastMM4 to your project to replace the old memory manager and see if that helps.
Also, you're probably better off keeping the list in an external file. Yes, it's another file, but it also means that you can change the list without having to recompile the entire program. This also makes creating (and distributing!) updates easier.
Mason is probably right for the cause of the AV; this is quite a large array to grow.
On a side note, when doing such a long processing on a StringList, it's recommended to surround it by BeginUpdate/EndUpdate to avoid firing any update event.
Even if you don't have any now, they might be added later and you'll get problems.
Set list.capacity to the number of items you plan to add, immediately after you create the list. Alternatively, place the list in an RC file (named other than with the name of your project) and add it to your project. This gets compiled into your application, but does not involve executable code to create the list.
I would also worry about compiler integrity with a 14,000 line procedure. People have found other cases where going beyond anything reasonable breaks the compiler in various ways.
You may also want to try THashedStringList, could see a speed boost (although not in this function), although I'm not sure if the add method is a whole lot different.
try using the following instead of your code to add the strings to the StringList
var
Str: string;
begin
Str := '125-AMPLE' + #13#10;
Str := Str + 'TCUMSON' + #13#10;
Str := Str + 'ATLV 4300' + #13#10;
Str := Str + '150T-15' + #13#10;
................
List.Text := Str;
end;