Direct file access in delphi - delphi

Question

my application opens files does transformations and saves the data out to another file..or possible the same file.. the file size changes but i dont know how big or small its gona be untill i see the data inside the first file..
At the moment i load the file into a dynamic array do all that i need to do in there then save it back... this was looking good untill i got to my testing stage where i found transforming multi gigabyte files on a system with 128mb ram caused some issues...LOL
here is my code..
procedure openfile(fname:string);
var
myfile: file;
filesizevalue:integer;
begin
AssignFile(myfile,fname);
filesizevalue := GetFileSize(fname);
Reset(myFile, 1);
SetLength(dataarray, filesizevalue);
BlockRead(myFile, dataarray[0], filesizevalue);
CloseFile(myfile);
end;
what i need is direct file access to minimise ram usage.. thats what i think i need/
is this what i need can it be done in delphi

Answer 1

I'd look at using a TFileStream, perhaps one with buffering, but you need to show what you are doing with the data really because it is hard to determine the best strategy. As gabr says, one option is memory mapped files, the code of which is in his link but since it is my code, I'll add it here too!
procedure TMyReader.InitialiseMapping(szFilename : string);
var
// nError : DWORD;
bGood : boolean;
begin
bGood := False;
m_hFile := CreateFile(PChar(szFilename), GENERIC_READ, 0, nil, OPEN_EXISTING, 0, 0);
if m_hFile <> INVALID_HANDLE_VALUE then
begin
m_hMap := CreateFileMapping(m_hFile, nil, PAGE_READONLY, 0, 0, nil);
if m_hMap <> 0 then
begin
m_pMemory := MapViewOfFile(m_hMap, FILE_MAP_READ, 0, 0, 0);
if m_pMemory <> nil then
begin
htlArray := Pointer(Integer(m_pMemory) + m_dwDataPosition);
bGood := True;
end
else
begin
// nError := GetLastError;
end;
end;
end;
if not bGood then
raise Exception.Create('Unable to map token file into memory');
end;

Answer 2

You can also map parts of file directly into memory. That's definitely the most direct way. See What is the fastest way to Parse a line in Delphi for an example.

Answer 3

If problem permits, you can use BlockRead and BlockWrite to read chunk of input file, process it and then write that chunk to output file. Something like this:
AssignFile(inFile,inFname);
AssignFile(outFile,outFname);
repeat
BlockRead(inFile, buff, SizeOf(buff), bytesRead);
ProcessBuffer(buff);
BlockWrite(outFile, buff, bytesRead, bytesWritten);
until (bytesRead = 0) or (bytesWritten <> bytesRead);
Code presumes that you won't change size of buffer while processing it. If size of file changes, then you should change last two lines of example code.

Answer 4

I prefer to use tFileStream for this kind of processing. In this example I am assuming there is a constant ArraySize which is set to the size of a single array element. For example, if your "array" is an array of integer then it would be set to:
ArraySize := SizeOf( Integer );
which would set the ArraySize to 4.
Function LoadPos(inFIlename:string;ArrayPos:Int64;var ArrayBuff) : boolean;
var
fs : tFileStream;
begin
result := false;
fs := tFileStream.Create(inFilename,fmOpenRead);
try
// seek to the array position
fs.Seek( ArrayPos * ArraySize, soFromBeginning);
// load the element
result := fs.Read( ArrayBuff, ArraySize ) = ArraySize;
finally
fs.free;
end;
end;
The only problem with this approach is it only works for fixed size structures, variable length strings require a different approach.

Answer 5

I don't think you'll get a "more direct" file access. Do you use all the data in the file? Otherwise you could perhaps use a stream and load only the data needed into memory. But if you use all the data, there's only one solution IMHO: read the file in chunks. But that highly depends on the kind of transformation you want to apply. If the transformation is not local (so that combined data elements are all in the same chunk), you gonna have problems.