hi was reading the following tutorial: http://www.forumkorner.com/thread-26894.html, the problem is that I find way to how to translate the following code (VB .net):
Dim file1 = File.OpenWrite("c:/test.exe")
Dim siza = file1.Seek(0, SeekOrigin.[End])
Dim size = Convert.ToInt32("20")
Dim bite As Decimal = size * 1048576
While siza < bite
siza += 1
file1.WriteByte(0)
End While
file1.Close()
End If
the question is where do I get the information on how to write bytes to a file ?
It is clearly that the tutoraial that OP refers to was written by some newbie so I wouldn't folow it.
Now if you wanna make program which simply increases the size of certain file there are easier ways to do it. You don't necessarily need to write into the file in order to increase its size.
This code does it without writing a single byte into it:
procedure TForm2.Button1Click(Sender: TObject);
var FS: TFileStream;
begin
if OpenDialog1.Execute then
begin
FS := TFileStream.Create(OpenDialog1.FileName,fmOpenReadWrite);
FS.Size := FS.Size *2;
FS.Free;
end;
end;
How? It simply tels the operating system that the size of that specific file needs to be set to specific value (usually used to reserve the disk space before writing large amounts of data into file) and operating system then only changes the information in the file allocation table.
Now biggest advantage of this is that on hard drive this procedure will be done almost instantly regardles of the desired file size while the approach shown in the tutorail OP refered to takes more time for larger files. If the file is on a Flash drive it could take some time due the fact that most flash drives use FAT32 partitions which works a bit different thatn NTFS and therefore require more data to be written into the allocation tables.
EDIT: WARNING! Never and I mean NEVER set the file size smaller that it currently is! If you do so you will cause the los of data which probably won't be posible to repair.
Related
I need to find a certain string, in a text file, from bottom (the end of the line).
Once the string has been found, the function exit.
Here is my code, which is working fine. But, it is kind of slow.
I meant, I run this code every 5 seconds. And it consumes about 0.5% to 1% CPU time.
The text file is about 10 MB.
How to speed this up? Like, really fast and it doesn't consume much CPU time.
function TMainForm.GetVMem: string;
var
TS: TStrings;
sm: string;
i: integer;
begin
TS := TStringList.Create;
TS.LoadFromFile(LogFileName);
for i := TS.Count-1 downto 0 do
begin
Application.ProcessMessages;
sm := Trim(TS[i]);
if Pos('Virtual Memory Total =', sm) > 0 then
begin
Result := sm;
TS.Free;
exit;
end;
end;
Result := '';
TS.Free;
end;
You can use a TMemoryStream and use LoadFromFile to load the complete file content.
The you can cast the property Memory to either PChar or PAnsiChar or other character type depending on file content.
When you have the pointer, you can use it to check for content. I would avoid using string handling because it is much slower than pointer operation.
You can move the pointer from the end of memory (use Stream.Size) and search backward for the CR/LF pair (or whatever is used as line delimiter). Then from that point check for the searched string. If found, you are done, if not, loop searching the previous CR/LF.
That is more complex than the method you used but - if done correctly - will be faster.
If the file is to big to fit in memory, specially in a x32 application, you'll have to resort to read the file line by line, keeping only one line, going to the end of file. Each time you find the searched string, then save his position. At the end of file, this saved position - if any - will be the last searched file.
If the file is really very large, and the probability that the searched string is near the end, you may read the file backward (Setting TStream.position to have direct access). block by block. Then is each block, use the previous algorithm. Pay attention that the searched string may be split in two blocks depending on the block size.
Again, depending on the file size, you may split the search in several parallel searches using multi threading. Do not create to much thread neither. Pay attention that the searched string may be split in two blocks assigned to different threads depending on the block size.
I want to convert a 2GB file to an array of byte with Delphi. I use this function, then load file into memory Stream to get bytes. But I get error "Out of memory". How I can solve this problem?
type
TByteArray = Array of Byte;
function StreamToByteArray(Stream: TStream): TByteArray;
begin
// Check stream
if Assigned(Stream) then
begin
// Reset stream position
Stream.Position:=0;
// Allocate size
SetLength(result, Stream.Size);
// Read contents of stream
Stream.Read(result[0], Stream.Size);
end
else
// Clear result
SetLength(result, 0);
end;
//////then in button control i use:
var
strmMem: TMemoryStream;
bytes: TByteArray;
begin
strmMem:=TMemoryStream.Create;
if OpenDialog1.Execute then
strmMem.LoadFromFile(OpenDialog1.FileName);
bytes:=StreamToByteArray(strmMem);
strmMem.Free;
A 32 bit process has a total of 4GB of address space. Unless it has the large address aware flag available, only 2GB of that address space is available to it.
You are attempting to load a 2GB file into memory, in a contiguous block of address space. There is no chance of you being able to succeed. Even with a large address aware 4GB address space there's little hope for you finding a contiguous 2GB block of address space.
Furthermore, you are also attempting to read the file into memory twice, so you actually need two 2GB contiguous blocks. One for the stream, and one for the array. This is a result of you using the memory stream anti-pattern as described below.
Some options:
Switch to a 64 bit process, or
load the entire file, but in discontinuous blocks, or
process the file piece by piece, in smaller chunks.
Regarding the use of a memory stream, this is a recurring anti-pattern. I'd say that >90% of the uses of memory streams that we see here in the Delphi Stack Overflow tag are needless and wasteful.
The mistake is to load into memory just to be able to copy to some other memory. You are trying to read the file into an array. So read it directly into an array. The memory stream is pointless. Use a file stream. Read from the file stream into the array. That way you only load a single copy of the file into memory.
Of course, you'll sill struggle to put a 2GB file into memory even with that change, but you should still aim to hold only one copy of the data in memory.
I would like to program productive and keep my FileSize very small.
However I would like to know a few tips how do accomplish that.
For example what is better for a small FileSize:
Either:
if .... = '1' then begin
...
end;
or:
if ..... = inttostr(1) then begin
...
end;
or:
if .... = inttostr($0001) then begin
...
end;
or:
case of intvar
1: ...
2: ...
end;
Then there is something that I tried and I was surprised.
I made another Unit in my project that stores Strings as constants and then I use the constant vars to replace the strings in my project. For some reason this raises my FileSize although I replace double used Strings as a var now.
Also is it better to store stuff in vars than put them directly into the code?!
For example:
Function1(param1, param2, param3); // this code is used about 20 times in my project
or is it better if I:
Avar = Function1 (param1,param2,param3); // Store this once in a var and then replace it
And what about:
if ... = TRUE
or:
if ....
Same as:
if .... = FALSE
or:
if not(...)...
Any other tips to program productive for a smaller FileSize?
Thanks in advance.
I use Delphi7
I'm sorry to be blunt, but you are putting the cart before the horse.
If you really want to know how to make your executable smaller without already knowing what differences will result from your code variations in your given examples, you should just stop right now and read/learn/practice until you know more about the language and the compiler.
Then you'll understand that your question makes little sense per se, as you can already see by all the pertinent comments you got.
the exact same source code can result in vastly different executables and different source code can result in the same executable, depending on the compiler options/optimizations
if your main goal is to micro-manage/control the generated exe, program directly in assembler.
if you want to know what is generated by the compiler, learn how to use the CPU View.
program for correctness first, then readability/maintainability
only then, if needed (implies using correct metrics/tools), you can optimize for speed, memory usage or file size (probably the least useful/needed)
Long time ago, i tried to make a program as small as possible, because it had to fit onto a floppy disk (yes i know i'm a dinosaur). This Splitter tool was written in Delphi and is about 50KB in size.
To get it this small, it was necessary to do without a lot of Delphi's units (especially the Forms unit and all units with references to it). The only way, was to use the Windows-API directly for the GUI, and i can't think of a reason to do this nowadays, but out of interest.
As soon as you use the VCL, the exe size will grow much more, than all micro optimizations in your code together can reduce it.
I was ready to develop an internal communication protocol, and tried to use XML or JSON as a serialization mechanism, but the text mode is less efficient, and lead to a large volume of packets. So, I hope to use a binary serialization encoding mechanism. However, I'm looking for a long time, could not find cross-language, Delphi-supported mechanism.
Google Protocol Buffers and MessagePack are the most efficient schemes around, unfortunately there very few ports to Delphi at this moment.
Google Protocol Buffers http://sourceforge.net/projects/protobuf-delphi/
If you are willing to implement for yourself (messagepack is really simple), I suggest you hack write() and parse() functions of existing libraries like SuperObject.
You will end up with a very decent library without much effort.
Apache Thrift supports Delphi XE and 2010.
We've implemented an optimized binary format for records and arrays serialization. You can also serialize any object of memory structure easily. It is optimized for speed and used space.
It is part of our mORMot Open Source project, works from Delphi 5 up to XE2. You do not have to use the whole ORM / Client-Server services features of the project, just the SynCommons.pas unit.
You can then use our SynLZ real-time compression format to make the resulting content even smaller.
See this blog article and the associated source code.
It has more features than serialization (i.e. sorting, finding, hashing, slicing, reversing...).
It can be used with TFileBufferWriter/TFileBufferReader classes to create any custom format, with variable-length encoding of integers, and some other optimizations.
For instance, we use this serialization to store the .map file of all symbols into a .mab binary format: it uses some TDynArray instances + SynLZ. For a .map text file of 4.44 MB, it creates a .mab of 378 KB. See TSynMapFile.SaveToStream and other.
We use this very same format for persistence of in-memory list of objects (see TSQLRestServerStaticInMemory class in SQLite3Commons.pas). For instance, a 502 KB People.json content is stored into a 92 KB People.data binary file.
Just a code snippet:
function TSQLRestServerStaticInMemory.SaveToBinary(Stream: TStream): integer;
var W: TFileBufferWriter;
MS: THeapMemoryStream;
IDs: TIntegerDynArray;
i, n, f: integer;
begin
result := 0;
if (self=nil) or (Stream=nil) then
exit;
MS := THeapMemoryStream.Create;
W := TFileBufferWriter.Create(MS);
try
// primitive magic and fields signature for file type identification
W.Write(RawUTF8(ClassName));
W.Write(StoredClassProps.SQLTableName);
n := Length(StoredClassProps.FieldsName);
W.WriteRawUTF8DynArray(StoredClassProps.FieldsName,n);
W.Write(pointer(StoredClassProps.FieldType),sizeof(TSQLFieldType)*n);
// write IDs
SetLength(IDs,Count);
with fValue do
for i := 0 to Count-1 do
IDs[i] := TSQLRecord(List[i]).fID;
W.WriteVarUInt32Array(IDs,Count,wkSorted); // efficient ID storage
// write content, grouped by field (for better compression)
for f := 0 to High(fStoredClassProps.Fields) do
with fStoredClassProps.Fields[f]^, fValue do
for i := 0 to Count-1 do
GetBinary(TSQLRecord(List[i]),W);
W.Flush;
result := StreamSynLZ(MS,Stream,TSQLRESTSERVERSTATICINMEMORY_MAGIC);
finally
W.Free;
MS.Free;
end;
end;
BEncode?
Here is the Delphi source:
http://www.torry.net/quicksearchd.php?String=bencode&Title=Yes
Wikipedia writes about it:
http://en.wikipedia.org/wiki/Bencode
There are sourcecodes there for other languages too.
My application builds many objects in memory based on filenames (among other string based information). I was hoping to optimise memory usage by storing the path and filename separately, and then sharing the path between objects in the same path. I wasn't trying to look at using a string pool or anything, basically my objects are sorted so if I have 10 objects with the same path I want objects 2-10 to have their path "pointed" at object 1's path (eg object[2].Path=object[1].Path);
I have a problem though, I don't believe that my objects are in fact sharing a reference to the same string after I think I am telling them to (by the object[2].Path=object[1].Path assignment).
When I do an experiment with a string list and set all the values to point to the first value in the list I can see the "memory conservation" in action, but when I use objects I see absolutely no change at all, admittedly I am only using task manager (private working set) to watch for memory use changes.
Here's a contrived example, I hope this makes sense.
I have an object:
TfileObject=class(Tobject)
FpathPart: string;
FfilePart: string;
end;
Now I create 1,000,000 instances of the object, using a new string for each one:
var x: integer;
MyFilePath: string;
fo: TfileObject;
begin
for x := 1 to 1000000 do
begin
// create a new string for every iteration of the loop
MyFilePath:=ExtractFilePath(Application.ExeName);
fo:=TfileObject.Create;
fo.FpathPart:=MyFilePath;
FobjectList.Add(fo);
end;
end;
Run this up and task manager says I am using 68MB of memory or something. (Note that if I allocated MyFilePath outside of the loop then I do save memory because of 1 instance of the string, but this is a contrived example and not actually how it would happen in the app).
Now I want to "optimise" my memory usage by making all objects share the same instance of the path string, since it's the same value:
var x: integer;
begin
for x:=1 to FobjectList.Count-1 do
begin
TfileObject(FobjectList[x]).FpathPart:=TfileObject(FobjectList[0]).FpathPart;
end;
end;
Task Manager shows absouletly no change.
However if I do something similar with a TstringList:
var x: integer;
begin
for x := 1 to 1000000 do
begin
FstringList.Add(ExtractFilePath(Application.ExeName));
end;
end;
Task Manager says 60MB memory use.
Now optimise with:
var x: integer;
begin
for x := 1 to FstringList.Count - 1 do
FstringList[x]:=FstringList[0];
end;
Task Manager shows the drop in memory usage that I would expect, now 10MB.
So I seem to be able to share strings in a string list, but not in objects. I am obviously missing something conceptually, in code or both!
I hope this makes sense, I can really see the ability to conserve memory using this technique as I have a lot of objects all with lots of string information, that data is sorted in many different ways and I would like to be able to iterate over this data once it is loaded into memory and free some of that memory back up again by sharing strings in this way.
Thanks in advance for any assistance you can offer.
PS: I am using Delphi 2007 but I have just tested on Delphi 2010 and the results are the same, except that Delphi 2010 uses twice as much memory due to unicode strings...
When your Delphi program allocates and deallocates memory it does this not by using Windows API functions directly, but it goes through the memory manager. What you are observing here is the fact that the memory manager does not release all allocated memory back to the OS when it's no longer needed in your program. It will keep some or all of it allocated for later, to speed up later memory requests in the application. So if you use the system tools the memory will be listed as allocated by the program, but it is not in active use, it is marked as available internally and is stored in lists of usable memory blocks which the MM will use for any further memory allocations in your program, before it goes to the OS and requests more memory.
If you want to really check how any changes to your programs affect the memory consumption you should not rely on external tools, but should use the diagnostics the memory manager provides. Download the full FastMM4 version and use it in your program by putting it as the first unit in the DPR file. You can get detailed information by using the GetMemoryManagerState() function, which will tell you how much small, medium and large memory blocks are used and how much memory is allocated for each block size. For a quick check however (which will be completely sufficient here) you can simply call the GetMemoryManagerUsageSummary() function. It will tell you the total allocated memory, and if you call it you will see that your reassignment of FPathPart does indeed free several MB of memory.
You will observe different behaviour when a TStringList is used, and all strings are added sequentially. Memory for these strings will be allocated from larger blocks, and those blocks will contain nothing else, so they can be released again when the string list elements are freed. If OTOH you create your objects, then the strings will be allocated alternating with other data elements, so freeing them will create empty memory regions in the larger blocks, but the blocks won't be released as they contain still valid memory for other things. You have basically increased memory fragmentation, which could be a problem in itself.
As noted by another answer, memory that is not being used is not always immediately released to the system by the Delphi Memory Manager.
Your code guarantees a large quantity of such memory by dynamically growing the object list.
A TObjectList (in common with a TList and a TStringList) uses an incremental memory allocator. A new instance of one of these containers starts with memory allocated for 4 items (the Capacity). When the number of items added exceeds the Capacity additional memory is allocated, initially by doubling the capacity and then once a certain number of items has been reached, by increasing the capacity by 25%.
Each time the Count exceeds the Capacity, additional memory is allocated, the current memory copied to the new memory and the previously used memory released (it is this memory which is not immediately returned to the system).
When you know how many items are to be loaded into one of these types of list you can avoid this memory re-allocation behaviour (and achieve a significant performance improvement) by pre-allocating the Capacity of the list accordingly.
You do not necessarily have to set the precise capacity needed - a best guess (that is more likely to be nearer, or higher than, the actual figure required is still going to be better than the initial, default capacity of 4 if the number of items is significantly > 64)
Because task manager does not tell you the whole truth. Compare with this code:
var
x: integer;
MyFilePath: string;
fo: TfileObject;
begin
MyFilePath:=ExtractFilePath(Application.ExeName);
for x := 1 to 1000000 do
begin
fo:=TfileObject.Create;
fo.FpathPart:=MyFilePath;
FobjectList.Add(fo);
end;
end;
To share a reference, strings need to be assigned directly and be of the same type (Obviously, you can't share a reference between UnicodeString and AnsiString).
The best way I can think of to achieve what you want is as follow:
var StrReference : TStringlist; //Sorted
function GetStrReference(const S : string) : string;
var idx : Integer;
begin
if not StrReference.Find(S,idx) then
idx := StrReference.Add(S);
Result := StrReference[idx];
end;
procedure YourProc;
var x: integer;
MyFilePath: string;
fo: TfileObject;
begin
for x := 1 to 1000000 do
begin
// create a new string for every iteration of the loop
MyFilePath := GetStrReference(ExtractFilePath(Application.ExeName));
fo := TfileObject.Create;
fo.FpathPart := MyFilePath;
FobjectList.Add(fo);
end;
end;
To make sure it has worked correctly, you can call the StringRefCount(unit system) function. I don't know in which version of delphi that was introduced, so here's the current implementation.
function StringRefCount(const S: UnicodeString): Longint;
begin
Result := Longint(S);
if Result <> 0 then
Result := PLongint(Result - 8)^;
end;
Let me know if it worked as you wanted.
EDIT: If you are afraid of the stringlist growing too big, you can safely scan it periodically and delete from the list any string with a StringRefCount of 1.
The list could be wiped clean too... But that will make the function reserve a new copy of any new string passed to the function.