How to extract the first instance of unique strings - delphi-2010

Question

I need to extract a list of unique items from 12 years' worth of consistent computer-generated one-per day text files. The filenames vary only by the included date, so it is easy to generate the required name in code. They consist of a list of all the aircraft movements at my local airport during the given day, in time order. Naturally, the same aircraft come and go many times, and the objective is to loop through the files, pick out the first instance of when each individual aircraft appears (the first visit or FV) copy it to a list and then ignore it from then on. The result should be a list of all the first visits in date order. Should be simple, but... My program is small so I am including the entire implementation code.
procedure TForm1.FormCreate(Sender: TObject);
begin
FileDate := StrToDate('01/01/2007');
FName := 'E:LGW Reports/SBSLGW2007-01-01.txt'; //1st file to be read
FDStr := copy(FName, 21, 10);
TempList := TStringList.Create; //temp holder for file contents
FVCheckList := TStringList.Create; //holds unique identifier (UID)
FVCheckList.Sorted := TRUE;
FVCheckList.Duplicates := dupIgnore;
FVList:= TStringList.Create; //the main output
end;
procedure TForm1.Button1Click(Sender: TObject);
var
i: integer;
begin
Memo1.Lines.Append('Started');
Repeat
TempList.Clear;
TempList.LoadFromFile(FName);
for i := 1 to TempList.Count-1 do
begin
Line := TempList.Strings[i];
//create a //create a Unique identifier (UID) from elements in Line
Serial := Trim(Copy(Line, 22, 9));
MsnPos1 := Pos('[', Line) + 1;
MsnPos2 := Pos(']', Line);
Msn := copy(Line, MsnPos1, (MsnPos2 - MsnPos1));
UID := Serial + '/' + Msn;
//
if (FVCheckList.IndexOf(UID) < 0) then
begin
FVCheckList.Append(UID);
//Add date of file to Line, otherwise it gives no clue when FV was
FVList.Append(FormatDateTime('YYYY-MM-DD', FileDate) + ' ' + Line);
FileDate := IncDay(FileDate, 1);
FName := 'E:LGW Reports/SBSLGW' + FormatDateTime('YYYY-MM-DD', FileDate) + '.txt';
end;
end;
Until FileExists(FName) = FALSE;
FVCheckList.SaveToFile('E:LGW Reports/First Visit Checklist.txt');
FVList.SaveToFile('E:LGW Reports/First Visits.txt');
Memo1.Lines.Append('Finished');
Memo1.Lines.SaveToFile('E:LGW Reports/Files parsed.txt');
end;
procedure TForm1.FormClose(Sender: TObject; var Action: TCloseAction);
begin
TempList.Free;
FVCheckList.Free;
FVList.Free;
end;
There are no compiler errors, it runs to completion in seconds and produces the two text files specified, correctly formatted. The big problem is that the lines actually listed in FVList are not always the very first visit of the aircraft, they can be the first, the most recent or somewhere in between. I cannot see any obvious clue as to why the wrong instance is appearing: if my code is right, then something is wrong with the functioning of TStringList FVCheckList. The fault is far more likely to be something I have overlooked, or my understanding of how .dupIgnore works, or maybe my looping isn't working as it should.
I should be very grateful for any practical help. Many thanks in advance.

Answer 1

Repeat
...
Until FileExists(FName) = FALSE;
Should be
While FileExists(FName) = TRUE do
Begin
End;
If the first 2007-01-01 file does not exist, your code will crash on the first LoadFromFile() since you don't check for the file's existence before loading it, unlike with the subsequent files.
Otherwise, I would suggest sticking with repeat but assign FName at the top of each loop iteration instead of initializing it outside the loop and then reassigning at the bottom of each iteration. No need to duplicate efforts.
If you check IndexOf() manually, you don't need to use Sorted or dupIgnore at all. This is what you should be doing in this situation. When dupIgnore ignores a new string, Append() doesn't tell you that the string was ignored. To do that, you would have to check whether the Count was actually increased or not.
Inside the outer loop, the reassignment of FileDate and FName should be outside of the inner for loop,not inside the for loop at all.
Try this instead:
procedure TForm1.FormCreate(Sender: TObject);
begin
FileDate := EncodeDate(2007,1,1);
FDStr := FormatDateTime('YYYY-MM-DD', FileDate);
TempList := TStringList.Create; //temp holder for file contents
FVCheckList := TStringList.Create; //holds unique identifier (UID)
FVList := TStringList.Create; //the main output
end;
procedure TForm1.Button1Click(Sender: TObject);
var
i: integer;
begin
Memo1.Lines.Append('Started');
Repeat
FName := 'E:LGW Reports/SBSLGW' + FormatDateTime('YYYY-MM-DD', FileDate) + '.txt';
if not FileExists(FName) then Break;
Memo1.Lines.Append(FName)
TempList.LoadFromFile(FName);
for i := 1 to TempList.Count-1 do
begin
Line := TempList.Strings[i];
//create a Unique identifier (UID) from elements in Line
Serial := Trim(Copy(Line, 22, 9));
MsnPos1 := Pos('[', Line) + 1;
MsnPos2 := PosEx(']', Line, MsnPos1);
Msn := copy(Line, MsnPos1, (MsnPos2 - MsnPos1));
UID := Serial + '/' + Msn;
if FVCheckList.IndexOf(UID) = -1 then
begin
FVCheckList.Append(UID);
//Add date of file to Line, otherwise it gives no clue when FV was
FVList.Append(FormatDateTime('YYYY-MM-DD', FileDate) + ' ' + Line);
end;
end;
FileDate := IncDay(FileDate, 1);
end;
FVCheckList.SaveToFile('E:LGW Reports/First Visit Checklist.txt');
FVList.SaveToFile('E:LGW Reports/First Visits.txt');
Memo1.Lines.Append('Finished');
Memo1.Lines.SaveToFile('E:LGW Reports/Files parsed.txt');
end;
procedure TForm1.FormDestroy(Sender: TObject);
begin
TempList.Free;
FVCheckList.Free;
FVList.Free;
end;