Most appropriate data structure for a CSV table? - delphi

Question

I'm looking for an advice on most appropriate data structure for holding CSV(Comma Separated Value) table in a memory.
It should cover both cases: table with and without a header.
If the table contains a header, all fields of all rows are determined by key->value pairs, where the key is a name from a header and value is an appropriate content of a field.
If the table does not contain a header, then rows are simply lists of strings or also key->value pairs with key names generated (like 'COL1', 'COL2', ... 'COLn').
I'm looking for most simple (less code) and most generic solution at the same time.
I'm thinking about the following subclassing, but doubt if it's the right/effective way of implementation:
TCSV = class (TObjectList<TDictionary<string, string>>)
...
public
constructor Create(fileName: string; header: Boolean; encoding: string = '';
delimiter: Char = ';'; quoteChar: Char = '"'); overload;
...
end;
It looks like I have to keep keys for every row of fields. What about TDictionary<string, TStringList> ? Would it be a better solution ?

Answer 1

What about a TClientDataset? Seems quite easy.
Just a simple guide on how to use TClientDataSet as an in-memory dataset, can be found here.

Answer 2

The structure you are proposing would mean that you would have a TDictionary instance for every row in your csv file. In essence duplicating the column names for every row. Seems like a bit of a waste.
Assuming that with TDictionary<string, TStringList> you would fill each TStringList with the values from a single column. That could work, but it still won't be easy to iterate over all columns per row of data.
As GolezTrol suggests, TClientDataSet comes standard with Delphi, is very powerful and as a dataset intended to be used with columnar data. Also, although it is a dataset, it does not require a database (connection) and is used in many application for exactly the goal you are trying to achieve: an in-memory dataset.

Answer 3

I recommend you try the TJvCsvDataSet, which I wrote and contributed to the JEDI JVCL. It works on CSV files with and without headers. It works with data aware controls including DB Grids.
It parses CSV data, and works entirely like the Client Dataset that others have suggested.
Internally it uses an array of byte records and parses each row and keeps an integer "lookup" so that it knows where each individual column starts on that particular row. That makes changing out one value for another value (modifying a field in a row) a very fast operation.
It supports most common field types (although not blob or currency right now) and it parses CSV features including embedded carriage return + linefeeds that are inside a field value, and embedded CSV "escape codes" so that you can put a double quote character inside a string, for instance.
It has a property called FieldDef which can be used to define the types of the columns, or it can simply read the header of the file, and treat each value inside as a string (if you don't tell it otherwise).
It can modify a CSV by adding or removing columns, and do most common things you'd want to do with a CSV table. I have used it and tested it heavily, and it works fine.

Answer 4

Depending of the usage instead of TDataSet you may also use Synopse TSynBigTable which is more perfomant and has less limitations.
For no "time or size critical" applications TDataSet is OK.

Answer 5

So you basically want to be able to access elements like:
for RowNum := 0 to csv.Count - 1 do
begin
Name := csv[RowNum]['Name'];
// Do something
end;
TObjectList<TDictionary<string, string>> would certainly do the job but its not very efficient.
Loading the csv into a Dataset would probably be the least amount of code but would have slightly more overhead.
You might want to consider a combination of a simple Tstringlist or TList<string> for the header and break the data into a new class that takes header list in its constructor. You would get the same result:
TCSVRow = class
private
FHeaders: TList<string>;
FFields: TList<string>;
public
constructor(Headers: TList<string>);
function GetField(index: string): string;
property Fields[index: string]: string read GetField; default;
end;
TCSV = class
private
FHeaders: TList<string>;
FRows:TList<TCSVRow>;
public
function GetRow(Index: integer):TCSVRow;
property Rows[index: integer]:TCSVRow read GetRow; default;
end;
implementation
function TCSVRow.GetField(index: string): string;
begin
Result := FFields[FHeaders.IndexOf(index)];
end;
function TCSV.GetRow(Index:integer):TCSVRow;
begin
Result := FRows[Index];
end;
This is incomplete and I typed directly into the browser so I haven't tested it for correctness but you get the general idea. This way the header information is only stored once instead of duplicated for each row.
You could save a small bit of memory by making FFields a string array instead of a TList<string> but TList<string> is easier to work with IMHO.
Update
On second thought David has a point. The CSVRow class could be eliminated. You could simply have either TList<TList<string>> or a 2d array. Either way I still think you should keep the headers in a separate list. In which case TCSV would look more like:
TCSV = class
private
FHeaders: TList<string>;
FData:TList<TList<string>>;
public
function GetData(Row: integer; Column:string):string;
property Data[Row: integer; Column:string]:string read GetData; default;
end;
function TCSV.GetData(Row: integer; Column:string):string;
begin
Result := FData[Row][FHeaders.IndexOf(Column)];
end;

Answer 6

There are many possible solutions to this.
If you want something really simple and generic as per your request (not necessarily the fanciest solution), why not just...
TMyRec =
record
HeaderNames: array of string;
StringValues: array of array of string
end;
Just set the length of the arrays as needed (using SetLength).