Unicode problems with Base64 strings and Streams - delphi

Question

I have been working on a project in Lazarus and have decided to move it to Delphi XE for the time being (due to some limitations).
A brief overview of what is going on:
At runtime I am loading external files and adding them to streams. The streams belong to several different classes that descend from one main object (TObject). These classes are added to a TList from the main object, basically each class has its own stream property and the class is child to the main object.
In this main object I have a save and load procedure:
When saving the object it also saves all the stream data from the other classes to file by using string to stream. The output string here must be base64 encoded as I am saving to XML.
When opening the file, the idea is to decode the base64 string and move it back into the streams just as if it were the original file before it was base64 encoded.
In Lazarus it works, and here is the important code (note, some of it was not written by me).
const
Keys64 = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+/';
function Encode64String(S: string): string;
function Decode64String(S: string): string;
function Encode64StringToStream(const Input: TStream; var Output: string): Boolean;
procedure Decode64StringToStream(const Input: string; Output: TStream);
procedure StringToStream(Stream: TStream; const S: string);
function StreamToString(MS: TMemoryStream): string;
implementation
function Encode64String(S: string): string;
var
i: Integer;
a: Integer;
x: Integer;
b: Integer;
begin
Result := '';
a := 0;
b := 0;
for i := 1 to Length(s) do
begin
x := Ord(s[i]);
b := b * 256 + x;
a := a + 8;
while a >= 6 do
begin
a := a - 6;
x := b div (1 shl a);
b := b mod (1 shl a);
Result := Result + Keys64[x + 1];
end;
end;
if a > 0 then
begin
x := b shl (6 - a);
Result := Result + Keys64[x + 1];
end;
end;
function Decode64String(S: string): string;
var
i: Integer;
a: Integer;
x: Integer;
b: Integer;
begin
Result := '';
a := 0;
b := 0;
for i := 1 to Length(s) do
begin
x := Pos(s[i], Keys64) - 1;
if x >= 0 then
begin
b := b * 64 + x;
a := a + 6;
if a >= 8 then
begin
a := a - 8;
x := b shr a;
b := b mod (1 shl a);
x := x mod 256;
Result := Result + chr(x);
end;
end
else
Exit;
end;
end;
function Encode64StringToStream(const Input: TStream; var Output: string): Boolean;
var
MS: TMemoryStream;
begin
Result := False;
MS := TMemoryStream.Create;
try
Input.Seek(0, soFromBeginning);
MS.CopyFrom(Input, Input.Size);
MS.Seek(0, soFromBeginning);
Output := Encode64String(StreamToString(MS));
finally
MS.Free;
end;
Result := True;
end;
procedure Decode64StringToStream(const Input: string; Output: TStream);
var
MS: TMemoryStream;
begin
try
MS := TMemoryStream.Create;
try
StringToStream(MS, Decode64String(Input));
MS.Seek(0, soFromBeginning);
Output.CopyFrom(MS, MS.Size);
Output.Position := 0;
finally
MS.Free;
end;
except on E: Exception do
raise Exception.Create('stream decode error - ' + E.Message);
end;
end;
procedure StringToStream(Stream: TStream; const S: string);
begin
Stream.Write(Pointer(S)^, Length(S));
end;
function StreamToString(MS: TMemoryStream): string;
begin
SetString(Result, PChar(MS.Memory), MS.Size div SizeOf(Char));
end;
I am 99% sure the problem here is going to be unicode related. It's a shame because I believe Lazarus/Freepascal has always been unicode but not Delphi and so uses different string types making it almost impossible for the less professional users like myself to solve!
To be honest I think all the code above is a bit of a mess, and it feels like I am just trying to guess what to change the strings to without really knowing what I am doing.
My first thought was to change everything from String to AnsiString. This nearly worked one time but when trying to use Decode64StringToStream I got zero data back. Other times the data was not properly saving as base64 encoded format, and sometimes I even got errors like TStream.Seek not implemented or something.
PS, I have read the guides and there is plenty around such as the whitepapers etc on how to migrate old Delphi projects to newer unicode versions and to be honest I am still at a loss with it. I thought replacing string to AnsiString would have been enough, but it seems it isn't.
Any tips, pointers or general advice or clues would be greatly appreciated thanks.

Answer 1

I think what you want to do is:
Convert the Unicode string to UTF-8 encoding. This is often the most space efficient format for Unicode text.
Encode the string using base64.
Then to decode you just reverse the steps.
The code looks like this:
function Encode(const Input: string): AnsiString;
var
utf8: UTF8String;
begin
utf8 := UTF8String(Input);
Result := EncdDecd.EncodeBase64(PAnsiChar(utf8), Length(utf8));
end;
function Decode(const Input: AnsiString): string;
var
bytes: TBytes;
utf8: UTF8String;
begin
bytes := EncdDecd.DecodeBase64(Input);
SetLength(utf8, Length(bytes));
Move(Pointer(bytes)^, Pointer(utf8)^, Length(bytes));
Result := string(utf8);
end;