Writing to a TMemoryStream and losing unicode - delphi

Question

I'm getting a weird behavior when copying a TMemoryStream (containing unicode string) to another TMemoryStream, using Delphi XE2:
I have two instances of a TMemoryStream. The first instance contains unicode text (SourceMS). I write some arbitrary data to the second MemoryStream (DestMS) and then copy the contents of the first stream to the second stream, like that:
var
SomeInt: Integer;
SomeByte: Byte;
SourceMS, DestMS: TMemoryStream;
begin
...
DestMS.Write(SomeInt, SizeOf(SomeInt));
DestMS.Write(SomeByte, SizeOf(SomeByte));
SourceMS.SaveToFile('c:\SourceMS.txt'); // SourceMS.txt contains the unicode chars
DestMS.CopyFrom(SourceMS, 0); // copy the whole content of SourceMS to DestMS
DestMS.SaveToFile('c:\DestMS.txt'); // DestMS.txt DOEST NOT contain unicode chars
end;
How can I copy the contents of the first stream to the second stream without losing unicode (having an implicit conversion)?
When I say "losing unicode", I mean: The unicode string is indeed copied to the second stream, but the unicode is lost. I get ANSI chars only.

Answer 1

It seems that DestMS is just some arbitrary bytes and that SourceMS is where your Unicode content resides. If you append source to dest, then the BOM from source will not be at the beginning of the memory stream. When you open the saved text file in Windows, it won't see the BOM because it isn't at the beginning of the file, so it won't know that other characters later in the file should be treated as Unicode.
It appears that you are trying to insert some content at the front of the Unicode content.
If this is true, then you could place the Unicode content in a Unicode compliant control, add the characters to the beginning and then capture the content from the control. This would keep the BOM at the beginning of the byte stream.

Answer 2

Here's what could happen, if we were to judge this based exclusively on the 5 lines of code that were posted. TMemoryStream does not alter the the bytes in any way, we have to assume the raw bytes were successfully copied from one .txt file to the other. Both files should contain the exact same bytes, yet when viewing the files with a Text Viewer application, those same bytes are not interpreted the same way.
I can only imagine one such case:
One of the files has a BOM, most likely UTF8.
The other file has no BOM, so it's interpreted as ANSI.
It doesn't even matter which file has a BOM: going through such a process changes the way bytes are interpreted. According to Wikipedia, the vast majority of code pages are a super-set of ASCII, meaning that all bytes that can be written using 7bit are interpreted the exact same way with both UTF8 and ANSI. The "Unicode" characters that the OP complains about are certainly in the "extend" ANSI (8bit) or, when using UTF8, they're composed using 2 or more bytes. This gives the failure modes:
If the original is a ANSI file that contains extended characters (non-ASCII), if those were interpreted as UTF8, the result would probably look a bit like garbage: Two (or more) characters of the original file would seem to be replaced by some weird character.
If the original was UTF8, then all international characters would be represented using a minimum of two bytes: When interpreted as ANSI those two bytes would be represented as two distinct characters, according to the code page of the PC.

Answer 3

CopyFrom does indeed copy the whole source stream into the target stream, but it starts at the current position of the target. The arbitrary data written before still exists!
You should set MS1.Position := 0 before you call CopyFrom.