I have an issue with TStringStream encoding on different OS region languages.
I am trying to transfer image data. I was having a problem with it before, and it was solved in another question.
But now I have an encoding issue on TStringStream itself. If the Operating System region language is set to English, I get corrupted data from TStringStream.
When I change the language to Arabic, the data comes out correctly.
I currently use Indy to encode the binary data, but before starting the encoding, the data from StringStream.DataString is already corrupted if the region encoding is not set to Arabic. I tried to add TEncoding.UTF8 to TStringStream.Create(), but the data still comes out incorrectly from StringStream.DataString.
function Encode64(const S: string; const ByteEncoding: IIdTextEncoding = nil): string;
begin
Result := TIdEncoderMIME.EncodeString(S, ByteEncoding);
end;
function Decode64(const S: string; const ByteEncoding: IIdTextEncoding = nil): string;
begin
Result := TIdDecoderMIME.DecodeString(S, ByteEncoding);
end;
StringStream := TStringStream.Create('');
try
Jpg.Performance := jpBestSpeed;
Jpg.ProgressiveEncoding := True;
Jpg.ProgressiveDisplay := True;
jpg.Assign(Image2.Picture.Bitmap);
jpg.CompressionQuality := 25;
jpg.Compress;
jpg.SaveToStream(StringStream);
StringImageData := StringStream.DataString; // the data here is corrupted when Os region is not set to arabic
strcams := '<[S:' + IntToStr(Length(StringImageData)) + 'B]>' + StringImageData;
if length(strcams) < byt then begin
Sendimgdata('IMGDATA123', Encode64(strcams, IndyTextEncoding_UTF8) + sep);
end;
...
You cannot save a JPG to a TStringStream to begin with. You need to encode the binary data without converting it to a string first, which will corrupt the data. Use a TMemoryStream instead for the binary data:
MemoryStream := TMemoryStream.Create;
...
jpg.SaveToStream(MemoryStream);
MemoryStream.Position := 0;
StringImageData := TIdEncoderMIME.EncodeStream(MemoryStream);
...
If you want to encode after you insert the text with length, you have to work with TMemoryStream too:
procedure StringToStream(aStream: TStream; const aString: AnsiString);
begin
aStream.Write(PAnsiChar(AString)^, Length(AString));
end;
...
JpegStream := TMemoryStream.Create;
jpg.SaveToStream(JpegStream);
CompleteStream := TMemoryStream.Create;
StringToStream(CompleteStream, '<[S:' + IntToStr(JpegStream.Size)+'B]>');
CompleteStream.CopyFrom(JpegStream, 0);
StringImageData := TIdEncoderMIME.EncodeStream(CompleteStream);
Related
I have a method that reads the data in the cells of a row of a TStringGrid, and copies it to the clipboard. And I have a corresponding method to paste the data from the clipboard into an empty row in the TStringGrid.
These methods were written for D7, but are broken after migration to XE2.
procedure TfrmBaseRamEditor.CopyLine(Sender: TObject; StrGridTemp: TStringGrid;
Row, Column: Integer);
var
Stream: TMemoryStream;
MemHandle: THandle;
MemBlock: Pointer;
i, Len: Integer;
RowStr: String;
begin
Stream := nil;
try
Stream := TMemoryStream.Create;
// The intermediate format to write to the stream.
// Separate each item by horizontal tab character.
RowStr := '';
for i := 0 to (StrGridTemp.ColCount - 1) do
RowStr := RowStr + StrGridTemp.Cells[i, Row] + #9;
// Write all elements in a string.
Len := Length(RowStr);
Stream.Write(Len, SizeOf(Len));
Stream.Write(PChar(RowStr)^, Length(RowStr));
// Request Memory for the clipboard.
MemHandle := GlobalAlloc(GMEM_DDESHARE, Stream.SIZE);
MemBlock := GlobalLock(MemHandle);
try
// Copy the contents of the stream into memory.
Stream.Seek(0, soFromBeginning);
Stream.Read(MemBlock^, Stream.SIZE);
finally
GlobalUnlock(MemHandle);
end;
// Pass the memory to the clipboard in the correct format.
Clipboard.Open;
Clipboard.SetAsHandle(TClipboardFormat, MemHandle);
Clipboard.Close;
finally
Stream.Free;
end;
end;
procedure TfrmBaseRamEditor.PasteLine(Sender: TObject; StrGridTemp: TStringGrid;
Row, Column: Integer);
var
Stream: TMemoryStream;
MemHandle: THandle;
MemBlock: Pointer;
ASize, Len, i: Integer;
TempStr: String;
begin
Clipboard.Open;
try
// If something is in the clipboard in the correct format.
if Clipboard.HasFormat(TClipboardFormat) then
begin
MemHandle := Clipboard.GetAsHandle(TClipboardFormat);
if MemHandle <> 0 then
begin
// Detect size (number of bytes).
ASize := GlobalSize(MemHandle);
Stream := nil;
try
Stream := TMemoryStream.Create;
// Lock the contents of the clipboard.
MemBlock := GlobalLock(MemHandle);
try
// Copy the data into the stream.
Stream.Write(MemBlock^, ASize);
finally
GlobalUnlock(MemHandle);
end;
Stream.Seek(0, soFromBeginning);
Stream.Read(Len, SizeOf(Len));
SetLength(TempStr, Len);
Stream.Read(PChar(TempStr)^, Stream.SIZE);
for i := 0 to StrGridTemp.RowCount do
StrGridTemp.Cells[i, Row] := NextStr(TempStr, #9);
finally
Stream.Free;
end;
end;
end;
finally
Clipboard.Close;
end;
end;
The problem manifests when I copy a row with some values, then paste it into an empty row. The first cell is pasted correctly, but the second cell contains garbage characters (and nothing is pasted in the 3rd column onwards). I know why nothing is pasted in 3rd column onwards: because the "horizontal tab" character which separates the columns is corrupted along with the cell contents.
I've looked through "Delphi and Unicode" by Marco Cantu, but haven't been able to figure out where it's all going wrong.
Char is an alias for WideChar. So in CopyLine
Stream.Write(PChar(RowStr)^, Length(RowStr));
only writes half the string. It should be
Stream.Write(PChar(RowStr)^, Length(RowStr)*SizeOf(Char));
In PasteLine I find this line odd:
Stream.Read(PChar(TempStr)^, Stream.SIZE);
Since you've already consumed some of the string you are attempting to read past the end. I'd write it like this:
Stream.Read(PChar(TempStr)^, Len*SizeOf(Char));
Note that if you use the same custom clipboard format identifier as your ANSI program then you'll have encoding mismatches if you copy from one and paste into the other. You might be wise to register under a different clipboard format for your new Unicode format.
Some other comments:
Stream := nil;
try
Stream := TMemoryStream.Create;
...
finally
Stream.Free;
end;
should be written as:
Stream := TMemoryStream.Create;
try
...
finally
Stream.Free;
end;
If the constructor raises an exception, the try block will not be entered.
You don't really need to write out the string length. You can rely on the stream size when reading to know how long the string is.
In CopyLine, the clipboard Open and Close calls should be protected by a try/finally block.
I have been hounded this problem for a few days, I have two cliendatasets with data in them and I want to convert the olevariant data to string using two functions I found here in Stack Overflow.
The purpose of conversion to string is to be able to transfer the string to another location and convert it back again to olevariant and assign it to another clientdataset.
To simulate it, I created a sample app with the following partial code(see block below).
The code executes properly but my problem is when I convert the windows locale to japanese(which is the requirement), I encounter a datapacket mismatch in the data assignment on the second dataset. but if I do this in the japanese locale:
clientdataset2.data := clientdataset1.data
it works fine. English locale, the code works just fine.
Is there a problem in the string conversion? or is there anything I can do? I really would appreciate help with this.
//to simulate the conversion
TempData := ClientDataSet1.Data;
TempString := OleVariantToString(ClientDataset1.Data);
TempData2 := StringToOleVariant(TempString);
ClientDataSet2.Data := TempData2; //mismatch in data packet happens here in japanese locale
//conversion functions
function TForm1.OleVariantToString(const Value: OleVariant): string;
var
ss: TStringStream;
Size: integer;
Data: PByteArray;
begin
Result := '';
if Length(Value) = 0 then
Exit;
ss := TStringStream.Create;
try
Size := VarArrayHighBound(Value, 1) - VarArrayLowBound(Value, 1) + 1;
Data := VarArrayLock(Value);
try
ss.Position := 0;
ss.WriteBuffer(Data^, Size);
ss.Position := 0;
Result := ss.DataString;
finally
VarArrayUnlock(Value);
end;
finally
ss.Free;
end;
end;
function TForm1.StringToOleVariant(const Value: string): OleVariant;
var
ss: TStringStream;
MyBuffer: Pointer;
begin
Result := null;
if Value = '' then
Exit;
ss := TStringStream.Create(Value);
try
Result := VarArrayCreate([0, ss.Size - 1], varByte);
MyBuffer := VarArrayLock(Result);
try
ss.Position := 0;
ss.ReadBuffer(MyBuffer^, ss.Size);
finally
VarArrayUnlock(Result);
end;
finally
ss.Free;
end;
end;
Streaming to string is already implemented, you can use
Writing: TClientDataSet.SaveToFile or TClientDataSet.SaveToStream
Reading: TClientDataSet.LoadFromFile or TClientDataSet.LoadFromStream
procedure SaveToStream(Stream: TStream; Format: TDataPacketFormat = dfBinary);
procedure SaveToFile(const FileName: string = ''; Format: TDataPacketFormat = fBinary);
procedure LoadFromStream(Stream: TStream);
procedure LoadFromFile(const FileName: string = '');
the TDataPacketFormat options are:
dfBinary: Information is encoded in binary format.
dfXML:Information is encoded in XML, with extended characters encoded using an escape sequence.
dfXMLUTF8:Information is encoded in XML, with extended characters represented using UTF8.
Using dfXMLUTF8 you should have no problems with non/ansi characters sets.
The following code works on Win32, anyway it is trowing exception if run on Android or iOS.
The exception is : "No mapping for the Unicode character exists in the target multi-byte code page"
function Form1.ZDecompressString(aText: String): String;
var
strInput,
strOutput : TStringStream;
Unzipper : TZDecompressionStream;
begin
Result := '';
strInput := TStringStream.Create( aText );
strOutput := TStringStream.Create;
try
Unzipper := TZDecompressionStream.Create( strInput );
try
strOutput.CopyFrom( Unzipper, Unzipper.Size );
finally
Unzipper.Free;
end;
Result := strOutput.DataString;
finally
strInput.Free;
strOutput.Free;
end;
end;
procedure Form1.getUsers;
var
XMLRqst : String;
XMLResponse : TStringStream;
XMLRequest : TStringStream;
idHTTP : TIdHTTP;
s : String;
begin
XMLRqst := UTF8ToString( '<root company="belvew"/>' );
XMLRequest := TStringStream.Create( XMLRqst, TEncoding.UTF8 );
XMLResponse := TStringStream.Create( '' );
try
try
idHTTP := TIdHTTP.Create( Self );
idHTTP.CookieManager := idCookieManager;
idHTTP.ReadTimeout := 60000;{ IdTimeoutInfinite; }
idHTTP.ConnectTimeout := 60000;
idHTTP.HandleRedirects := True;
XMLResponse.Position := 0;
XMLRequest.Position := 0;
idHTTP.Post( 'http://localhost/API/getUsers.aspx', XMLRequest, XMLResponse );
idHTTP.Disconnect;
unique_id := ZDecompressString( XMLResponse.DataString );
XMLRequest.Free;
XMLResponse.Free;
except
on E : Exception do
begin
ShowMessage( 'exception : '#13 + E.Message );
end;
end;
finally
idHTTP.Free;
end;
end;
procedure Form1.onCreate( Sender : TObject );
begin
getUsers;
end;
Why the code does not work under Android or iOS ?
I would say that you need to throw this code away and start again. Like all compression algorithms that I have ever encountered, zlib does not operate on text. It operates on byte arrays. Back in the days when a Delphi string used 8 bit encoding, people played fast and loose with such strings, treating them as if they were byte arrays. And ever since those days, the misconception that compression can operate on strings has endured.
You need to pick an encoding to use to convert text into byte arrays. A good choice would be UTF-8. And then if you want to represent the compressed data as text, you need to use an encoding like base64.
The compression process is as follows:
Encode the text as a UTF-8 byte array.
Compress with zlib.
Encode the compressed byte array with base64.
In the opposite direction you would:
Decode the base64 text to a compressed byte array.
Decompress this with zlib.
Decode the UTF-8 encoded byte array to text.
To encode/decode UTF-8 you use TEncoding.UTF8. The GetBytes and GetString methods are what you need. And for base64 encode/decode you would probably be best advised to use the Indy library since you are already using it.
According to the comments, you've got as far as decompressing to a UTF-8 encoded byte array. In which case the final step is to write:
text := TEncoding.UTF8.GetString(utf8byteArray);
In addition to what David said, your code is doing too many byte<->string conversions using unspecified encodings for some of them, so you are subject to ANSI<->Unicode conversions that can lose data.
More importantly, all you are really doing is decompressing the HTTP response's raw binary data, but using a TStringStream to do that. That in of itself is wrong, since compression operates on bytes, not on characters. But also, TIdHTTP has built-in support for decompressing an HTTP response, if the response has a Content-Encoding header of either deflate or gzip. In this case, since you are not filling in the TIdHTTP.Request.AcceptEncoding property, there should be no HTTP-level compression. But let's say there was (faulty server implementation, for instance). All you have to do to enable that functionality is assign the TIdHTTP.Compressor property, and then let TIdHTTP do the rest internally for you. That will greatly simplify your code, for example:
uses
..., IdCompressorZLib;
procedure Form1.getUsers;
var
XMLRqst : String;
XMLRequest : TStringStream;
idHTTP : TIdHTTP;
begin
XMLRqst := '<root company="belvew"/>';
try
XMLRequest := TStringStream.Create( XMLRqst, TEncoding.UTF8 );
try
idHTTP := TIdHTTP.Create;
try
idHTTP.CookieManager := idCookieManager;
idHTTP.ReadTimeout := 60000;{ IdTimeoutInfinite; }
idHTTP.ConnectTimeout := 60000;
idHTTP.HandleRedirects := True;
idHTTP.Compressor := TIdCompressorZLib.Create(idHTTP);
unique_id := idHTTP.Post( 'http://localhost/API/getUsers.aspx', XMLRequest );
finally
idHTTP.Free;
end;
finally
XMLRequest.Free;
end;
except
on E : Exception do
begin
ShowMessage( 'exception : '#13 + E.Message );
end;
end;
end;
We have a library function that goes like this:
class function TFileUtils.ReadTextStream(const AStream: TStream): string;
var
StringStream: TStringStream;
begin
StringStream := TStringStream.Create('', TEncoding.Unicode);
try
// This is WRONG since CopyFrom might rewind the stream (see Remys comment)
StringStream.CopyFrom(AStream, AStream.Size - AStream.Position);
Result := StringStream.DataString;
finally
StringStream.Free;
end;
end;
When I check the string that is returned by the function the first Char is the (little-endian) BOM.
Why doesn't TStringStream ignore the BOM?
Is there a better way to do this? I don't need backwards compatibility with older Delphi versions, a working solution for XE2 would be fine.
The BOM has to be coming from the source TStream, as TStringStream does not write a BOM. If you want to ignore the BOM if it is present in the source, you have to do it manually before then copying the data, eg:
class function TFileUtils.ReadTextStream(const AStream: TStream): string;
var
StreamPos, StreamSize: Int64;
Buf: TBytes;
NumBytes: Integer;
Encoding: TEncoding;
begin
Result := '';
StreamPos := AStream.Position;
StreamSize := AStream.Size - StreamPos;
// Anything available to read?
if StreamSize < 1 then Exit;
// Read the first few bytes from the stream...
SetLength(Buf, 4);
NumBytes := AStream.Read(Buf[0], Length(Buf));
if NumBytes < 1 then Exit;
Inc(StreamPos, NumBytes);
Dec(StreamSize, NumBytes);
// Detect the BOM. If you know for a fact what the TStream data is encoded as,
// you can assign the Encoding variable to the appropriate TEncoding object and
// GetBufferEncoding() will check for that encoding's BOM only...
SetLength(Buf, NumBytes);
Encoding := nil;
Dec(NumBytes, TEncoding.GetBufferEncoding(Buf, Encoding));
// If any non-BOM bytes were read than rewind the stream back to that position...
if NumBytes > 0 then
begin
AStream.Seek(-NumBytes, soCurrent);
Dec(StreamPos, NumBytes);
Inc(StreamSize, NumBytes);
end else
begin
// Anything left to read after the BOM?
if StreamSize < 1 then Exit;
end;
// Now read and decode whatever is left in the stream...
StringStream := TStringStream.Create('', Encoding);
try
StringStream.CopyFrom(AStream, StreamSize);
Result := StringStream.DataString;
finally
StringStream.Free;
end;
end;
Apparently TStreamReader doesn't suffer from the same problem:
var
StreamReader: TStreamReader;
begin
StreamReader := TStreamReader.Create(AStream);
try
Result := StreamReader.ReadToEnd;
finally
StreamReader.Free;
end;
end;
TStringList also works (thanks whosrdaddy):
var
Strings: TStringList;
begin
Strings := TStringList.Create;
try
Strings.LoadFromStream(AStream);
Result := Strings.Text;
finally
Strings.Free;
end;
end;
I also measured both methods and TStreamReader seems to be about twice as fast.
I am trying to remotely read a binary (REG_BINARY) registry value, but I get nothing but junk back. Any ideas what is wrong with this code? I'm using Delphi 2010:
function GetBinaryRegistryData(ARootKey: HKEY; AKey, AValue, sMachine: string; var sResult: string): boolean;
var
MyReg: TRegistry;
RegDataType: TRegDataType;
DataSize, Len: integer;
sBinData: string;
bResult: Boolean;
begin
bResult := False;
MyReg := TRegistry.Create(KEY_QUERY_VALUE);
try
MyReg.RootKey := ARootKey;
if MyReg.RegistryConnect('\\' + sMachine) then
begin
if MyReg.KeyExists(AKey) then
begin
if MyReg.OpenKeyReadOnly(AKey) then
begin
try
RegDataType := MyReg.GetDataType(AValue);
if RegDataType = rdBinary then
begin
DataSize := MyReg.GetDataSize(AValue);
if DataSize > 0 then
begin
SetLength(sBinData, DataSize);
Len := MyReg.ReadBinaryData(AValue, PChar(sBinData)^, DataSize);
if Len <> DataSize then
raise Exception.Create(SysErrorMessage(ERROR_CANTREAD))
else
begin
sResult := sBinData;
bResult := True;
end;
end;
end;
except
MyReg.CloseKey;
end;
MyReg.CloseKey;
end;
end;
end;
finally
MyReg.Free;
end;
Result := bResult;
end;
And I call it like this:
GetBinaryRegistryData(
HKEY_LOCAL_MACHINE,
'\SOFTWARE\Microsoft\Windows NT\CurrentVersion',
'DigitalProductId', '192.168.100.105',
sProductId
);
WriteLn(sProductId);
The result I receive from the WriteLn on the console is:
ñ ♥ ???????????6Z ????1 ???????☺ ???♦ ??3 ? ??? ?
??
Assuming that you are already connected remotely, try using the GetDataAsString function
to read binary data from the registry.
sResult := MyReg.GetDataAsString(AValue);
You're using Delphi 2010, so all your characters are two bytes wide. When you set the length of your result string, you're allocating twice the amount of space you need. Then you call ReadBinaryData, and it fills half your buffer. There are two bytes of data in each character. Look at each byte separately, and you'll probably find that your data looks less garbage-like.
Don't use strings for storing arbitrary data. Use strings for storing text. To store arbitrary blobs of data, use TBytes, which is an array of bytes.