I am sending and receiving messages from an electronic board through UDP using Delphi 6 and Indy 8. But since updating to Delphi XE4, the TIdUDPClient component sends wrong data packets. I think the problem is with the Send() function only sends in Unicode. Is it possible to send AnsiString through TIdUDPClient.Send()?
Here is the code I am using:
idudpclient1.Send(#$7e#$b8#$c7#$81#$10#$8d#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$9d#$02#$0d);
You are sending binary data as a String. In XE4, Strings are Unicode, and Indy's default encoding is ASCII. Your String data contains characters that are outside of the ASCII range.
Don't use String for binary data. That is not what it is meant for. You can get away with that in Delphi 2007 an earlier, but not in Delphi 2009 and later.
You can either:
continue using Send(), but tell it to use Indy's 8bit encoding instead of Indy's default encoding:
IdUDPClient1.Send(#$7e#$b8#$c7#$81#$10#$8d#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$00#$9d#$02#$0d, Indy8BitEncoding);
switch to SendBuffer() instead (which you should do, even in your Indy 8 code):
var
Buf: TIdBytes;
begin
SetLength(Buf, 34);
FillBytes(Buf, 34, $00);
Buf[0] := $7e;
Buf[1] := $b8;
Buf[2] := $c7;
Buf[3] := $81;
Buf[4] := $10;
Buf[5] := $8d;
Buf[31] := $9d;
Buf[32] := $02;
Buf[33] := $0d;
IdUDPClient1.Send(Buf);
end;
Related
I am using the latest Delphi 10.4.2 with Indy 10.
In a REST server, JSON commands are received and handled. It works fine except for Unicode.
A simple JSON like this:
{"driverNote": "Test"}
is shown correctly
If I now change to Unicode Russian characters:
{"driverNote": "Статья"}
Not sure where I should begin to track this. I expect ARequestInfo.FormParams to have the same value in debugger as s variable.
If I debug Indy itself, FormParams are set in this code:
if LRequestInfo.PostStream <> nil then
begin
// decoding percent-encoded octets and applying the CharSet is handled by
// DecodeAndSetParams() further below...
EnsureEncoding(LEncoding, enc8Bit);
LRequestInfo.FormParams :=
ReadStringFromStream( LRequestInfo.PostStream,
-1,
LEncoding
{$IFDEF STRING_IS_ANSI}, LEncoding{$ENDIF});
DoneWithPostStream(AContext, LRequestInfo); // don't need the PostStream anymore
end;
It use enc8Bit. But my string has 16-bits characters.
Is this handled incorrect in Indy?
The code snippet you quoted from IdCustomHTTPServer.pas is not what is in Indy's GitHub repo.
In the official code, TIdHTTPServer does not decode the PostStream to FormParams unless the ContentType is 'application/x-www-form-urlencoded':
if LRequestInfo.PostStream <> nil then begin
if TextIsSame(LContentType, ContentTypeFormUrlencoded) then
begin
// decoding percent-encoded octets and applying the CharSet is handled by DecodeAndSetParams() further below...
EnsureEncoding(LEncoding, enc8Bit);
LRequestInfo.FormParams := ReadStringFromStream(LRequestInfo.PostStream, -1, LEncoding{$IFDEF STRING_IS_ANSI}, LEncoding{$ENDIF});
DoneWithPostStream(AContext, LRequestInfo); // don't need the PostStream anymore
end;
end;
That ContentType check was added way back in 2010, so I don't know why it is not present in your version.
In your example, the ContentType is 'application/json', so the raw JSON should be in the PostStream and the FormParams should be blank.
That being said, in your version of Indy, TIdHTTPServer is simply reading the raw bytes from the PostStream and zero-extending each byte to a 16-bit character in the FormParams. To recover the original bytes, simply truncate each Char to an 8-bit Byte. For instance, you can use Indy's ToBytes() function in the IdGlobal unit, specifying enc8Bit/IndyTextEncoding_8Bit as the byte encoding.
JSON is most commonly transmitted as UTF-8 (and that is the case in your example), so when you have access to the raw bytes, in any version, make sure you parse the JSON bytes as UTF-8.
I'm trying to send bytes between two delphi 2010 applications using Indy 10, but without success. Received bytes are different from sent bytes. This is my example code:
Application 1, send button click:
var s:TIdBytes;
begin
setlength(s,3);
s[0]:=48;
s[1]:=227;
s[2]:=0;
IdTCPClient.IOHandler.WriteDirect(s); // or .Write(s);
end;
Application 2, idtcpserver execute event (first attempt):
procedure TForm1.IdTCPServerExecute(AContext: TIdContext);
var rec:TIdBytes;
b:byte;
begin
SetLength(rec,0);
b:=AContext.Connection.IOHandler.ReadByte;
while b<>0 do
begin
setlength(rec, length(rec)+1);
rec[length(rec)-1]:=b;
b:=AContext.Connection.IOHandler.ReadByte;
end;
// Here I expect rec[0] = 48, rec[1]=227, rec[2]=0.
// But I get: rec[0] = 48, rec[1]=63, rec[2]=0. Rec[1] is incorrect
// ... rest of code
end;
Application 2, idtcpserver execute event (second attempt):
procedure TForm1.IdTCPServerExecute(AContext: TIdContext);
var c:ansistring;
begin
c:= AContext.Connection.IOHandler.ReadLn(Char(0));
// Here I expect c[0] = 48, c[1]=227, c[2]=0.
// But I get: c[0] = 48, c[1]=63, c[2]=0. c[1] is incorrect
// ... rest of code
end;
The most strange thing is those applications were developed with Delphi 5 some years ago and they worked good (with readln(char(0)). When I translate both applications to Delphi 2010 they stop working. I supoused It was due unicode string, but I haven't found a solution.
First off, don't use TIdIOHandler.WriteDirect() directly, use only the TIdIOHandler.Write() overloads instead.
Second, there is no possible way that your 1st attempt code can produce the result you are claiming. TIdIOHandler.Write[Direct](TIdBytes) and TIdIOHandler.ReadByte() operate on raw bytes only, and bytes are transmitted as-is. So you are guaranteed to get exactly what you send, there is simply no possible way the byte values could change as you have suggested.
However, your 2nd attempt code certainly can produce the result you are claiming, because TIdIOHandler.ReadLn() will read in the bytes and convert them to a UnicodeString, which you are then assigning to an AnsiString, which means the received data is going through 2 lossy charset conversions:
first, the received bytes are decoded as-is into a UTF-16 UnicodeString using the TIdIOHandler.DefStringEncoding property as the decoding charset, which is set to IndyTextEncoding_ASCII by default (you can change that, for example to IndyTextEncoding_8bit). Byte 227 is outside of the ASCII range, so that byte gets decoded to Unicode character '?' (#63), which is an indication that data loss has occurred.
then, the UnicodeString is assigned to AnsiString, which converts the UTF-16 data to ANSI using the RTL's default charset (which is set to the user's OS locale by default, but can be changed using the System.SetMultiByteConversionCodePage() function). In this case, no more loss occurs, since the UnicodeString is holding characters in the US-ASCII range, which convert to ANSI as-is.
That being said, I would not suggest using TIdIOHandler.ReadByte() in a manual loop like you are doing. Although TIdIOHandler does not have a WaitFor() method for bytes, there is nonetheless a more efficient way to approach this, eg:
procedure TForm1.IdTCPServerExecute(AContext: TIdContext);
var
rec: TIdBytes;
LPos: Integer;
begin
SetLength(rec, 0);
// adapted from code in TIdIOHandler.WaitFor()...
with AContext.Connection.IOHandler do
begin
LPos := 0;
repeat
LPos := InputBuffer.IndexOf(Byte(0), LPos);
if LPos <> -1 then begin
InputBuffer.ExtractToBytes(rec, LPos+1);
{ or, if you don't want the terminating $00 byte put in the TIdBytes:
InputBuffer.ExtractToBytes(rec, LPos);
InputBuffer.Remove(1);
}
Break;
end;
LPos := InputBuffer.Size;
ReadFromSource(True, IdTimeoutDefault, True);
until False;
end;
// ... rest of code
end;
Or:
procedure TForm1.IdTCPServerExecute(AContext: TIdContext);
var
rec: string
begin
rec := AContext.Connection.IOHandler.ReadLn(#0, IndyTextEncoding_8Bit);
// ... rest of code
end;
The most strange thing is those applications were developed with Delphi 5 some years ago and they worked good (with readln(char(0)). When I translate both applications to Delphi 2010 they stop working. I supoused It was due unicode string, but I haven't found a solution.
Yes, the string type in Delphi 5 was AnsiString, but it was changed to UnicodeString in Delphi 2009.
But even so, in pre-Unicode versions of Delphi, TIdIOHandler.ReadLn() in Indy 10 will still read in raw bytes and convert them to Unicode using the TIdIOHanlder.DefStringEncoding. It will then convert that Unicode data to AnsiString before exiting, using the TIdIOHandler.DefAnsiEncoding property as the converting charset, which is set to IndyTextEncoding_OSDefault by default.
The morale of the story is - whenever you are reading in bytes and converting them to string characters, the bytes must be decoded using the correct charset, or else you will get data loss. This was true in Delphi 5 (but not as strictly enforced), it is true more so in Delphi 2009+.
I try to load Cyrillic web page (default codepage Western) and put it to TMemo component.
But I see "Âûñòàâêè" instead of "Выставки" in Memo.
How to transform string from Western to Windows 1251 codepage?
Delphi XE 8 sp1
TMemo (and most of the RTL/VCL/FMX in general) in XE8 expects UnicodeString data in UTF-16 format. You would have to decode the webpage data from its actual charset (which is presumably already Windows-1251, as it does not make sense for Russian text to be encoded in Windows-1252) to UTF-16 before then loading it into the TMemo. The actual charset used for the raw data needs to be reported in the HTTP Content-Type header, or in the HTML itself.
You would not decode the raw data to Windows 1251. That would have been necessary only if you were using a pre-Unicode version of Delphi (2007 and earlier) running your app on a Windows Russian machine that uses Windows-1251 as its default codepage. Those days are gone in a Unicode environment like XE8.
Delphi ships with Indy pre-installed. Indy's TIdHTTP component handles the charset-to-UTF16 decoding for you, eg:
Memo1.Text := IdHTTP1.Get(URL);
If you download the webpage data any other way, you would have to download it as raw bytes and decode them yourself, such as by using TEncoding.GetEncoding(1251) followed by TEncoding.GetString(). Or, if the bytes are in a TStream, you can use Memo1.Lines.LoadFromStream() specifying TEncoding.GetEncoding(1251) as the encoding.
type
TSrcStr = type AnsiString(1251);
TDstStr = type AnsiString(1252);
function Decode(const s: string): string;
var
a: TSrcStr;
b: TDstStr;
begin
setlength(a, length(s));
b := s;
move(b[low(b)], a[low(a)], length(b)*sizeof(b[low(b)]));
result := a;
end;
procedure Test;
var
s: string;
begin
s := 'Âûñòàâêè';
s := Decode(s);
Assert(s='Выставки');
end;
I am using Indy 10 with Delphi. Following is my code which uses EncodeString method of Indy to encode a string.
var
EncodedString : String;
StringToBeEncoded : String;
EncoderMIME: TIdEncoderMIME;
....
....
EncodedString := EncoderMIME.EncodeString(StringToBeEncoded);
I am not getting the correct value in encoded sting.
What is the purpose of IndyTextEncoding_OSDefault?
Here's the source code for IndyTextEncoding_OSDefault.
function IndyTextEncoding_OSDefault: IIdTextEncoding;
begin
if GIdOSDefaultEncoding = nil then begin
LEncoding := TIdMBCSEncoding.Create;
if InterlockedCompareExchangeIntf(IInterface(GIdOSDefaultEncoding), LEncoding, nil) <> nil then begin
LEncoding := nil;
end;
end;
Result := GIdOSDefaultEncoding;
end;
Note that I stripped out the .net conditional code for simplicity. Most of this code is to arrange singleton thread-safety. The actual value returned is synthesised by a call to TIdMBCSEncoding.Create. Let's look at that.
constructor TIdMBCSEncoding.Create;
begin
Create(CP_ACP, 0, 0);
end;
Again I've remove conditional code that does not apply to your Windows setting. Now, CP_ACP is the Active Code Page, the current system Windows ANSI code page. So, on Windows at least, IndyTextEncoding_OSDefault is an encoding for the current system Windows ANSI code page.
Why did using IndyTextEncoding_OSDefault give the same behaviour as my Delphi 7 code?
That's because the Delphi 7 / Indy 9 code for TEncoderMIME.EncodeString does not perform any code page transformation and MIME encodes the input string as though it were a byte array. Since the Delphi 7 string is encoded in the active ANSI code page, this has the same effect as passing IndyTextEncoding_OSDefault to TEncoderMIME.EncodeString in your Unicode version of the code.
What is the difference between IndyTextEncoding_Default and IndyTextEncoding_OSDefault?
Here is the source code for IndyTextEncoding_OSDefault:
function IndyTextEncoding_Default: IIdTextEncoding;
var
LType: IdTextEncodingType;
begin
LType := GIdDefaultTextEncoding;
if LType = encIndyDefault then begin
LType := encASCII;
end;
Result := IndyTextEncoding(LType);
end;
This returns an encoding that is determined by the value of GIdDefaultTextEncoding. By default, GIdDefaultTextEncoding is encASCII. And so, by default, IndyTextEncoding_Default yields an ASCII encoding.
Beyond all this you should be asking yourself which encoding you want to be using. Relying on default values leaves you at the mercy of those defaults. What if those defaults don't do what you want to do. Especially as the defaults are not Unicode encodings and so support only a limited range of characters. And what's more are dependent on system settings.
If you wish to encode international text, you would normally choose to use the UTF-8 encoding.
One other point to make is that you are calling EncodeString as though it were an instance method, but it is actually a class method. You can remove EncoderMIME and call TEncoderMIME.EncodeString. Or keep EncoderMIME and call EncoderMIME.Encode.
I am using HttpCli component form ICS to POST a request. I use an example that comes with the component. It says:
procedure TForm4.Button2Click(Sender: TObject);
var
Data : String;
begin
Data:='status=no';
HttpCli1.SendStream := TMemoryStream.Create;
HttpCli1.SendStream.Write(Data[1], Length(Data));
HttpCli1.SendStream.Seek(0, 0);
HttpCli1.RcvdStream := TMemoryStream.Create;
HttpCli1.URL := Trim('http://server/something');
HttpCli1.PostAsync;
end;
But it fact, it sends not
status=no
but
s.t.a.t.u
I can't understand, where is the problem. Maybe someone can show an example, how to send POST request with the help of HttpCli component?
PS I can't use Indy =)
I suppose you're using Delphi 2009 or later, where the string type holds two-byte-per-character Unicode data. The Length function gives the number of characters, not the number of bytes, so when you put your string into the memory stream, you only copy half the bytes from the string. Even if you'd copied all of them, though, you'd still have a bunch of extra data in the stream since each character has two bytes and the server probably only expects to get one.
Use a different string type, such as AnsiString or UTF8String.