I'm using FireMonkey in Delphi 10.1 Berlin for developing an Android mobile client application, and I'm using VCL in Delphi 10.1 Berlin for developing a Windows server application.
In the mobile application, I am using TIdTCPClient for sending the following record:
PSampleReq = ^TSampleReq ;
TSampleReq = packed record
Value1: array [0..10] of Char;
Value2: array [0..59] of Char;
Value3: array [0..40] of Char;
Value4: Int64;
Value5: array [0..9] of Char;
Value6: array [0..9] of Char;
Value7: Integer;
end;
I have filled the packet with data and am sending the packet using the following code:
FIdTCPClient.IOHandler.Write(RawToBytes(TSampleReq,SizeOf(TSampleReq)));
While reading the data in the Server application, I am not able to read the Value5, Value6and Value7 fields. Below is the code that is reading the data:
Move(tyTIDBytes[0], SampleReq, SizeOf(TSampleReq));
For receiving the data which is send from the client socket, I have used the TIDTcpServer and handled the below code in Execute method:
TServerRecord = packed record
PointerMessage : TIndyBytes;
ClientSocket : TIdTCPConnection;
end;
Var
ReceivedIDBytes: TServerRecord;
begin
if not AContext.Connection.IOHandler.InputBufferIsEmpty then
begin
AContext.Connection.IOHandler.InputBuffer.ExtractToBytes(ReceivedIDBytes.PointerMessage.tyTIDBytes) ;
ReceivedIDBytes.ClientSocket := AContext.Connection;
MessageProcessorThread.ProcessMessageQueue.Enqueue(ReceivedIDBytes);
end;
After this I'm processing the data from Queue and the processing method I have mentioned below:
var
InputRec: TServerRecord;
begin
InputRec := DBWorkerThread.DBWorkerQueue.Dequeue;
MessageHeaderPtr := #InputRec.PointerMessage.tyTIDBytes[0];
iHMMessageCode := StrToIntDef( Trim(MessageHeaderPtr^.MessageCode), UNKNOWN_MESSAGE_CODE);
case iHMMessageCode of
1001:
begin
Move(InputRec.PointerMessage.tyTIDBytes[0], SampleReq, SizeOf(TSampleReq));
end;
end;
And in this I'm not able to read the Value5, Value6 and Value7 fields.
With the below Link, I have found some optimized technique and how I can handle the packets properly without any packet missing. Please help me out to resolve this issue.
Sending the right record size over socket
Your use of ExtractToBytes() is completely wrong. That method returns whatever arbitrary bytes are stored in the InputBuffer at that particular moment, which may be less than, or more than, what you are actually expecting.
If your client is sending a fixed-sized record each time, you should be reading exactly that many bytes, no more, no less:
var
ReceivedIDBytes: TServerRecord;
begin
AContext.Connection.IOHandler.ReadBytes(ReceivedIDBytes.PointerMessage.tyTIDBytes, SizeOf(TSampleReq)); // <-- HERE!!!
ReceivedIDBytes.ClientSocket := AContext.Connection;
MessageProcessorThread.ProcessMessageQueue.Enqueue(ReceivedIDBytes);
end;
However, if the size of the record depends on the message code, then your client should send the number of bytes in a record before sending the actual record bytes:
var
tyTIDBytes: TIdBytes;
begin
tyTIDBytes := RawToBytes(TSampleReq, SizeOf(TSampleReq));
FIdTCPClient.IOHandler.Write(Int32(Length(tyTIDBytes)));
FIdTCPClient.IOHandler.Write(tyTIDBytes);
end;
And then the server can read the byte count before reading the bytes:
var
ReceivedIDBytes: TServerRecord;
begin
AContext.Connection.IOHandler.ReadBytes(ReceivedIDBytes.PointerMessage.tyTIDBytes, AContext.Connection.IOHandler.ReadInt32); // <-- HERE!!!
ReceivedIDBytes.ClientSocket := AContext.Connection;
MessageProcessorThread.ProcessMessageQueue.Enqueue(ReceivedIDBytes);
end;
Related
This code work's fine when I send data across the LAN with an Indy client component, but when I receive data from an external application from the web, it's causing it to fail. Could there be something on the client-side that is causing IdTCPServer to disconnect before all the data is read? An average of 33,000 characters are being sent by the client. Any suggestions?
procedure TFrmMain.IdTCPServer1Execute(AContext: TIdContext);
var
strm: TMemoryStream;
RxBuf: TIdBytes;
begin
Memo1.Clear;
strm := TMemoryStream.Create;
try
// read until disconnected
AContext.Connection.IOHandler.ReadStream(strm, -1, true);
strm.Position := 0;
ReadTIdBytesFromStream(strm, RxBuf, strm.Size);
finally
strm.Free;
end;
Memo1.Lines.Add(BytesToString(RxBuf));
AContext.Connection.IOHandler.WriteLn('000');
end;
I also tryed this other code, in this case unlike the first code it only reads part of the data beeing sent. Is there a way to make the IdTCPServer Handler wait until all the data is collected?
procedure TFrmMain.IdTCPServer1Execute(AContext: TIdContext);
var
RxBuf: TIdBytes;
begin
RxBuf := nil;
with AContext.Connection.IOHandler do
begin
CheckForDataOnSource(10);
if not InputBufferIsEmpty then
begin
InputBuffer.ExtractToBytes(RxBuf);
end;
end;
AContext.Connection.IOHandler.WriteLn('000');
Memo1.Lines.Add( BytesToString(RxBuf) );
end;
This code you posted as an answer is all wrong.
First off, you can't use BytesToString() on arbitrary byte blocks, that won't handle multi-byte encodings like UTF-8 correctly.
Also, you are not looking for the EOT terminator correctly. There is no guarantee that it will be the last byte of RxBuf after each read, if the client sends multiple XML messages. And even if it were, using Copy(BytesToString(), ...) to extract it into a string will never result in a blank string, like your code is expecting.
If the client sends an EOT terminator at the end of the XML, there is no need for a manual reading loop. Simply call TIdIOHandler.ReadLn() with the EOT terminator, and let it handle the read looping internally until the EOT is reached.
Also, the CoInitialize() and CoUninitialize() calls should be done in the OnConnect and OnDisconnect events, respectively (actually, they would be better called in a TIdThreadWithTask descendant assigned to the TIdSchedulerOfThread.ThreadClass property, but that is a more advanced topic for another time).
Try something more like this:
procedure TFrmMain.IdTCPServer1Connect(AContext: TIdContext);
begin
CoInitialize(nil);
AContext.Connection.IOHandler.DefStringEncoding := IndyTextEncoding_UTF8;
end;
procedure TFrmMain.IdTCPServer1Disconnect(AContext: TIdContext);
begin
CoUninitialize();
end;
procedure TFrmMain.IdTCPServer1Execute(AContext: TIdContext);
var
XML: string;
begin
cdsSurescripts.Close;
XML := AContext.Connection.IOHandler.ReadLn(#4);
Display('CLIENT', XML);
AContext.Connection.IOHandler.WriteLn('000');
end;
Personally, I would take a different approach. I would suggest using an XML parser that supports a push model. Then you can read arbitrary blocks of bytes from the connection and push them into the parser, letting it fire events to you for completed XML elements, until the terminator is reached. This way, you don't have to waste time and memory buffering the entire XML in memory before you can then process it.
For further reference to anyone, I had to create a loop and wait for an EOT chr(4) send by the client in order to collect all the data on the IdTCPServer1Execute. This happens because the data is fragmented by Indy, The code looks something like this:
procedure TFrmMain.IdTCPServer1Execute(AContext: TIdContext);
var
Len: Integer;
Loop: Boolean;
begin
CoInitialize(nil);
cdsSurescripts.Close;
Loop := True;
while Loop = true do
begin
if AContext.Connection.IOHandler.Readable then
begin
AContext.Connection.IOHandler.ReadBytes( RxBuf,-1, True);
Len := Length(BytesToString(RxBuf));
if Copy(BytesToString(RxBuf), Len, 1) = '' then
begin
loop := False;
end;
end;
end;
Display('CLIENT', BytesToString(RxBuf));
AContext.Connection.IOHandler.WriteLn('000');
CoUninitialize();
end;
using Delphi XE2 and TJvHidDevice class from Jedi library, I managed to successfully communicate with a USB device (pic32mx7 board, with my code running on it). The usual way of "send request, wait for single response" works.
The problem is with a command that results in a larger number of consecutive responses. If those responses are sent by the device as fast as possible - or even if I add a small delay between them like 5ms - I lose packets (reports? frames?). The OnDeviceData event simply doesn't seem to fire for all of them. If I add larger delays in the device's code, the problem goes away.
I used USBPcap program to capture USB data and dump it to a file which, once I open it in WireShark, contains all of the data sent by the device (I send 255 packets as a test, with all zeroes and one "1" shifting its place by 1 position in every packet). So, I think both the device and Windows are doing their job.
To make sure my Delphi code is not faulty, I tried the Jedi example project "DevReader" (here is the main.pas code) which dumps data on screen and it is missing packets as well.
I feel like there should be more information on the net about Jedi's USB classes but I am having trouble finding it.
I may be able to avoid this problem by aggregating/condensing the device's responses, but would still like to know what's going on.
Edit:
Tried from a console app: packets were not lost anymore.
Modified the Jedi demo app to only count received packets and update a counter label on screen (no forced window repaint) - no lost packets.
Added sleep(1) in the OnData event - no lost packets.
Added sleep(2) in the OnData event - losing packets again.
This looks like the Jedi thread that reads data must not be delayed by any processing - shouldn't there be some buffering of data going on (by Windows?) that would allow for this type of processing delays? Judging by the packet loss "pattern" it seems as if there is buffering, but it is insufficient because I can receive e.g. 30 packets then lose 5 then receive another 20 etc.
I will modify my code to copy the data and exit the OnData event as quickly as possible so that the thread has minimum "downtime" and I will report the outcome.
Since the cause of the problem appears to be related to the amount of time the USB reading thread is blocked by Synchronise, i.e. the data processing carried out by the main thread, I made changes in the thread code, (TJvHidDeviceReadThread class, JvHidControllerClass.pas unit). Any code that used this unit and the classes contained should still work without any modifications, nothing public was changed.
New behavior: every time the data is read, it is placed in a thread safe list. Instead of Synchronise it now uses Queue, but only if it is not queued already. The Queued method reads from the thread safe list until it is empty. It fires an event (same event as in the old code) for each buffered report in the list. Once the list is empty, the "Queued" flag is reset and the next read will cause Queuing again.
In the tests so far I did not encounter lost packets.
The thread class was extended:
TJvHidDeviceReadThread = class(TJvCustomThread)
private
FErr: DWORD;
// start of additions
ReceivedReports : TThreadList;
Queued: boolean;
procedure PushReceivedReport(const bytes: array of byte; const NumBytesRead: cardinal);
function PopReceivedReport(var ReportID: byte; var ReportBytes: TBytes): boolean;
procedure FlushBuffer;
// end of additions
procedure DoData;
procedure DoDataError;
constructor CtlCreate(const Dev: TJvHidDevice);
protected
procedure Execute; override;
public
Device: TJvHidDevice;
NumBytesRead: Cardinal;
Report: array of Byte;
constructor Create(CreateSuspended: Boolean);
//added destructor:
destructor Destroy; override;
end;
In the implementation section, the following was modified:
constructor TJvHidDeviceReadThread.CtlCreate(const Dev: TJvHidDevice);
begin
inherited Create(False);
// start of changes
ReceivedReports := TThreadList.Create;
// end of changes
Device := Dev;
NumBytesRead := 0;
SetLength(Report, Dev.Caps.InputReportByteLength);
end;
procedure TJvHidDeviceReadThread.Execute;
...
...
...
//replaced: Synchronize(DoData); with:
PushReceivedReport (Report, NumBytesRead);
...
And the following was added:
type
TReport = class
ID: byte;
Bytes: TBytes;
end;
destructor TJvHidDeviceReadThread.Destroy;
var
l: TList;
begin
RemoveQueuedEvents (self);
try
l := ReceivedReports.LockList;
while l.Count>0 do
begin
TReport(l[0]).Free;
l.Delete(0);
end;
finally
ReceivedReports.UnlockList;
FreeAndNil (ReceivedReports);
end;
inherited;
end;
procedure TJvHidDeviceReadThread.FlushBuffer;
var
ReportID: byte;
ReportBytes: TBytes;
begin
while PopReceivedReport (ReportID, ReportBytes) do
Device.OnData(Device, ReportID, ReportBytes, length(ReportBytes));
end;
function TJvHidDeviceReadThread.PopReceivedReport(var ReportID: byte; var ReportBytes: TBytes): boolean;
var
l: TList;
rep: TReport;
begin
l := ReceivedReports.LockList;
rep := nil;
try
result := l.Count>0;
if result
then
begin
rep := l[0];
l.Delete(0);
end
else Queued := false;
finally
ReceivedReports.UnlockList;
end;
if result then
begin
ReportID := rep.ID;
SetLength(ReportBytes, length(rep.Bytes));
System.move (rep.Bytes[0], ReportBytes[0], length(rep.Bytes));
rep.Free;
end;
end;
procedure TJvHidDeviceReadThread.PushReceivedReport(const bytes: array of byte; const NumBytesRead: cardinal);
var
rep: TReport;
begin
rep := TReport.Create;
setlength (rep.Bytes, NumBytesRead-1);
rep.ID := Bytes[0];
System.move (Bytes[1], rep.Bytes[0], NumBytesRead-1);
// explicitely lock the list just to provide a locking mechanism for the Queue flag as well
ReceivedReports.LockList;
try
if not Queued then
begin
Queued := true;
Queue (FlushBuffer);
end;
ReceivedReports.Add(rep);
finally
ReceivedReports.UnlockList;
end;
end;
I'm using Delphi XE4, with UDPSocketClient I send a Request to the Server.
The Request is successful, but the responce from the Server is always about 200 bytes.
It should be about 1000 bytes.
I have no clue why?! There is no EOL break or something like that.
Is it possible to read in Chunks?
procedure TForm1.SendCommand(const Pass, ACommand: string);
var
Cmd: string;
begin
if UDPSocketClient.Connected then
begin
Cmd := Pass + ' ' + ACommand;
UDPSocketClient.Sendln(AnsiString(Cmd));
Memo1.Lines.Add('');
Memo1.Lines.Add('######################');
Memo1.Lines.Add(ACommand);
Memo1.Lines.Add('######################');
Memo1.Lines.Add('');
end;
end;
procedure TForm1.BtnSendCmdClick(Sender: TObject);
var
Buff: AnsiString;
received: string;
begin
if Assigned(CurrentServer) and (CmdEdit.Text <> '') and
(CmdEdit.Text <> CmdEditPlaceHolder) then
begin
SendCommand(CurrentServer.Password, CmdEdit.Text);
end;
try
received := String(UDPSocketClient.Receiveln(Buff));
Memo1.Lines.Add(received);
finally
CmdEdit.SetFocus;
end;
I assume you're using TUDPSocket.
The parameter for ReceiveLn is the delimiter string to search for. You're passing it Buff which is an empty string, so the behaviour is likely to be undefined.
So, now I have an approach to a solution.
When i send the Command to the Server, there are two possibilitys to receive the Data:
The Incoming UDP Package is less than 200bytes, everything will be correct.
If the Incoming UDP Package reached the 1.3kb, then there are two udp packages in the socket buffer. But i can't get the Packages to my Application.
I have a record:
Tconnecting = record
var
a: int64;
b: integer;
c: integer;
end;
which I need send to server using UDP protocol
I fill it
packet.a := StrToInt64('0x1234567890');
packet.b := 0;
packet.c := RandomRange(1, 9999999);
and sending it
SetLength(send_data, sizeof(packet));
send_data := RawToBytes(packet, SizeOf(packet));
udp.SendBuffer(make_it_big_endian(send_data)); <-- the question... "network byte order"
or maybe I'm doing something wrong? I need to send "bign endian" packet
pack("N*", int64, int, int); (this is in PHP)
Thanks.
What you are actually doing is converting from host byte order to network byte order. All standard sockets libraries provide helper functions to do that.
For example, Winsock offers htons, htonl etc. And in the opposite direction you have ntohs, ntohl etc. If you are using Indy, then the equivalent functions are GStack.HostToNetwork and GStack.NetworkToHost.
You should serialize each field into a byte stream, with each field being transformed by the appropriate hton* function. Then you can put the byte stream on the wire.
Assuming your code is running in a little-endian OS/Machine (like Windows/Intel), you can just reverse the individual bytes of your packet, for example, include the FMX.Types unit in your uses clause:
type
Tconnecting = record
a: int64;
b: integer;
c: integer;
end;
procedure TForm1.Button1Click(Sender: TObject);
var
Packet, BEPacket: Tconnecting;
begin
packet.a := $0102030405060708;
packet.b := $01020304;
packet.c := $05060708;
//big endian packet
BEPacket := Packet;
ReverseBytes(#BEPacket.a, SizeOf(BePacket.a));
ReverseBytes(#BEPacket.b, SizeOf(BePacket.b));
ReverseBytes(#BEPacket.c, SizeOf(BePacket.c));
Memo1.Lines.Add(Format('a: %16.16x => %16.16x', [Packet.a, BEPacket.a]));
Memo1.Lines.Add(Format('b: %8.8x => %8.8x', [Packet.b, BEPacket.b]));
Memo1.Lines.Add(Format('c: %8.8x => %8.8x', [Packet.c, BEPacket.c]));
end;
I am working on DataSnap project in Delphi XE2 using TCP/IP protocol that needs to pass a stream of binary data to the server as a method parameter. The problem I am running into is that there seems to be a size limit of about 32 KB on the stream contents. Beyond this limit the stream received at the server is empty. If I pass additional method parameters they arrive intact so it seems to be an issue at the parameter level.
Here is how the DataSnap service class is declared:
TDataSnapTestClient = class(TDSAdminClient)
private
FSendDataCommand: TDBXCommand;
public
constructor Create(ADBXConnection: TDBXConnection); overload;
constructor Create(ADBXConnection: TDBXConnection; AInstanceOwner: Boolean); overload;
destructor Destroy; override;
procedure SendData(Data: TStream);
end;
The approach I am using should work, at least according to the article by Jim Tierney. That said, there apparently have been changes since Delphi 2009 that have broken Jim Tierney's sample code.
DataSnap Server Method Stream Parameters
Any ideas on how to resolve this issue would be greatly appreciated.
DataSnap transfers the data in 32k chunks. The receiving end has no way of knowing how many bytes will be received until after all chunks have been reassembled. Once all the data has been received, DataSnap doesn't set the size of the TStream that received the data, so you can't use it until you move it to another stream that knows how many bytes are in the stream.
I know that pulling 32k+ from a DataSnap server is not the same as pushing 32k+ to a DataSnap server, but this may work for you as well. Try running the TStream through this code after the DataSnap server finishes receiving the data:
procedure CopyStreamToMemoryStream(const ASource: TStream; var ADest: TMemoryStream; const Rewind: Boolean = True);
const
LBufSize = $F000;
var
LBuffer: PByte;
LReadCount: Integer;
begin
GetMem(LBuffer, LBufSize);
ADest.Clear;
try
repeat
LReadCount := ASource.Read(LBuffer^, LBufSize);
if LReadCount > 0 then
ADest.WriteBuffer(LBuffer^, LReadCount);
until LReadCount < LBufSize;
finally
FreeMem(LBuffer, LBufSize);
end;
if Rewind then
ADest.Seek(0, TSeekOrigin.soBeginning);
end;
I can't remember where I found this code (years ago), so I can't give credit where credit is due, but it has been working for me reliably for years now.
I got thinking about it and it occurred to me that transferring the data to another memory stream just wastes memory, especially if the file is very large. All we need to do is count the bytes and set the stream size, right?!
procedure FixStream(const AStream: TStream);
const
LBufSize = $F000;
var
LBuffer: PByte;
LReadCount, StreamSize: Integer;
begin
GetMem(LBuffer, LBufSize);
try
StreamSize := 0;
repeat
LReadCount := AStream.Read(LBuffer^, LBufSize);
Inc(StreamSize, LReadCount);
until LReadCount < LBufSize;
AStream.Size := StreamSize;
finally
FreeMem(LBuffer, LBufSize);
end;
end;
Do you want to give that a try? I'm not able to test the code right now or I would...