Can't seem to find anywhere informations about this, but, is TIdUDPServer.OnRead event passing everything that comes in to the AData array or not?
According to WireShark readings, I'm missing 42 bytes of data; While I should be getting 572 bytes of data on each reading, the AData size is always 530, and seems like always the same bytes are missing.
The device that sends data is broadcasting it, and I can get everything I need except for 2 bytes, which seems to be 2 of those that are missing.
Any hints on this one?
Edit:
I should mention that these are the very first 42 bytes; Everything afterwards is received fine;
The OnUDPRead event passes everything the socket receives from the OS. UDP operates on messages. Unlike TCP, a UDP read is an all-or-nothing operation, either a whole UDP message is read or an error occurs, there is no in-between.
If you are missing data, then either the OS is not providing it (such as if it belongs to the UDP and/or IP headers), or you are not reading data from the AData parameter correctly. If you think this is not the case, then you need to update your question to show your actual OnUDPRead handler code, an example WireShark dump showing the data being captured from the network, and the data that is making it to your OnUDPRead handler.
Update: The OS does not provide access to the packet headers (unless you are using a RAW socket, which TIdUDPServer does not use, but that is a whole other topic of discussion). The AData parameter of the OnUDPRead event provides only the application data portion of a packet, as that is what the OS provides. You cannot access the packet headers.
That being said, you can get the packet's source IP:Port, at least, via the ABinding.PeerIP and ABinding.PeerPort properties of the OnUDPRead event. However, there is no way to retrieve the other packet header values (nor should you ever need them in most situations), unless you sniff the network yourself, such as with a pcap library.
Related
Sorry for my question (and my English too). I am a newbie in CAN-bus. I have theoretical question. If I pass any data to one CAN-socket, will I be able to read the same data from the same socket?. After all, the transmitted data appears not only on other nodes of the CAN-bus, but also on the one from which they were transmitted?
Thank you all, I understand. I wrote a small program in C and it turned out that after successfully (it's important!) sending the data to the socket corresponding to the CAN interface, you can immediately read back the same data transmitted to the CAN bus. Perhaps this is how the CAN driver is implemented on Linux (can4linux).
Moreover, in order to be able to read the sent data back, it is important that the CAN interface is not alone in the network, since the transmitter node sets the ACK bit value in the transmitted frame to a recessive state and listens to its own transmission, waiting for all receiving nodes to set the ACK confirmation bit in the dominant state in this transmitted frame, which means that there is no error. When listening to its transmission reveals that the ACK bit is in the dominant state (meaning the transmission was successful), the transmitted frame is placed by the CAN interface driver in the input buffer and can be read back from the our socket.
If the transmission fails (the ACK bit remains in a recessive state), the CAN controller will attempt to transmit the frame in a loop. In the program, this will look like the write function has ended, but when trying to immediately read the data, the program blocks inside the read function if the blocking read mode is set.
I am writing a bulletin board system (BBS) reader on ios. I use GCDAsyncSocket library to handle packets sending and receiving. The issue that I have is the server always splits the data to send into multiple packets. I can see that happens by printing out the receiving string in didReceiveData() function.
From the GCDAsyncSocket readme, I understand TCP is a stream. I also know there are some end of stream mechanisms, such as double CR LFs at the end. I have used WireShark to parse the packets, but there is no sign of some sort of pattern in the last data packet. The site is not owned by me, so I couldn't make it to send certain bytes. There must be some way to detect the last packet, otherwise how BBS clients handle displaying data?
Double CR LFs are not end of stream. That is just part of the details of HTTP protocol, for example, but has nothing to do with closing the stream. HTTP 1.1 allows me to send multiple responses on a single stream, with double CR LF after HTTP header, without end of stream.
The TCP socket stream will return 0 on a read when it is closed from the other end, blocking or non-blocking.
So assuming the server will close the socket when it is done sending to you, you can loop and perform a blocking read and if returns > 0, process the data, then read again. if < 0, process the error code (could be fatal or not), and if == 0, socket is closed from the other side, don't read again.
For non-blocking sockets, you can use select() or some other API to detect when the stream becomes ready to read. I'm not familiar with the specific lib you are using but if it is a POSIX / Berkeley sockets API, it will work that way.
In either case, you should build a buffer of input, concatenating the results of each read until you are ready to process. As you've found, you can't assume that a single read will return a complete application level packet. But as to your question, unless the server wants you to close the socket, you should wait for read to return 0.
Hi sorry if this is a stupid question (I just started learning network programming), but I've been looking all over google about how files/data are divided into packets. I've read everywhere that somehow files are broken up into packets have headers/footers applied as they go through the OSI model and are sent through the wire where the recipient basically does the reverse and removes the headers.
My question is how exactly are files/data broken up into packets and how are they reassembled at the other end?
How does whatever doing the reassembling know when the last packet of the data has arrived and etc?
Is it possible to reassemble packets captured from another machine? And if so how?
(Also if it means anything I'm mostly interested in how this work for TCP type packets)
I also have some packets captured from an application on my computer through WireShark, they're labeled as TCP protocol, what I want to do is reassemble them back into the original data, but how can you tell which packets belong to which set of data?
Any pointers towards resources is much appreciated, thank you!
My question is how exactly are files/data broken up into packets
What's being sent over a network isn't necessarily a file. In the cases where it is a file, there are several different protocols that can send files, and the answer to the question depends on the protocol.
For FTP and HTTP, the entire contents of the file is probably being sent as a single data stream over TCP (preceded by headers in the case of HTTP, and just raw, over the connection, in the case of FTP).
For TCP, there's a "maximum segment size" negotiated by the client and server, based on factors such as the maximum packet size on the various networks between the server and client, and the file data is sent, sequentially, in chunks whose size is limited by the maximum packet size and the size of IP and TCP headers.
For remote file access protocols such as SMB, NFS, and AFP, what goes over the wire are "file read" and "file write" requests; the reply to a "file read" request includes some reply headers and, if the read is successful, the chunk of file data that the read request asked for, and a "file write" request includes some request headers and the chunk of file data being written. Those are not guaranteed to be an entire file, in order, but if the program reading or writing the file is reading or writing the entire file in sequential order, the entire file's data will be available. The packet sizes will depend on the size of the read reply/write request headers and on the read or write size being used; those packets might be broken into multiple TCP segments, based on the TCP "maximum segment size" and the size of the IP and TCP headers.
My question is how exactly are files/data broken up into packets
For FTP, the recipient of the data knows that there is no more data when the side of the TCP connection over which the data is being transmitted is closed.
For HTTP, the recipient of the data knows that there is no more data when the side of the TCP connection over which the data is being transmitted is closed or, if the connection is "persistent" (i.e., it remains open for more requests and replies), when the amount of data specified by the "Content-Size:" header, sent before the data, has been transmitted (or other similar mechanisms, such as the "last chunk" indication for chunked encoding).
For file access protocols, there's no real "we're at the end of data" indication; the closest approximation, for SMB, AFP, and NFSv4, is a "file close" operation.
Is it possible to reassemble packets captured from another machine? And if so how?
It depends on the protocol, but, for HTTP and SMB, if the capture has been read into Wireshark (and all the file data is in the capture!), you can use the "Export Objects" menu, and, for some protocols, you might also be able to use tcpflow.
My question is how exactly are files/data broken up into packets and how are they reassembled at the other end?
They are basically just chopped up. Each internet packet (with header info add) can only hold a few hundred bytes of actual data.
How does whatever doing the reassembling know when the last packet of the data has arrived and etc?
For a transfer the packets are numbered, so the receiving process knows how to put them together. If it loses a packet, it can request a resend.
Is it possible to reassemble packets captured from another machine? And if so how?
I don't understand the question. How would you get these packets unless you were a man-in-the-middle?
These answers are true for TCP packets.
First determine what size you want to transmit.
then put header, data and footer for each transmission.
See buffer length and data array should be divisible by number of packets without giving fractions.
Here header should contain frame number, time stamp, packet number
payload data
footer ---your company information.
prepare data fragments before sending
I got a problem when using NSInputStream.
I have client app which connect to a server then server will start to send message to my client app through TCP repeatedly about 1 message per second. Server is just broadcasting message to client and message is xml format. The server send a message as one packet.
Now the problem is that when I read byte from NSInputStream the data got truncated which mean instead of receive 1 complete message, I got 2 separate data(partial xml) respond from time to time. I am not able to debug because it already happen when I read data byte from NSInputStream.
I use Wireshark to analyse every packet I receive and when it happen data got truncated too, because TCP so partial data retransmit to my client.
I have tried to log every partial data byte, the sum of partial data always around 1600 byte.
I have no idea how did they design and implement server side, but I do know there are many of people connect to that server and continuous get broadcasting message from it.
Does anyone encounter this problem? Can anyone help? Is it possible that data is over the max size and get splited?
This is not a problem per se. It is part of the design of TCP and also of NSInputStream. You may receive partial messages. It's your job to deal with that fact, wait until you receive a full message, and then process the completed message.
1600 bytes is a little strange. I would expect 1500 bytes, since that's the largest legal Ethernet packet (or especially somewhere around 1472, which is a really common MTU, minus some for the headers). Or I might expect a multiple of 1k or 4k due to buffering in NSInputStream. But none of that matters. You have to deal with the fact that you will not get messages necessarily at their boundaries.
I am using the following few lines of code to write and read from an external Modem/Router (aka device) via IP.
TCPClient.IOHandler.Write(MsgStr);
TCPClient.IOHandler.InputBuffer.Clear;
TCPClient.IOHandler.ReadBytes(Buffer, 10, True);
MsgStr is a string type which contains the text that I am sending to my device.
Buffer is declared as TIdBytes.
I can confirm that IOHandler.InputBufferIsEmpty returns True immediately prior to calling ReadBytes.
I'm expecting the first 10 bytes received to be very specific hence from my point of view I am only interested in the first 10 bytes received after I've sent my string.
The trouble I am having is, when talking to certain devices, the first byte returned the first time I've sent a string after establishing a connection puts a rogue (random) byte in my Buffer output. The subsequent bytes following are correct.
eg 10 bytes I'm expecting might be: #6A1EF1090#3 but what I get is .#6A1EF1090. in this example I have a full stop where there shouldn't be one.
If I try to send again, it works fine. (ie the 2nd Write sent after a connection has been established). What's weird (to me) is using a Socket Sniffer doesn't show the random byte being returned. If I create my own "server" to receive the response and send something back it works fine 100% of the time. Other software - ie, not my software - communicates fine with the device (but of course I have no idea how they're parsing the data).
Is there anything that I'm doing incorrectly above that would cause this - bearing in mind it only occurs the first time I'm using Write after establishing a connection?
Thanks
EDIT
I'm using Delphi 7 and Indy 10.5.8
UPDATE
Ok. After much testing and looking, I am no closer to finding this solution. I am getting two main scenarios. 1 - First byte missing and 2 - "introduced" byte at the start of received packet. Using TIdLogEvent and TIdLogDebug both either show the missing byte or the initial introduced byte as appropriate. So my ReadBytes statement above is showing consistently what Indy believes is there (in my opinion).
Also, to test it further, I downloaded and installed ICS components. Unfortunately (or fortunately depending on how you look at it) this didn't show the same issues as Indy. This didn't show the first byte missing nor did it show an introduced byte at the beginning. However, I have only done superficial testing, but Indy produces the behaviour "pretty much straight away" whereas ICS hasn't produced it at all yet.
If anyone is interested I can supply a small demo app illustrating the issue and IP I connect to - it's a public IP so anyone can access it. Otherwise for now, I'll just have to work around it. I'm reluctant to switch to ICS as ICS may work fine in this instance and given the use of this socket stuff is pretty much the whole crux of the program, it would be nasty to have to entirely replace Indy with ICS.
The last parameter (True)
TCPClient.IOHandler.ReadBytes(Buffer, 10, True);
causes the read to append instead of replace the buffer content.
This requires that size and content of the buffer are set up correctly first.
If the parameter is False, the buffer content will be replaced for the given number of bytes.
ReadBytes() does not inject rogue bytes into the buffer, so there are only two possibilities I can think of right now given the limited information you have provided:
The device really is sending an extra byte upon initial connection, like mj2008 suggested. If a packet sniffer is not detecting it, try attaching one of Indy's own TIdLog... components to your TIdTCPClient, such as TIdLogFile or TIdLogEvent, to verify what TIdTCPClient is actually receiving from the socket.
you have another thread reading from the same connection at the same time, corrupting the InputBuffer. Even a call to TIdTCPClient.Connected() will perform a read. Don't perform reads in multiple threads at the same time, if you are using the threads.