I got a problem when using NSInputStream.
I have client app which connect to a server then server will start to send message to my client app through TCP repeatedly about 1 message per second. Server is just broadcasting message to client and message is xml format. The server send a message as one packet.
Now the problem is that when I read byte from NSInputStream the data got truncated which mean instead of receive 1 complete message, I got 2 separate data(partial xml) respond from time to time. I am not able to debug because it already happen when I read data byte from NSInputStream.
I use Wireshark to analyse every packet I receive and when it happen data got truncated too, because TCP so partial data retransmit to my client.
I have tried to log every partial data byte, the sum of partial data always around 1600 byte.
I have no idea how did they design and implement server side, but I do know there are many of people connect to that server and continuous get broadcasting message from it.
Does anyone encounter this problem? Can anyone help? Is it possible that data is over the max size and get splited?
This is not a problem per se. It is part of the design of TCP and also of NSInputStream. You may receive partial messages. It's your job to deal with that fact, wait until you receive a full message, and then process the completed message.
1600 bytes is a little strange. I would expect 1500 bytes, since that's the largest legal Ethernet packet (or especially somewhere around 1472, which is a really common MTU, minus some for the headers). Or I might expect a multiple of 1k or 4k due to buffering in NSInputStream. But none of that matters. You have to deal with the fact that you will not get messages necessarily at their boundaries.
Related
I have been trying to implement a wrapper library for the Linux interface to SCTP sockets, and I am not sure how to integrate the asynchronous style of errors (where they are delivered via events). All example code I have seen, if it deals with the errors at all, simply prints out the information related to the error when it is received, but inserting error-handling code there seems like it would be ineffective, because by that point all of the context related to the original message which was sent has been lost and only a 32-bit integer sinfo_context remains. It also seems that there is no way to directly tell when a given message has been acknowledged successfully by the remote peer, which would make it impossible to implement an approach which listens for errors after sending a message, because the context information for successfully-delivered messages could never be freed.
Is there a way to handle the errors related to a given sending operation as part of the call to a send function, or is there a different way to approach error handling for SCTP which does not lose the context of the error?
One solution which I have considered is using the SCTP_SENDER_DRY notification to tell when packets have been sent, however this requires sending only one packet at a time. Another idea is to use the peer's receiver window size together with the sinfo_cumtsn field of sctp_sndrcvinfo to calculate how much data has been acknowledged as fully received using the cumulative TSN, however there are a couple of disadvantages to this: first, it requires bookkeeping overhead to calculate a number of bytes received by the peer based on the cumulative TSN (especially if the peer's window size may change); second, it requires waiting until all earlier packets were received before reporting success, which seems to defeat the purpose of SCTP's multistreaming; and third, it seems like it would not work for unordered packets.
I am receiving data through a TCP socket and although this code has been working for years, I came across a very odd behaviour trying to integrate a new device (that acts as a server) into my system:
Before receiving the HTTP Body response, the recv() kernel function gives me strange characters like '283' or '7b'.
I am actually debuging with gdb and I can see that the variables hold these values right after recv() was called (so it is not just what printf shows me)
I always read byte-after-byte (one at a time) with the recv() function and the returned value is always positive.
This first line of the received HTTP Body cannot be seen in Wireshark (!) and is also not expected to be there. In Wireshark I see what I would expect to receive.
I changed the device that sends me the data and I still receive the exact same values
I performed a clean debug build and also tried a release version of my programm and still get the exact same values, so I assume these are not random values that happened to be in memory.
i am running Linux kernel 3.2.58 without the option to upgrade/update.
I am not sure what other information i should provide and I have no idea what else to try.
Found it. The problem is that I did not take the Transfer-Encoding into consideration, which is chunked. I was lucky because also older versions of Wireshark were showing these bytes in the payload so other people also posted similar problems in the wireshark forum.
Those "strange" bytes show you the payload length that you are supposed to receive. When you are done reading this amount of bytes, you will receive again a number that tells you whether you should continue reading (and, again, how many bytes you will receive). As far as I understood, this is usefull when you have data that change dynamically and you might want to continuously get their current value.
Hi sorry if this is a stupid question (I just started learning network programming), but I've been looking all over google about how files/data are divided into packets. I've read everywhere that somehow files are broken up into packets have headers/footers applied as they go through the OSI model and are sent through the wire where the recipient basically does the reverse and removes the headers.
My question is how exactly are files/data broken up into packets and how are they reassembled at the other end?
How does whatever doing the reassembling know when the last packet of the data has arrived and etc?
Is it possible to reassemble packets captured from another machine? And if so how?
(Also if it means anything I'm mostly interested in how this work for TCP type packets)
I also have some packets captured from an application on my computer through WireShark, they're labeled as TCP protocol, what I want to do is reassemble them back into the original data, but how can you tell which packets belong to which set of data?
Any pointers towards resources is much appreciated, thank you!
My question is how exactly are files/data broken up into packets
What's being sent over a network isn't necessarily a file. In the cases where it is a file, there are several different protocols that can send files, and the answer to the question depends on the protocol.
For FTP and HTTP, the entire contents of the file is probably being sent as a single data stream over TCP (preceded by headers in the case of HTTP, and just raw, over the connection, in the case of FTP).
For TCP, there's a "maximum segment size" negotiated by the client and server, based on factors such as the maximum packet size on the various networks between the server and client, and the file data is sent, sequentially, in chunks whose size is limited by the maximum packet size and the size of IP and TCP headers.
For remote file access protocols such as SMB, NFS, and AFP, what goes over the wire are "file read" and "file write" requests; the reply to a "file read" request includes some reply headers and, if the read is successful, the chunk of file data that the read request asked for, and a "file write" request includes some request headers and the chunk of file data being written. Those are not guaranteed to be an entire file, in order, but if the program reading or writing the file is reading or writing the entire file in sequential order, the entire file's data will be available. The packet sizes will depend on the size of the read reply/write request headers and on the read or write size being used; those packets might be broken into multiple TCP segments, based on the TCP "maximum segment size" and the size of the IP and TCP headers.
My question is how exactly are files/data broken up into packets
For FTP, the recipient of the data knows that there is no more data when the side of the TCP connection over which the data is being transmitted is closed.
For HTTP, the recipient of the data knows that there is no more data when the side of the TCP connection over which the data is being transmitted is closed or, if the connection is "persistent" (i.e., it remains open for more requests and replies), when the amount of data specified by the "Content-Size:" header, sent before the data, has been transmitted (or other similar mechanisms, such as the "last chunk" indication for chunked encoding).
For file access protocols, there's no real "we're at the end of data" indication; the closest approximation, for SMB, AFP, and NFSv4, is a "file close" operation.
Is it possible to reassemble packets captured from another machine? And if so how?
It depends on the protocol, but, for HTTP and SMB, if the capture has been read into Wireshark (and all the file data is in the capture!), you can use the "Export Objects" menu, and, for some protocols, you might also be able to use tcpflow.
My question is how exactly are files/data broken up into packets and how are they reassembled at the other end?
They are basically just chopped up. Each internet packet (with header info add) can only hold a few hundred bytes of actual data.
How does whatever doing the reassembling know when the last packet of the data has arrived and etc?
For a transfer the packets are numbered, so the receiving process knows how to put them together. If it loses a packet, it can request a resend.
Is it possible to reassemble packets captured from another machine? And if so how?
I don't understand the question. How would you get these packets unless you were a man-in-the-middle?
These answers are true for TCP packets.
First determine what size you want to transmit.
then put header, data and footer for each transmission.
See buffer length and data array should be divisible by number of packets without giving fractions.
Here header should contain frame number, time stamp, packet number
payload data
footer ---your company information.
prepare data fragments before sending
My app communicates with a server over TCP, using AsyncSocket. There are two situations in which communication takes place:
The app sends the server something, the server responds. The app needs to read this response and do something with the information in it. This response is always the same length, e.g., a response is always 6 bytes.
The app is "idling" and the server initiates communication at some time (unknown to the app). The app needs to read whatever the server is sending (could be any number of bytes, but the first byte will indicate how many bytes are following so I know when to stop reading) and process this information.
The first situation is working fine. readDataToLength:timeout:tag returns what I need and I can do with it what I want. It's the second situation that I'm unsure of how to implement. I can't use readDataToLength:timeout:tag, since I don't know the length beforehand.
I'm thinking I could do something with readDataWithTimeout:tag:, setting the timeout to -1. That makes the socket to constantly listen for anything that's coming in, I believe. However, that will probably interfere with data that's coming in as response to something I sent out (situation 1). The app can't distinguish incoming data from situation 1 or situation 2 anymore.
Anybody here who can give me help me solve this?
Your error is in the network protocol design.
Unless your protocol has this information, there's no way to distinguish the response from the server-initiated communication. And network latency prevents obvious time-based approach you've described from working reliably.
One simple way to fix the protocol in your case (if the server-initiated messages are always less then 255 bytes) - add the 7-th byte to the beginning of the response, with the value FF.
This way you can readDataWithTimeout:tag: for 1 byte.
On timeout you retry until there's a data.
If the received value is FF, you read 6 more bytes with readDataToLength:6 timeout: tag:, and interpret it as the response to the request you've sent earlier.
If it's some other value, you read the message with readDataToLength:theValue timeout: tag:, and process the server-initiated message.
Our school project is a BitTorrent client. Today i suddenly got a {tcp_error,#Port<0.2095>,emsgsize} error and my question is what caused this error? I have option {packet,4} on gen_tcp so my guess is that the length off the package does not match the 4 first bytes? That would be really strange because all BitTorrent messages except for the handshake have first 4 bytes len. Yesterday we were able to download and now i get these messages. Note that some messages arrives just fine. Thanks for your thoughts on the problem.
you will get an emsgsize error when the packet is bigger than your receive buffer (recbuf option) or when the packet is bigger than the specified maximum packet size (packet_size options).
Probably one of the packets is not sent with the correct header, which Erlang is interpreting as a header that claims the packet is very large.