I am writing a bulletin board system (BBS) reader on ios. I use GCDAsyncSocket library to handle packets sending and receiving. The issue that I have is the server always splits the data to send into multiple packets. I can see that happens by printing out the receiving string in didReceiveData() function.
From the GCDAsyncSocket readme, I understand TCP is a stream. I also know there are some end of stream mechanisms, such as double CR LFs at the end. I have used WireShark to parse the packets, but there is no sign of some sort of pattern in the last data packet. The site is not owned by me, so I couldn't make it to send certain bytes. There must be some way to detect the last packet, otherwise how BBS clients handle displaying data?
Double CR LFs are not end of stream. That is just part of the details of HTTP protocol, for example, but has nothing to do with closing the stream. HTTP 1.1 allows me to send multiple responses on a single stream, with double CR LF after HTTP header, without end of stream.
The TCP socket stream will return 0 on a read when it is closed from the other end, blocking or non-blocking.
So assuming the server will close the socket when it is done sending to you, you can loop and perform a blocking read and if returns > 0, process the data, then read again. if < 0, process the error code (could be fatal or not), and if == 0, socket is closed from the other side, don't read again.
For non-blocking sockets, you can use select() or some other API to detect when the stream becomes ready to read. I'm not familiar with the specific lib you are using but if it is a POSIX / Berkeley sockets API, it will work that way.
In either case, you should build a buffer of input, concatenating the results of each read until you are ready to process. As you've found, you can't assume that a single read will return a complete application level packet. But as to your question, unless the server wants you to close the socket, you should wait for read to return 0.
Related
I am developing an app that needs to send large amounts of data between an iPhone and a device (it takes approximately 10 seconds to send the data). But I want to be able to cancel the data communication anytime. I am aware I can simply drop the connection to the device at anytime with
centralManager.cancelPeripheral(peripheral)
but that is not what I am actually looking for, as I want to stop sending data but without terminating the bluetooth connection.
Is there a way to terminate the data transmission without dropping the connection to the device?
the codes of sending data is as follow:
for (var Hex: UInt8 = 0x01; Hex <= 0x14; Hex+=1){
var outbuffer = [UInt8](count: 16, repeatedValue: 0x00)
outbuffer[0] = (0x68)
outbuffer[1] = (Hex)
let data = NSData(bytes: outbuffer, length: 7)
print("data\(data)")
connectingPeripheral.writeValue(data, forCharacteristic: connectingCharacteristicPassword , type: CBCharacteristicWriteType.WithResponse)
}
I figured that I would go ahead and give my input on this. There is no way in CoreBluetooth to stop the transmission of a data packet that has already been written to the output buffer. The reason for why this is the case is simply because it is not needed and it would be a useless functionality. The only reason for why you are having this issue is because your methodology is wrong in my opinion. Do not put everything in a for-loop and push the data all at once. Instead you should implement some sort of flow control mechanism.
In Bluetooth LE there are two main ways of writing data to a peripheral: “Write Commands” and “Write Requests”. You can look at it a bit like the TCP vs UDP protocols. With write commands you are just sending data without knowing whether or not the data was received by the application on the other side of the bluetooth link. With write requests you are sending data and letting the peripheral know that you want to be notified (ack’ed) that the data was in fact received. These two types are in CoreBluetooth called CBCharacteristicWriteWithResponse and CBCharacteristicWriteWithoutResponse. When writing data using the CBCharacteristicWriteWithResponse (like you are doing in your code) you will get a peripheral:didWriteValueForCharacteristic:error: callback which verifies that the data has arrived at the other side. At this point you now have the option to go ahead and send the next packet if you want to, but if you for some reason want to stop sending data, then you can do that as well. Doing it this way you are in control of the whole flow and not just simply pushing everything though a for-loop.
But wait, why would you ever want to use write commands then? Well, since write requests requires the receiver to respond back to the sender it means that data must be sent in both directions. In this case, since the ack is sent by the application layer, you have to wait for the next connection interval before the ack can be sent. This means that when sending large amounts of data you can only send one packet per every two connection intervals which will give you a very poor overall bit rate.
With write commands, since they are not ack’ed, you can send as manny packets as possible within one connection event window. In most cases you should be able to send about 10-20 packets per connection window. But be aware that if you send too many packets then you will fill the outgoing buffer and packets will be lost. So, something that you can try is to directly send 9 packets with the WriteWithoutResponse type, followed by 1 packet of the WriteWithResponse type. After doing this you can wait for the peripheral:didWriteValueForCharacteristic:error: callback in which you can then send 10 more packets the same way. This way you will manage to send 10 packets per every 2 connection intervals while still being able to control the flow better.
You can of course experiment with the ratio a bit, but remember that the buffer is shared between multiple applications on the iOS device so you don’t want to be too close to the limit.
Can't seem to find anywhere informations about this, but, is TIdUDPServer.OnRead event passing everything that comes in to the AData array or not?
According to WireShark readings, I'm missing 42 bytes of data; While I should be getting 572 bytes of data on each reading, the AData size is always 530, and seems like always the same bytes are missing.
The device that sends data is broadcasting it, and I can get everything I need except for 2 bytes, which seems to be 2 of those that are missing.
Any hints on this one?
Edit:
I should mention that these are the very first 42 bytes; Everything afterwards is received fine;
The OnUDPRead event passes everything the socket receives from the OS. UDP operates on messages. Unlike TCP, a UDP read is an all-or-nothing operation, either a whole UDP message is read or an error occurs, there is no in-between.
If you are missing data, then either the OS is not providing it (such as if it belongs to the UDP and/or IP headers), or you are not reading data from the AData parameter correctly. If you think this is not the case, then you need to update your question to show your actual OnUDPRead handler code, an example WireShark dump showing the data being captured from the network, and the data that is making it to your OnUDPRead handler.
Update: The OS does not provide access to the packet headers (unless you are using a RAW socket, which TIdUDPServer does not use, but that is a whole other topic of discussion). The AData parameter of the OnUDPRead event provides only the application data portion of a packet, as that is what the OS provides. You cannot access the packet headers.
That being said, you can get the packet's source IP:Port, at least, via the ABinding.PeerIP and ABinding.PeerPort properties of the OnUDPRead event. However, there is no way to retrieve the other packet header values (nor should you ever need them in most situations), unless you sniff the network yourself, such as with a pcap library.
Hi sorry if this is a stupid question (I just started learning network programming), but I've been looking all over google about how files/data are divided into packets. I've read everywhere that somehow files are broken up into packets have headers/footers applied as they go through the OSI model and are sent through the wire where the recipient basically does the reverse and removes the headers.
My question is how exactly are files/data broken up into packets and how are they reassembled at the other end?
How does whatever doing the reassembling know when the last packet of the data has arrived and etc?
Is it possible to reassemble packets captured from another machine? And if so how?
(Also if it means anything I'm mostly interested in how this work for TCP type packets)
I also have some packets captured from an application on my computer through WireShark, they're labeled as TCP protocol, what I want to do is reassemble them back into the original data, but how can you tell which packets belong to which set of data?
Any pointers towards resources is much appreciated, thank you!
My question is how exactly are files/data broken up into packets
What's being sent over a network isn't necessarily a file. In the cases where it is a file, there are several different protocols that can send files, and the answer to the question depends on the protocol.
For FTP and HTTP, the entire contents of the file is probably being sent as a single data stream over TCP (preceded by headers in the case of HTTP, and just raw, over the connection, in the case of FTP).
For TCP, there's a "maximum segment size" negotiated by the client and server, based on factors such as the maximum packet size on the various networks between the server and client, and the file data is sent, sequentially, in chunks whose size is limited by the maximum packet size and the size of IP and TCP headers.
For remote file access protocols such as SMB, NFS, and AFP, what goes over the wire are "file read" and "file write" requests; the reply to a "file read" request includes some reply headers and, if the read is successful, the chunk of file data that the read request asked for, and a "file write" request includes some request headers and the chunk of file data being written. Those are not guaranteed to be an entire file, in order, but if the program reading or writing the file is reading or writing the entire file in sequential order, the entire file's data will be available. The packet sizes will depend on the size of the read reply/write request headers and on the read or write size being used; those packets might be broken into multiple TCP segments, based on the TCP "maximum segment size" and the size of the IP and TCP headers.
My question is how exactly are files/data broken up into packets
For FTP, the recipient of the data knows that there is no more data when the side of the TCP connection over which the data is being transmitted is closed.
For HTTP, the recipient of the data knows that there is no more data when the side of the TCP connection over which the data is being transmitted is closed or, if the connection is "persistent" (i.e., it remains open for more requests and replies), when the amount of data specified by the "Content-Size:" header, sent before the data, has been transmitted (or other similar mechanisms, such as the "last chunk" indication for chunked encoding).
For file access protocols, there's no real "we're at the end of data" indication; the closest approximation, for SMB, AFP, and NFSv4, is a "file close" operation.
Is it possible to reassemble packets captured from another machine? And if so how?
It depends on the protocol, but, for HTTP and SMB, if the capture has been read into Wireshark (and all the file data is in the capture!), you can use the "Export Objects" menu, and, for some protocols, you might also be able to use tcpflow.
My question is how exactly are files/data broken up into packets and how are they reassembled at the other end?
They are basically just chopped up. Each internet packet (with header info add) can only hold a few hundred bytes of actual data.
How does whatever doing the reassembling know when the last packet of the data has arrived and etc?
For a transfer the packets are numbered, so the receiving process knows how to put them together. If it loses a packet, it can request a resend.
Is it possible to reassemble packets captured from another machine? And if so how?
I don't understand the question. How would you get these packets unless you were a man-in-the-middle?
These answers are true for TCP packets.
First determine what size you want to transmit.
then put header, data and footer for each transmission.
See buffer length and data array should be divisible by number of packets without giving fractions.
Here header should contain frame number, time stamp, packet number
payload data
footer ---your company information.
prepare data fragments before sending
I got a problem when using NSInputStream.
I have client app which connect to a server then server will start to send message to my client app through TCP repeatedly about 1 message per second. Server is just broadcasting message to client and message is xml format. The server send a message as one packet.
Now the problem is that when I read byte from NSInputStream the data got truncated which mean instead of receive 1 complete message, I got 2 separate data(partial xml) respond from time to time. I am not able to debug because it already happen when I read data byte from NSInputStream.
I use Wireshark to analyse every packet I receive and when it happen data got truncated too, because TCP so partial data retransmit to my client.
I have tried to log every partial data byte, the sum of partial data always around 1600 byte.
I have no idea how did they design and implement server side, but I do know there are many of people connect to that server and continuous get broadcasting message from it.
Does anyone encounter this problem? Can anyone help? Is it possible that data is over the max size and get splited?
This is not a problem per se. It is part of the design of TCP and also of NSInputStream. You may receive partial messages. It's your job to deal with that fact, wait until you receive a full message, and then process the completed message.
1600 bytes is a little strange. I would expect 1500 bytes, since that's the largest legal Ethernet packet (or especially somewhere around 1472, which is a really common MTU, minus some for the headers). Or I might expect a multiple of 1k or 4k due to buffering in NSInputStream. But none of that matters. You have to deal with the fact that you will not get messages necessarily at their boundaries.
Sometimes when I try to send some packets continuously( I am using the send() API ) I receive this error. Now I am not sure what should I do than. I have these questions:
1) Can I re-send again ? If yes then after how much time should I try again. Is there any particular strategy to be followed
2) Is buffer size has exceeded its limits is the only reason ?
3) Can someone please give me a better idea/code, how to handle such scenario.
Thanks.
Sambit.
From send(): "EAGAIN -- The socket is marked non-blocking and the requested operation would block." and also When the message does not fit into the send buffer of the socket, send normally blocks, unless the socket has been placed in non-blocking I/O mode. In non-blocking mode it would return EAGAIN in this case. The select(2) call may be used to determine when it is possible to send more data.
This thread has a simple example of using select() to deal with EAGAIN, and is followed by significant discussion about what sorts of surprises lurk beneath the surface.
EAGAIN is usually returned when there is no outbound buffer space left. How long to wait depends on the speed of the underlying connection. The normal way is to wait until select() or poll() tells you that the socket is available for writing. If on Linux, take a look at the select_tut(2) manpage, and of course the send(2) manpage.
You could change to blocking operation (which is the default) if you want the call to wait until there is space available. Or you could call select(2) to wait until the socket is writeable and then try again.
There is one other important consideration. If you are sending UDP packets, then keep in mind that there is no guarantee of congestion control, and if you're sending packets over the Internet you will almost certainly get packet loss if you just try sending UDP packets as fast as possible (this doesn't necessarily apply to other datagram sockets such as Unix sockets).