I want to write a receiver program using raw socket
it will use recvfrom() to receive packets
so I want to check the IP header and tcp header of a packet
when a program sends a packet, it will pay attention to the network byte order and host byte order problem
but for my recever program, when I use recvfrom(sockfd,mesg,1000,0,(struct sockaddr *)&cliaddr,&len);
what is the byte order of the data in the packets? it is network byte order or host byte order?
and how to deal with it?
for this example
http://www.binarytides.com/packet-sniffer-code-in-c-using-linux-sockets-bsd/
the author doesn't take into account the byte order problem when dealing with the received packets, why?
thanks!
what is the byte order of the data in the packets?
The convention is that network order is big endian order. However, the data you receive is the data you sent: nobody magically modifies "integers" to change their endianness.
and how to deal with it?
Use ntohl and ntohs when interpreting integer data
Be aware that bitfield endianness isn't standard
the author doesn't take into account the byte order problem when
dealing with the received packets,
The link you posted shows ntohs and ntohl calls. The author does handle endianness at least to some extent.
Related
I've known about SCTP for a decade or so, and although I never got to use it yet, I've always wanted to, because of some of its promising (purported) features:
multi-homing
multiplexing w/o head-of-line blocking
mixed order/unordered delivery on the same connection (aka association)
no TIME_WAIT
no SYN flooding
A Comparison between QUIC and SCTP however claims
SCTP intended to get rid of HOL-Blocking by substreams, but its
Transmission Sequence Number (TSN) couples together the transmission
of all data chunks. [...] As a result, in SCTP if a packet is lost,
all the packets with TSN after this lost packet cannot be received
until it is retransmitted.
That statement surprised me because:
removing head-of-line blocking is a stated goal of SCTP
SCTP does have a per-stream sequence number, see below quote from RFC 4960, which should allow processing per stream, regardless of the association-global TSN
SCTP has been in use in the telecommunications sector for perhaps close to 2 decades, so how could this have been missed?
Internally, SCTP assigns a Stream Sequence Number to each message
passed to it by the SCTP user. On the receiving side, SCTP ensures
that messages are delivered to the SCTP user in sequence within a
given stream. However, while one stream may be blocked waiting for
the next in-sequence user message, delivery from other streams may
proceed.
Also, there is a paper Head-of-line Blocking in TCP and SCTP: Analysis and Measurements that actually measures round-trip time of a multiplexed echo service in the face of package loss and concludes:
Our results reveal that [..] a small number of SCTP streams or SCTP unordered mode can avoid this head-of-line blocking. The alternative solution of multiple TCP connections performs worse in most cases.
The answer is not very scholarly, but at least according to the specification in RFC 4960, SCTP seems capable of circumventing head-of-line blocking. The relevant claim seems to be in Section 7.1.
Note: TCP guarantees in-sequence delivery of data to its upper-layer protocol within a single TCP session. This means that when TCP notices a gap in the received sequence number, it waits until the gap is filled before delivering the data that was received with sequence numbers higher than that of the missing data. On the other hand, SCTP can deliver data to its upper-layer protocol even if there is a gap in TSN if the Stream Sequence Numbers are in sequence for a particular stream (i.e., the missing DATA chunks are for a different stream) or if unordered delivery is indicated. Although this does not affect cwnd, it might affect rwnd calculation.
A dilemma is what does "are in sequence for a particular stream" entail? There is some stipulation about delaying delivery to the upper layer until packages are reordered (see Section 6.6, below), but reordering doesn't seem to be conditioned by filling the gaps at the level of the association. Also note the mention in Section 6.2 on the complex distinction between ACK and delivery to the ULP (Upper Layer Protocol).
Whether other stipulations of the RFC indirectly result in the occurence of HOL, and whether it is effective in real-life implementations and situations - these questions warrant further investigation.
Below are some of the excerpts which I've come across in the RFC and which may be relevant.
RFC 4960, Section 6.2 Acknowledgement on Reception of DATA Chunks
When the receiver's advertised window is 0, the receiver MUST drop any new incoming DATA chunk with a TSN larger than the largest TSN received so far. If the new incoming DATA chunk holds a TSN value less than the largest TSN received so far, then the receiver SHOULD drop the largest TSN held for reordering and accept the new incoming DATA chunk. In either case, if such a DATA chunk is dropped, the receiver MUST immediately send back a SACK with the current receive window showing only DATA chunks received and accepted so far. The dropped DATA chunk(s) MUST NOT be included in the SACK, as they were not accepted.
Under certain circumstances, the data receiver may need to drop DATA chunks that it has received but hasn't released from its receive buffers (i.e., delivered to the ULP). These DATA chunks may have been acked in Gap Ack Blocks. For example, the data receiver may be holding data in its receive buffers while reassembling a fragmented user message from its peer when it runs out of receive buffer space. It may drop these DATA chunks even though it has acknowledged them in Gap Ack Blocks. If a data receiver drops DATA chunks, it MUST NOT include them in Gap Ack Blocks in subsequent SACKs until they are received again via retransmission. In addition, the endpoint should take into account the dropped data when calculating its a_rwnd.
Circumstances which highlight how senders may receive acknowledgement for chunks which are ultimately not delivered to the ULP (Upper Layer Protocol).Note this applies to chunks with TSN higher than the Cumulative TSN (i.e. from Gap Ack Blocks). This together with unreliability of SACK order represent good reasons for the stipulation in Section 7.1 (see below).
RFC 4960, Section 6.6 Ordered and Unordered Delivery
Within a stream, an endpoint MUST deliver DATA chunks received with the U flag set to 0 to the upper layer according to the order of their Stream Sequence Number. If DATA chunks arrive out of order of their Stream Sequence Number, the endpoint MUST hold the received DATA chunks from delivery to the ULP until they are reordered.
This is the only stipulation on ordered delivery within a stream in this section; seemingly, reordering does not depend on filling the gaps in ACK-ed chunks.
RFC 4960, Section 7.1 SCTP Differences from TCP Congestion Control
Gap Ack Blocks in the SCTP SACK carry the same semantic meaning as the TCP SACK. TCP considers the information carried in the SACK as advisory information only. SCTP considers the information carried in the Gap Ack Blocks in the SACK chunk as advisory. In SCTP, any DATA chunk that has been acknowledged by SACK, including DATA that arrived at the receiving end out of order, is not considered fully delivered until the Cumulative TSN Ack Point passes the TSN of the DATA chunk (i.e., the DATA chunk has been acknowledged by the Cumulative TSN Ack field in the SACK).
This is stated from the perspective of the sending endpoint, and is accurate for the reason emphasized in section 6.6 above.
Note: TCP guarantees in-sequence delivery of data to its upper-layer protocol within a single TCP session. This means that when TCP notices a gap in the received sequence number, it waits until the gap is filled before delivering the data that was received with sequence numbers higher than that of the missing data. On the other hand, SCTP can deliver data to its upper-layer protocol even if there is a gap in TSN if the Stream Sequence Numbers are in sequence for a particular stream (i.e., the missing DATA chunks are for a different stream) or if unordered delivery is indicated. Although this does not affect cwnd, it might affect rwnd calculation.
This seems to be the core answer to what interests you.
In support of this argument, the format of the SCTP SACK chunk as exposed here and here.
After that I send different runicast messages with the function runicast_send, how can I understand which message was acknowledged when the callback sent_runicast is triggered?
The runicast.h file states:
The runicast primitive adds two packet attributes: the single-hop
packet type and the single-hop packet ID. The runicast primitive
uses the packet ID attribute as a sequence number for matching
acknowledgement packets to the corresponding data packets.
but I didn't understand how to do it in practice. Can somebody provide an example?
One way would be to look at the field sndnxt of struct runicast_conn *c before you send the packet, and then compare that value of the packetbuf_attr(PACKETBUF_ATTR_PACKET_ID) in the "sent" callback of your code.
However note that by default the runicast packet ID is just 2 bits long. Enough to demultiplex the ACK in most cases, but may be insufficient for your purposes. (The packet ID size in bits can be changed by redefining RUNICAST_PACKET_ID_BITS.)
Also Rime is obsolete. Don't use it in your code, especially production code unless you know what you're doing. runicast was never one of the highlights of Rime, I doubt there are no better alternatives (e.g. the uIPv6 stack) for what you want to do.
How can I handle buffer overflow in ndis driver. Can anybody tell some buffer overflow scenarios or some use cases of buffer overflow conditions.
For NDIS miniport drivers
If you receive a packet that is larger than the MTU, discard it. Do not indicate the packet up to NDIS (i.e., do not pass the packet to NdisMIndicateReceiveNetBufferLists). If possible, increment the ifInErrors statistical counter.
The above rule is not affected by the NDIS_PACKET_TYPE_PROMISCUOUS flag; do not indicate excessively-large packets even when in promiscuous mode. However, you should indicate excessively-small (aka "runt") packets when in promiscuous mode, if your hardware permits it.
If you are asked to transmit a packet that is larger than the MTU, do not attempt to transmit it. Assign NET_BUFFER_LIST::Status = NDIS_STATUS_INVALID_LENGTH and return the NBL back to NDIS with NdisMSendNetBufferListsComplete. (I wouldn't expect you to ever see such a packet; it would be a bug for NDIS to attempt to send you such a packet.)
For NDIS protocol drivers
If you receive a packet that is larger than the MTU, you are free to discard it.
Never attempt to send a packet that is larger than the MTU.
For NDIS filter drivers
If a filter receives a packet that is larger than the MTU (FilterReceiveNetBufferLists), the filter may immediately discard the packet (NdisFReturnNetBufferLists if the receive indication is not made with NDIS_RECEIVE_FLAGS_RESOURCES, or just returning immediately if the resources flag is set).
If a filter is asked to send a packet that is larger than the MTU (FilterSendNetBufferLists), the filter may assign NET_BUFFER_LIST::Status = NDIS_STATUS_INVALID_LENGTH and return the packet immediately (NdisFSendNetBufferListsComplete).
Filters are not obligated to validate the size of every packet that passes through them. However, your filter should validate the size of any packets where a malformed packet would otherwise cause your filter to trigger a buffer overflow. For example, if your filter copies all ARP replies into a pre-allocated buffer, first validate that the ARP reply isn't too large to fit into the buffer. (This is not strictly necessary, since the miniport "shouldn't" give you an excessively-large packet. However, you are on the network datapath, which means you're handling untrusted data being processed by a potentially-buggy miniport. A little extra defense-in-depth is a good idea.)
Filters must not originate packets that are larger than the MTU (on either the send or receive paths).
A client and server communicate with each other via TCP. The server and client send each other UTF-8 encoded messages.
When encoding UTF-8, the amount of bytes per character is variable. It could take one or more bytes to represent a single character.
Lets say that I am reading a UTF-8 encoded message on the network stream and it is a huge message. In my case it was about 145k bytes. To create a buffer of this size to read from the network stream could lead to an OutMemoryException since the byte array needs that amount of sequential memory.
It would be best then to read from the network stream in a while loop until the entire message is read, reading the pieces in to a smaller buffer (probably 4kb) and then decoding the string and concatenating.
What I am wondering is what happens when the very last byte of the read buffer is actually one of the bytes of a character which is represented by multiple bytes. When I decode the read buffer, that last byte and the beginning bytes of the next read would either be invalid or the wrong character. The quickest way to solve this in my mind would be to encode using a non variable encoding (like UTF-16), and then make your buffer a multiple of the amount of bytes in each character (with UTF-16 being a buffer using the power 2, UTF-32 the power of 4).
But UTF-8 seems to be a common encoding, which would leave me to believe this is a solved problem. Is there another way to solve my concern other than changing the encoding? Perhaps using a linked-list type object to store the bytes would be the way to handle this since it would not use sequential memory.
It is a solved problem. Woot woot!
http://mikehadlow.blogspot.com/2012/07/reading-utf-8-characters-from-infinite.html
I want to send files(text or binary) through winsock,I have a buffer with 32768 byte size, In the other side the buffer size is same,But when the packet size <32768 then i don't know how determine the end of packet in buffer,Also with binary file it seems mark the end of packet with a unique character is not possible,Any solution there?
thx
With fixed-size "packets," we would usually that every packet except the last will be completely full of valid data. Only the last one will be "partial," and if the recipient knows how many bytes to expect (because, using Davita's suggestion, the sender told it the file size in advance), then that's no problem. The recipient can simply ignore the remainder of the last packet.
But your further description makes it sound like there may be multiple partially full packets associated with a single file transmission. There is a similarly easy solution to that: Prefix each packet with the number of valid bytes.
You later mention TCustomWinSocket.ReceiveText, and you wonder how it knows how much text to read, and then you quote the answer, which is that it calls ReceiveBuf(Pointer(nul)^, -1)) to set the length of the result buffer before filling it. Perhaps you just didn't understand what that code is doing. It's easier to understand if you look at that same code in another context, the ReceiveLength method. It makes that same call to ReceiveBuf, indicating that when you pass -1 to ReceiveBuf, it returns the number of bytes it received.
In order for that to work for your purposes, you cannot send fixed-size packets. If you always send 32KB packets, and just pad the end with zeroes, then ReceiveLength will always return 32768, and you'll have to combine Davita's and my solutions of sending file and packet lengths along with the payload. But if you ensure that every byte in your packet is always valid, then the recipient can know how much to save based on the size of the packet.
One way or another, you need to make sure the sender provides the recipient with the information it needs to do its job. If the sender sends garbage without giving the recipient a way to distinguish garbage from valid data, then you're stuck.
Well, you can always send file size before you start file transfer, so you'll know when to stop writing to file.