I have one concern as I have just started working with MQTT and CoAP in my school.
I did data packet transmission using both protocols separately, For MQTT I used mosquitto and libcoap is used for CoAP, the transmitted packet contains the information “Temperature=22*C”. I know the difference in Header sizes in the packet for both protocols. But My question is why the payload size is different even though it is containing the same information.
I know the TCP header is larger than UDP header. But for MQTT, the payload is 22 bytes and for CoAP it is 4 bytes. Why the payload is compressed for CoAP with the same info?
Please help me with an answer.
As you noted TCP packet and UDP datagram sizes will differ at least due to TCP and UDP headers sizes difference.
MQTT message (TCP packed body) and CoAP packed (UDP datagram body) will differ at least due to MQTT & CoAP messages headers difference.
MQTT payload and CoAP payload (omitting MQTT and CoAP headers) might differ due to different serialization protocol. The string can be transmitted being serialized in different formats: JSON, TLV, CBOR, plaintext. You need to check what is used for each message in your case.
Related
In most of IoT platforms, MQTT is used as M2M communication, one of the reasons being lightweight.
Device ---N/W--> MQTT Broker ---> another device
The device communicates to MQTT Broker over TCP/IP, which means there would be a payload addition as part of the TCP/IP layer.
This is my start of confusion:
If MQTT runs over TCP/IP, then how come it's a lightweight protocol?
What are you comparing MQTT to?
The problem with your question is the starting premise to compare MQTT to the underlying TCP/IP that it's using as it's base transport.
Since MQTT runs on top of TCP/IP it's not a valid comparison, try comparing it to say HTTP (with it's HUGE header) which also runs over TCP/IP.
Setting up a MQTT connection and then subscribing to a topic is handled in a few bytes + the topic name and the connection is persisted. When a message is sent it again has a couple of bytes of head + topic and the payload.
By comparison HTTP requests start with the URL + a bunch request headers, the response includes a whole bunch more response headers (there can easily be 100s of bytes of headers as it's all encoded as text) before we get to the payload and in general the connection is closed after the payload.
If you add in TLS/SSL overheads of starting up a new connection for each payload HTTP get even worse.
I made a tcpdump and captured packets, the configured MTU is 2140. I am analysing pcap files using Wireshark.
According to the configured MTU the expected maximum size of the packets should be 2154 (2140 bytes +14 ethernet header bytes). But I see packets of size greater than 2154 (ex 9010 bytes), On analyzing I found that these packets are generated on the machine where I made tcpdump (let's say A) and have the destination to another machine (let's say B). I expect a packet to be fragmented before it is sent to another host. I found some explanations online that says tcpdump captures packets before NIC breakdown, though this seems to be a valid explanation but it seems to contradict in my case because on machine A, I received packets of size greater than 2154 from B. Any thoughts, on why machine A is sending and receiving packets greater than configured MTU.
What you are seeing is most likely the result of TCP Segment Reassembly Offloading. This is a feature available on some network cards with matching drivers.
The idea is that the reassembly of many of the TCP segments is handled in the NIC itself. This turns out to be pretty effective in reducing overhead on the CPU/OS side since the network driver need only handle, perhaps, 1 "packet" out of 10, seeing just one large packet, rather than receiving and reassembling all 10.
You can read more about it here.
Updated answer
If your packet is UDP
This behaviour is normal. but there is not much you can do to see the individual packets on the end machines. The UDP packet is broken down into MTU compliant packets and reassembled at the Link layer, usually by specific hardware. This is too low to to be captured by Wireshark/pcap.
If you want to capture the individual broken down packets, you have to do this on an intermediate machine/network card, for example a gateway between the two hosts, because the original UDP packet is not reassembled until it reaches its final destination. Note : this gateway can be virtual.
See notes.shichao.io/tcpv1/ch10
Leaving this here in case someone with the same problem comes...
If your packet is TCP :
It sounds like Wireshark is reassembling packets for you. This is often the default for TCP streams. You can change this by richt-click on a stream -> Protocol Preferences -> Allow subdissectors to reassemble TCP.
I have a great confusion about tshark.
What is the basic unit of the messages captured by tshark? Ip, tcp, or http?
I see TCP http or ssl and so on in the protocol column in wireshark.
These protocols are in different layer.
In addition, what is the tshark command to capturing the http message with tshark.
See the man page. To quote:
TShark is a network protocol analyzer. It lets you capture packet data from a live network, or read packets from a previously saved capture file
Packets are usually either TCP or UDP (you can find a list of protocols here), When wireshark states that it is HTTP or SSL, then it is based on introspection of the packet (Probably based on destination port).
I want to send broadcast message to all peers in my local network. Message is a 32 bit integer. I can be sure, that message will not me fragmented, right? There will be two options:
- peer will receive whole message at once
- peer will not receive message at all
Going further, 4 bytes is maximum size of data, that can be sent in one UDP datagram? I use Ethernet based network in 99%.
IPv4 specifies a minimum supported MTU of 576 bytes, including the IP header. Your 4 byte UDP payload will result in an IP packet far smaller than this, so you need not fear fragmentation.
Furthermore, your desired outcome - "peer will receive whole message at once or peer will not receive message at all" is always how UDP works, even in the presence of fragmentation. If a fragment doesn't arrive, your app won't recieve the packet at all.
The rules for UDP are "The packet may arrive out-of-order, duplicated, or not at all. If the packet does arrive, it will be the whole packet and error-free.". ("Error-free" is obviously only true within the modest limits of the IP checksum).
Ethernet packets can be up to around 1500 bytes (and that's not counting jumbo frames). If you send broadcast messages with a payload of only 4 bytes, they shouldn't get fragmented at all. Fragmentation should only occur when the packet is larger than the Maximum Transmission Unit (so about 1500 bytes over Ethernet).
I have created a raw socket which takes all IPv4 packets from data link layer (with data link layer header removed). And for reading the packets I use recvfrom.
My doubt is:
Suppose due to some scheduling done by OS, my process was asleep for 1 sec. When it woke up,it did recvfrom (with number of bytes to be received say 1000) on this raw socket (with the intention of receiving only one IPv4 packet and say the size of this packet is 380 bytes). And suppose many network applications were also running simultaneously during this time, so all IPv4 packets must have been queued in the receive buffer of this socket. So now recvfrom will return all 1000 bytes (with other IPv4 packets from 381th byte onwards) bcoz it has enough data in its buffer to return. Although my program was meant to understand only one IPv4 packet
So how to prevent this thing? Should I read byte by byte and parse each byte but it is very inefficient.
IIRC, recvfrom() will only return one packet at a time, even if there are more in the queue.
Raw sockets operate at the packet layer, there is no concept of data streams.
You might be interested in recvmmsg() if you want to read multiple packets in one system call. Recent Linux kernels only, there is no equivalent send side implementation.