When you want to bind to port in C you have to use htons(port) to convert the port from host byte order to network byte order. This happens because the port number is copied directly to the TCP packets, so they have to match on little-endian and big-endian machines.
Consider the following example in C:
int port = 5000;
struct sockaddr_in addr;
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_ANY);
addr.sin_port = htons(port);
Notice the use of htons. Now, if you run netstat -anp --tcp (on Linux) you'll see that 0.0.0.0:5000 is being listened. It looks like the port number uses host endianness.
Now, a question arises: if port number is host-endian in URL, does this mean that a big-endian client can't use http://a.b.c.d:5000 URL to connect to a little-endian server listening on 0.0.0.0:5000?
No, you're almost certainly misapplying the rules at the wrong abstraction level.
The local browser, in all likelihood, will grab the :5000 string off the end of the URL and use that to create the integer 5000 in host format. It will then pass that to htons as part of constructing the session, in exactly the same way your code snippet does.
And, voila, the structures contain the correct network order.
I trying to use mosquitto broker for an IoT application. I have a embedded hardware, actually not much documented/exampled on the internet. I've succesfully implemented an TCP client on this hardware, and now i can send/listen messages throught any port i want via TCP and i listen via netcat. But when i tried to connect mosquitto, it doesn't accept the literal language. I digged on the internet. The broker take messages like shown below, but even this one not working.
I can not found any documentation. I even tried to watch Wireshark packages, and i can not find any pattern. Any help will appreciated.
$ echo -en "\x10\x0d\x00\x04MQTT\x04\x00\x00\x00\x00\x01a" |nc localhost 1883|hd
00000000 20 02 01 00 | ...|
00000004
Had a similar usecase like you and this is how I managed to decode this message and create my own connect request.
echo -en "\x10\x0d\x00\x04MQTT\x04\x00\x00\x00\x00\x01a" |nc localhost 1883
The above message is broken down as follows:
nc localhost 1883, opens a tcp socket to the mqtt port 1883 (to the broker (on localhost) listening to port 1883)
\x10\x0d\x00\x04MQTT\x04\x00\x00\x00\x00\x01a is the connect packet sent to the socket that was opened. This connect packet can be broken down as follows:
\x10: MQTT connect packet (Constitutes the control field with the 1st 4 bits representing the command type "0001" and the 2nd 4 bits the control flag.
\x0d: Remaining length: is the total length of both the variable header and the payload. This needs to be set after the whole payload is complete.
*Variable header = Protocol Name + Protocol level + Connect Flag byte + Keep Alive
Payload = ClientId, username, password, e.t.c, each entry is provided in the format lengthOfEntry + Entry e.g for a clientId and username and password, this gives: lengthOfClientId + ClientId + lengthOfUsername + Username + lengthOfPassword + Password. *Note: the length is always provided as two bytes.
\x00\x04MQTT: Represents the protocol name "MQTT". The first two bytes 0x00 and 0x04 are the protocol length and MQTT is then the protocol. These give a total of 6 bytes.
\x04: is the protocol level. From MQTT specification, the value of the Protocol Level field for the version 3.1.1 of the protocol is 4 (0x04)
\0x00: is the connect flag byte. each byte is represented as follows:
UsernameFlag|PasswordFlag|Will Retain|Will|QoS|Will Flag|CleanSession|Reserved. 0x00 means none of the flags are set and persistent sessions will be used. Hence the payload doesn't require a username or password as will be seen in the payload bytes.
\0x00\0x00: two bytes that represent the keepAlive time. In this case 0 is provided which means that the server is not required to disconnect the client on grounds of inactivity. keepalive mechanism is turned off
The next bytes are already the payload. Since the connect flag was set such that no username or password is required, then we only have to provide the client id. In the above example the clientid is "a". This is of length 1 byte.
\0x00\0x01: represents the length of the clientId. since we have the character "a" as the clientId we have just the length 1. The length is always given in 16bit (2 bytes).
a: the clientId.
If we count all the bytes, we come up to a total of 13 bytes which gives us our remaining le0gnth as 0x0d.
If you need to add a username and password, set the connect flag accordingly 0b11000000 = 0xC0. The username and password are added to the payload right after the clientId in the order lengthofClientId, clientId, lengthOfUSername, Username, LengthofPassword, Password. The reminaining length needs to be adjusted to reflect this.
Tip: The variable header has a fixed number of bytes of 10. The payload length always depends on the data provided. the length of each element is always 2 bytes. so for the above case, we have a clientid of 1 byte and the length which take 2 bytes to give a total of 10 + 2 + 1 = 13.
I hope this helps you out and answers your question.
Useful links:
https://openlabpro.com/guide/mqtt-packet-format/
http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718024
I am trying to receive data at client side, but nothing is received.
Server code that sends message
client(Socket, Server) ->
gen_tcp:send(Socket,"Please enter your name"),
io:format("Sent confirmation"),
{ok, N} = gen_tcp:recv(Socket,0),
case string:tokens(N,"\r\n") of
[Name] ->
Client = #client{socket=Socket, name=Name, pid=self()},
Server ! {'new client', Client},
client_loop(Client, Server)
end.
Client that should receive and print out
client(Port)->
{ok, Sock} = gen_tcp:connect("localhost",Port,[{active,false},{packet,2}]),
A = gen_tcp:recv(Sock,0),
A.
I think your client is faulty because it specifies:
{packet, 2}
yet the server specifies (in code not shown) :
{packet, 0}
In Programming Erlang (2nd) on p. 269 it says:
Note that the arguments to packet used by the client and the server
must agree. If the server was opened with {packet,2} and the client with {packet,4}, then nothing would work.
The following client can successfully receive text from the server:
%%=== Server: {active,false}, {packet,0} ====
client(Port) ->
{ok, Socket} = gen_tcp:connect(
localhost,
Port,
[{active,false},{packet,0}]
),
{ok, Chunk} = gen_tcp:recv(Socket, 0),
io:format("Client received: ~s", [Chunk]),
timer:sleep(1000),
Name = "Marko",
io:format("Client sending: ~s~n", [Name]),
gen_tcp:send(Socket, Name),
loop(Socket).
loop(Socket) ->
{ok, Chunk} = gen_tcp:recv(Socket, 0),
io:format("Client received: ~s~n", [Chunk]),
loop(Socket).
However, I think that both the chatserver and my client have serious issues. When you send a message through a TCP (or UDP) connection, you have to assume that the message will get split into an indeterminate number of chunks--each with an arbitrary length. When {packet,0} is specified, I think recv(Socket, 0) will only read one chunk from the socket, then return. That chunk may be the entire message, or it might be only a piece of the message. To guarantee that you've read the entire message from the socket, I think you have to loop over the recv():
get_msg(Socket, Chunks) ->
Chunk = gen_tcp:recv(Socket, 0),
get_msg(Socket, [Chunk|Chunks]).
Then the question becomes: how do you know when you've read the entire message so that you can end the loop? {packet,0} tells Erlang not to prepend a length header to a message, so how do you know where the end of the message is? Are more chunks coming, or did the recv() already read the last chunk? I think the marker for the end of the message is when the other side closes the socket:
get_msg(Socket, Chunks) ->
case gen_tcp:recv(Socket, 0) of
{ok, Chunk} ->
get_msg(Socket, [Chunk|Chunks]);
{error, closed} ->
lists:reverse(Chunks);
{error, Other} ->
Other
end.
But that raises another issue: if the chatserver is looping on a recv() waiting for a message from the client, and after the client sends a message to the server the client loops on a recv() waiting for a message from the server, and both sides need the other side to close the socket to break out of their recv() loops, then you will get deadlock because neither side is closing their socket. As a result, the client will have to close the socket in order for the chatserver to break out of its recv() loop and process the message. But, then the server can't send() anything back to the client because the client closed the socket. As a result, I don't know if you can do two way communication when {packet,0} is specified.
Here are my conclusions about {packet, N} and {active, true|false} from reading the docs and searching around:
send():
When you call send(), no data* is actually transferred to the destination. Instead, send() blocks until the destination calls recv(), and only then is data transferred to the destination.
* In "Programming Erlang (2nd)", on p. 176 it says that a small amount of data will be pushed to the destination when you call send() due to the way an OS buffers data, and thereafer send() will block until a recv() pulls data to the destination.
Default options:
You can get the defaults for a socket by specifying an empty list for its options, then doing:
Defaults = inet:getopts(Socket, [mode, active, packet]),
io:format("Default options: ~w~n", [Defaults]).
--output:--
Default options: {ok,[{mode,list},{active,true},{packet,0}]}
You can use inet:getopts() to show that gen_tcp:accept(Socket) returns a socket with the same options as Socket.
{active, true} {active,false}
+--------------+----------------+
{packet, 1|2|4}: | receive | recv() |
| no loop | no loop |
+--------------+----------------+
{packet, 0|raw}: | receive | recv() |
(equivalent) | loop | loop |
+--------------+----------------+
{active, false}
Messages do not land in the mailbox. This option is used to prevent clients from flooding a server's mailbox with messages. Do not try to use a receive block to extract 'tcp' messages from the mailbox--there won't be any. When a process wants to read a message, the process needs to read the message directly from the socket by calling recv().
{packet, 1|2|4}:
The packet tuple specifies the protocol that each side expects messages to conform to. {packet, 2} specifies that each message will be preceded by two bytes, which will contain the length of the message. That way, a receiver of a message will know how long to keep reading from the stream of bytes to reach the end of the message. When you send a message over a TCP connection, you have no idea how many chunks the message will get split into. If the receiver stops reading after one chunk, it might not have read the whole message. Therefore, the receiver needs an indicator to tell it when the whole message has been read.
With {packet, 2}, a receiver will read two bytes to get the length of the message, say 100, then the receiver will wait until it has read 100 bytes from the randomly sized chunks of bytes that are streaming to the receiver.
Note that when you call send(), erlang automatically calculates the number of bytes in the message and inserts the length into N bytes, as specified by {packet, N}, and appends the message. Likewise, when you call recv() erlang automatically reads N bytes from the stream, as specified by {packet, N}, to get the length of the message, then recv() blocks until it reads length bytes from the socket, then recv() returns the whole message.
{packet, 0 | raw} (equivalent):
When {packet, 0} is specified, recv() will read the number of bytes specified by its Length argument. If Length is 0, then I think recv() will read one chunk from the stream, which will be an arbitrary number of bytes. As a result, the combination of {packet, 0} and recv(Socket, 0) requires that you create a loop to read all the chunks of a message, and the indicator for recv() to stop reading because it has reached the end of the message will be when the other side closes the socket:
get_msg(Socket, Chunks) ->
case gen_tcp:recv(Socket, 0) of
{ok, Chunk} ->
get_msg(Socket, [Chunk|Chunks]);
{error, closed} ->
lists:reverse(Chunks);
{error, Other} ->
Other
end.
Note that a sender cannot simply call gen_tcp:close(Socket) to signal that it is done sending data (see the description of gen_tcp:close/1 in the docs). Instead, a sender has to signal that is is done sending data by calling gen_tcp:shutdown/2.
I think the chatserver is faulty because it specifies {packet, 0} in combination with recv(Socket, 0), yet it does not use a loop for the recv():
client_handler(Sock, Server) ->
gen_tcp:send(Sock, "Please respond with a sensible name.\r\n"),
{ok,N} = gen_tcp:recv(Sock,0), %% <**** HERE ****
case string:tokens(N,"\r\n") of
{active, true}
Messages sent through a TCP (or UDP) connection are automatically read from the socket for you and placed in the controlling process's mailbox. The controlling process is the process that called accept() or the process that called connect(). Instead of calling recv() to read messages directly from the socket, you extract messages from the mailbox with a receive block:
get_msg(Socket)
receive
{tcp, Socket, Chunk} -> %Socket is already bound!
...
end
{packet, 1|2|4}:
Erlang automatically reads all the chunks of a message from the socket for you and places a complete message (with the length header stripped off) in the mailbox:
get_msg(Socket) ->
receive
{tcp, Socket, CompleteMsg} ->
CompleteMsg,
{tcp_closed, Socket} ->
io:format("Server closed socket.~n")
end.
{packet, 0 | raw} (equivalent):
Messages will not have a length header, so when Erlang reads from the socket, Erlang has no way of knowing when the end of the message has arrived. As a result, Erlang places each chunk it reads from the socket into the mailbox. You need a loop to extract all the chunks from the mailbox, and the other side has to close the socket to signal that no more chunks are coming:
get_msg(ClientSocket, Chunks) ->
receive
{tcp, ClientSocket, Chunk} ->
get_msg(ClientSocket, [Chunk|Chunks]);
{tcp_closed, ClientSocket} ->
lists:reverse(Chunks)
end.
The recv() docs mention something about recv()'s Length argument only being applicable to sockets in raw mode. But because I don't know when a Socket is in raw mode, I don't trust the Length argument. But see here: Erlang gen_tcp:recv(Socket, Length) semantics. Okay, now I'm getting somewhere: from the erlang inet docs:
{packet, PacketType}(TCP/IP sockets)
Defines the type of packets to use for a socket. Possible values:
raw | 0
No packaging is done.
1 | 2 | 4
Packets consist of a header specifying the number of bytes in the packet, followed by that
number of bytes. The header length can be one, two, or four bytes, and containing an
unsigned integer in big-endian byte order. Each send operation generates the header, and the
header is stripped off on each receive operation.
The 4-byte header is limited to 2Gb [message length].
As the examples at Erlang gen_tcp:recv(Socket, Length) semantics confirm, when {packet,0} is specified, a recv() can specify the Length to read from the TCP stream.
I was told to ask this here:
10:53:04.042608 IP 172.17.2.12.42654 > 172.17.2.6.6000: Flags [FPU], seq 3891587770, win 1024, urg 0, length 0
10:53:04.045939 IP 172.17.2.6.6000 > 172.17.2.12.42654: Flags [R.], seq 0, ack 3891587770, win 0, length 0
This states that the flags set are FPU and R. What flags do these stand for and what kind of exchange is this?
The flags are:
F - FIN, used to terminate an active TCP connection from one end.
P - PUSH, asks that any data the receiving end is buffering be sent to the receiving process.
U - URGENT, indicating that there is data referenced by the urgent "pointer."
R - RESET, indicating that a packet was received that was NOT part of an existing connection.
It looks like the first packet was manufactured, or possibly delayed. The argument for it being manufactured is the urgent flag being set, with no urgent data. If it was delayed, it indicates the normal end of a connection between .12 and .6 on port 6000, along with a request that the last of any pending data sent across the wire be flushed to the service on .6.
.6 has clearly forgotten about this connection, if it even existed. .6 is indicating that while it got the FIN packet, it believes that the connection that FIN packet refers to did not exist.
If .6 had a current matching connection, it would have replied with a FIN-ACK instead of RST, acknowledging the termination of the connection.
I'm trying to write a dissector for the Safari Remote Debug protocol which is based on bplists and have been reasonably successful (current code is here: https://github.com/andydavies/bplist-dissector).
I'm running into difficultly with reassembling packets though.
Normally the protocol sends a packet with 4 bytes containing the length of the next packet, then the packet with the bplist in.
Unfortunately some packets from the iOS simulator don't follow this convention and the four bytes are either tagged onto the front of the bplist packet, or onto the end of the previous bplist packet, or the data is multiple bplists.
I've tried reassembling them using desegment_len and desegment_offset as follows:
function p_bplist.dissector(buf, pkt, root)
-- length of data packet
local dataPacketLength = tonumber(buf(0, 4):uint())
local desiredPacketLength = dataPacketLength + 4
-- if not enough data indicate how much more we need
if desiredPacketLen > buf:len() then
pkt.desegment_len = dataPacketLength
pkt.desegment_offset = 0
return
end
-- have more than needed so set offset for next dissection
if buf:len() > desiredPacketLength then
pkt.desegment_len = DESEGMENT_ONE_MORE_SEGMENT
pkt.desegment_offset = desiredPacketLength
end
-- copy data needed
buffer = buf:range(4, dataPacketLen)
...
What I'm attempting to do here is always force the size bytes to be the first four bytes of a packet to be dissected but it doesn't work I still see a 4 bytes packet, followed by a x byte packet.
I can think of other ways of managing the extra four bytes on the front, but the protocol contains a lookup table thats 32 bytes from the end of the packet so need a way of accurately splicing the packet into bplists.
Here's an example cap: http://www.cloudshark.org/captures/2a826ee6045b #338 is an example of a packet where the bplist size is at the start of the data and there are multiple plists in the data.
Am I doing this right (looking other questions on SO, and examples around the web I seem to be) or is there a better way?
TCP Dissector packet-tcp.c has tcp_dissect_pdus(), which
Loop for dissecting PDUs within a TCP stream; assumes that a PDU
consists of a fixed-length chunk of data that contains enough information
to determine the length of the PDU, followed by rest of the PDU.
There is no such function in lua api, but it is a good example how to do it.
One more example. I used this a year ago for tests:
local slicer = Proto("slicer","Slicer")
function slicer.dissector(tvb, pinfo, tree)
local offset = pinfo.desegment_offset or 0
local len = get_len() -- for tests i used a constant, but can be taken from tvb
while true do
local nxtpdu = offset + len
if nxtpdu > tvb:len() then
pinfo.desegment_len = nxtpdu - tvb:len()
pinfo.desegment_offset = offset
return
end
tree:add(slicer, tvb(offset, len))
offset = nxtpdu
if nxtpdu == tvb:len() then
return
end
end
end
local tcp_table = DissectorTable.get("tcp.port")
tcp_table:add(2506, slicer)