Reassembling packets in a Lua Wireshark Dissector - lua

I'm trying to write a dissector for the Safari Remote Debug protocol which is based on bplists and have been reasonably successful (current code is here: https://github.com/andydavies/bplist-dissector).
I'm running into difficultly with reassembling packets though.
Normally the protocol sends a packet with 4 bytes containing the length of the next packet, then the packet with the bplist in.
Unfortunately some packets from the iOS simulator don't follow this convention and the four bytes are either tagged onto the front of the bplist packet, or onto the end of the previous bplist packet, or the data is multiple bplists.
I've tried reassembling them using desegment_len and desegment_offset as follows:
function p_bplist.dissector(buf, pkt, root)
-- length of data packet
local dataPacketLength = tonumber(buf(0, 4):uint())
local desiredPacketLength = dataPacketLength + 4
-- if not enough data indicate how much more we need
if desiredPacketLen > buf:len() then
pkt.desegment_len = dataPacketLength
pkt.desegment_offset = 0
return
end
-- have more than needed so set offset for next dissection
if buf:len() > desiredPacketLength then
pkt.desegment_len = DESEGMENT_ONE_MORE_SEGMENT
pkt.desegment_offset = desiredPacketLength
end
-- copy data needed
buffer = buf:range(4, dataPacketLen)
...
What I'm attempting to do here is always force the size bytes to be the first four bytes of a packet to be dissected but it doesn't work I still see a 4 bytes packet, followed by a x byte packet.
I can think of other ways of managing the extra four bytes on the front, but the protocol contains a lookup table thats 32 bytes from the end of the packet so need a way of accurately splicing the packet into bplists.
Here's an example cap: http://www.cloudshark.org/captures/2a826ee6045b #338 is an example of a packet where the bplist size is at the start of the data and there are multiple plists in the data.
Am I doing this right (looking other questions on SO, and examples around the web I seem to be) or is there a better way?

TCP Dissector packet-tcp.c has tcp_dissect_pdus(), which
Loop for dissecting PDUs within a TCP stream; assumes that a PDU
consists of a fixed-length chunk of data that contains enough information
to determine the length of the PDU, followed by rest of the PDU.
There is no such function in lua api, but it is a good example how to do it.
One more example. I used this a year ago for tests:
local slicer = Proto("slicer","Slicer")
function slicer.dissector(tvb, pinfo, tree)
local offset = pinfo.desegment_offset or 0
local len = get_len() -- for tests i used a constant, but can be taken from tvb
while true do
local nxtpdu = offset + len
if nxtpdu > tvb:len() then
pinfo.desegment_len = nxtpdu - tvb:len()
pinfo.desegment_offset = offset
return
end
tree:add(slicer, tvb(offset, len))
offset = nxtpdu
if nxtpdu == tvb:len() then
return
end
end
end
local tcp_table = DissectorTable.get("tcp.port")
tcp_table:add(2506, slicer)

Related

Why is my program reporting more captured packets than Wireshark?

I am writing a packet sniffer using pcap and visual studio. I've taken sample code for offline capturing and combined it with code that looks for an interface and captures packets live. This is what I have to display the packets information gotten from 1.
while (int returnValue = pcap_next_ex(pcap, &header, &data) >= 0)
{
// Print using printf. See printf reference:
// http://www.cplusplus.com/reference/clibrary/cstdio/printf/
// Show the packet number
printf("Packet # %i\n", ++packetCount);
// Show the size in bytes of the packet
printf("Packet size: %d bytes\n", header->len);
// Show a warning if the length captured is different
if (header->len != header->caplen)
printf("Warning! Capture size different than packet size: %ld bytes\n", header->len);
// Show Epoch Time
printf("Epoch Time: %d:%d seconds\n", header->ts.tv_sec, header->ts.tv_usec);
// loop through the packet and print it as hexidecimal representations of octets
// We also have a function that does this similarly below: PrintData()
for (u_int i=0; (i < header->caplen ) ; i++)
{
// Start printing on the next after every 16 octets
if ( (i % 16) == 0) printf("\n");
// Print each octet as hex (x), make sure there is always two characters (.2).
printf("%.2x ", data[i]);
}
// Add two lines between packets
printf("\n\n");
}
The problem I'm having is that if I run a WireShark live capture and also run my program, both capture packets live, but WireShark will show that it's capturing packet 20 and VS will show packetCount = 200.(Note: arbitrary numbers chosen to show Wireshark hasn't captured many packets, but VS is counting extremely fast.)
From what I understand, it seems the while loop is just running much faster than the packets are coming in and so it's just printing information from the last packet it received over and over again until a new one comes in.
How can I get VS to only capture packets as they come in?
I was unaware that Visual Studio included a packet sniffer; did you mean "how can I get my application, which I'm building with Visual Studio, to only capture packets as they come in?"
If that's what you meant, then that's what your code is doing.
Unfortunately, what it's not doing is "checking whether a packet has actually come in".
To quote the man page for pcap_next_ex() (yeah, UN*X, but this applies to Windows and WinPcap as well):
pcap_next_ex() returns 1 if the packet was read without problems,
0 if packets are being read from a live capture, and the timeout
expired, -1 if an error occurred while reading the packet, and -2
if packets are being read from a ``savefile'', and there are no
more packets to read from the savefile. If -1 is returned,
pcap_geterr() or pcap_perror() may be called with p as an argument
to fetch or display the error text.
Emphasis on "0 if packets are being read from a live capture, and the timeout expired"; 0 means that you did not get a packet.
Do not act as if a packet were captured if pcap_next_ex() returned 0, only do so if it returned 1:
while (int returnValue = pcap_next_ex(pcap, &header, &data) >= 0)
{
if (returnValue == 1) {
// Print using printf. See printf reference:
// http://www.cplusplus.com/reference/clibrary/cstdio/printf/
// Show the packet number
printf("Packet # %i\n", ++packetCount);
...
// Add two lines between packets
printf("\n\n");
}
}
SOLUTION: So apparently, adding parentheses around the argument fixes the issue.
while ( (( int returnValue = pcap_next_ex(pcap, &header, &data) ) >= 0 ) )

lua dissector for custom protocol

I have written several Lua Dissectors for custom protocols we use and they work fine. In order to spot problems with missing packets I need to check the custom protocol sequence numbers against older packets.
The IP source and Destination addresses are always the same for device A to device B.
Inside this packet we have one custom ID.
Each ID has a sequence number so device B can determine if a packet is missing. The sequence number increments by 256 and rolls over when it reaches 65k
I have tried using global dictionary but when you scroll up and down the trace the decoder is rerun and the values change.
a couple of lines below show where the information is stored.
ID = buffer(0,6):bitfield(12,12)
SeqNum = buffer(0,6):bitfield(32,16)
Ideally I would like to list in each decoded frame if the previous sequence number is more than 256 away and to produce a table lists all these bad frames.
Src IP; Dst IP; ID; Seq
1 10.12.1.2; 10.12.1.3; 10; 0
2 10.12.1.2; 10.12.1.3; 11; 0
3 10.12.1.2; 10.12.1.3; 12; 0
4 10.12.1.2; 10.12.1.3; 11; 255
5 10.12.1.2; 10.12.1.3; 12; 255
6 10.12.1.2; 10.12.1.3; 10; 511 Packet with seq 255 is missing
I have now managed to get the dissector to check the current packet against previous packets by using a global array, where I store specific information about each frame. In the current packet being dissected I recheck the most recent packet and work my way back to the start to find a suitable packet.
dict[pinfo.number] = {frame = pinfo.number, dID = ID, dSEQNUM = SeqNum}
local frameCount = 0
local frameFound = false
while frameFound == false do
if pinfo.number > frameCount then
frameCount = frameCount + 1
if dict[(pinfo.number - frameCount)] ~= nil then
if dict[(pinfo.number - frameCount)].dID == dict[pinfo.number].dID then
seq_difference = (dict[(pinfo.number)].dSEQNUM - dict[(pinfo.number - frameCount)].dSEQNUM)
if seq_difference > 256 then
pinfo.cols.info = string.format('ID-%d SeqNum-%d missing packet(s) %d last frame %d ', ID,SeqNum, seq_difference, dict[(pinfo.number - frameCount)].frame)
end
frameFound = true
end
end
else
frameFound = true
end
end
I'm not sure I see a question to answer? If you're asking "how can I avoid having to deal with the dissector being invoked multiple times and screwing up the previous decoding of the values" - the answer to that is using the pinfo.visited boolean. It will be false the first time a given packet is dissected, and true thereafter no matter how much clicking around the user does - until the file is reloaded or a new one loaded.
To handle the reloading/new-file case, you'd hook into the init() function call for your proto, by defining a function myproto.init() function, and in that you'd clear your entire array table.
Also, you might want to google for related questions/answer on ask.wireshark.org, as that site is more frequently used for wireshark Lua API questions. For example this question/answer is similar and related to your case.

serial data flow: How to ensure completion

I have a device that sends serial data over a USB to COM port to my program at various speeds and lengths.
Within the data there is a chunk of several thousands bytes that starts and ends with special distinct code ('FDDD' for start, 'FEEE' for end).
Due to the stream's length, occasionally not all data is received in one piece.
What is the recommended way to combine all bytes into one message BEFORE parsing it?
(I took care of the buffer size, but have no control over the serial line quality, and can not use hardware control with USB)
Thanks
One possible way to accomplish this is to have something along these lines:
# variables
# buffer: byte buffer
# buffer_length: maximum number of bytes in the buffer
# new_char: char last read from the UART
# prev_char: second last char read from the UART
# n: index to the buffer
new_char := 0
loop forever:
prev_char := new_char
new_char := receive_from_uart()
# start marker
if prev_char = 0xfd and new_char = 0xdd
# set the index to the beginning of the buffer
n := 0
# end marker
else if prev_char = 0xfe and new_char = 0xee
# the frame is ready, do whatever you need to do with a complete message
# the length of the payload is n-1 bytes
handle_complete_message(buffer, n-1)
# otherwise
else
if n < buffer_length - 1
n := n + 1
buffer[n] := new_char
A few tips/comments:
you do not necessarily need a separate start and end markers (you can the same for both purposes)
if you want to have two-byte markers, it would be easier to have them with the same first byte
you need to make sure the marker combinations do no occur in your data stream
if you use escape codes to avoid the markers in your payload, it is convenient to take care of them in the same code
see HDLC asynchronous framing (simply to encode, simple to decode, takes care of the escaping)
handle_complete_message usually either copies the contents of buffer elsewhere or swaps another buffer instead of buffer if in hurry
if your data frames do not have integrity checking, you should check if the payload length is equal to buffer_length- 1, because then you may have an overflow
After several tests, I came up with the following simple solution to my own question (for c#).
Shown is a minimal simplified solution. Can add length checking, etc.
'Start' and 'End' are string markers of any length.
public void comPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
SerialPort port = (SerialPort)sender;
inData = port.ReadExisting();
{
if (inData.Contains("start"))
{
//Loop to collect all message parts
while (!inData.Contains("end"))
inData += port.ReadExisting();
//Complete by adding the last data chunk
inData += port.ReadExisting();
}
//Use your collected message
diaplaydata(inData);

Reassemble PDUs in lua wireshark dissector

In a system I have a custom protocol and I would like to implement a Wireshark dissector so that I can use Wireshark to analyze the communication.
Objects are sent over the protocol, let us call them "Messages". Each message can be large, maybe up to 100 MB, they can also be very small for example 50 byte.
Each Message is split up in chunks of around 1 kB and tagged with a sequence number and a guid message id and those can be used at the other end to reassemble the messages.
So far I have successfully made a dissector that will individually log all chunks to Wireshark but I want to take this further and also log all messages (chunks assembled into messages) in Wireshark. Can this be done and how? Is it maybe possible to implement a dissector on top of the dissector I have developed below?
If it is possible to implement a dissector on top of the one below, how can I make sure it will only analyze myproto PDUs? The dissector below triggers on a specific tcp port, but that is not going to work for the second phase dissector...
myproto_proto = Proto("myproto", "My Protocol")
function myproto_proto.dissector(buffer, pinfo, tree)
pinfo.cols.protocol = "myproto"
local message_length = buffer(0, 4):le_uint()+4
if message_length>pinfo.len then
pinfo.desegment_len = message_length
pinfo.desegment_offset = 0
return;
end
local subtree = tree:add(myproto_proto, buffer(), "My Protocol Data")
local packet = subtree:add(buffer(0, message_length), "Chunk")
packet:add(buffer(0, 4), "Packet length: " .. buffer(0, 4):le_uint())
packet:add(buffer(32, 16), "Message ID")
packet:add(buffer(48, 4), "Block ID: " .. buffer(48, 4):le_uint())
packet:add(buffer(52, 4), "Max Block ID: " .. buffer(52, 4):le_uint())
packet:add(buffer(68, message_length-68-20), "Data")
pinfo.desegment_len = 0
pinfo.desegment_offset = message_length
return
end
tcp_table = DissectorTable.get("tcp.port")
tcp_table:add(1234, myproto_proto)
Let's say you have created a second dissector msgproto. Since you don't seem to have any multiplexing between chunks and messages, you don't need to setup a dissector table. Instead, at the end of myproto_proto.dissector you do a
msgproto.dissector:call(buffer(68, message_length-68-20):tvb, pinfo, tree)
This will pass all the chunk data to your msgproto . In the message protocol dissector you can use the chunk protocol's fields and of course the tvb that will contain just the data of one chunk. You will now need to piece together the chunks to one gloriously huge tvb. Make the msgprotohave state:
local stateMap = {}
function msgproto.init()
stateMap = {}
end
Convert your the tvb into a ByteArray and store into the stateMap together with the arrays from the other calls to your dissector. When you have assembled all your data into one array, let's call it oarr, make a tvb out of it:
local otvb = ByteArray.tvb(oarr, "message data")
-- starting from 0, need to pass a tvb range, not a tvb.
stree:add(msgproto.fields.payload, otvb(0))
supposed you have payload field of type Protofield.bytes. This code will make a new data pane called "message data" appear next to your regular "Frame" pane at the bottom of your Wireshark window.
I'm not sure how well Wireshark will like your ultra large buffers. Apart from that, I'm quite convinced that the approach outlined above will work. I have not used the described techniques in this exact order, but I have used all the described techniques, for example to make a new byte pane filled with data deflated in pure Lua.

How to manual set minimal value for dynamic buffer in Netty 3.2.6? For example 2048 bytes

I need to receive full packet from other IP (navigation device) by TCP/IP.
The device has to send 966 bytes periodically (over one minute), for example.
In my case first received buffer has length 256 bytes (first piece of packet), the second is 710 bytes (last piece of packet), the third is full packet (966 bytes).
How to manual set minimal value for first received buffer length?
This is piece of my code:
Executor bossExecutors = Executors.newCachedThreadPool();
Executor workerExecutors = Executors.newCachedThreadPool();
NioServerSocketChannelFactory channelsFactory =
new NioServerSocketChannelFactory(bossExecutors, workerExecutors);
ServerBootstrap bootstrap = new ServerBootstrap(channelsFactory);
ChannelPipelineFactory pipelineFactory = new NettyServerPipelineFactory(this.HWController);
bootstrap.setPipelineFactory(pipelineFactory);
bootstrap.setOption("child.tcpNoDelay", true);
bootstrap.setOption("child.keepAlive", true);
bootstrap.setOption("child.receiveBufferSizePredictorFactory",
new FixedReceiveBufferSizePredictorFactory(2048)
);
bootstrap.bind(new InetSocketAddress(this.port));
No matter what receiveBufferSizePredictorFactory you specify, you will see a message is split into multiple MessageEvents. It's because TCP/IP is not a message-oriented protocol but a stream-oriented one. Please read the user guide that explains how to write a proper decoder that deals with this common issue.

Resources