How to manual set minimal value for dynamic buffer in Netty 3.2.6? For example 2048 bytes - buffer
I need to receive full packet from other IP (navigation device) by TCP/IP.
The device has to send 966 bytes periodically (over one minute), for example.
In my case first received buffer has length 256 bytes (first piece of packet), the second is 710 bytes (last piece of packet), the third is full packet (966 bytes).
How to manual set minimal value for first received buffer length?
This is piece of my code:
Executor bossExecutors = Executors.newCachedThreadPool();
Executor workerExecutors = Executors.newCachedThreadPool();
NioServerSocketChannelFactory channelsFactory =
new NioServerSocketChannelFactory(bossExecutors, workerExecutors);
ServerBootstrap bootstrap = new ServerBootstrap(channelsFactory);
ChannelPipelineFactory pipelineFactory = new NettyServerPipelineFactory(this.HWController);
bootstrap.setPipelineFactory(pipelineFactory);
bootstrap.setOption("child.tcpNoDelay", true);
bootstrap.setOption("child.keepAlive", true);
bootstrap.setOption("child.receiveBufferSizePredictorFactory",
new FixedReceiveBufferSizePredictorFactory(2048)
);
bootstrap.bind(new InetSocketAddress(this.port));
No matter what receiveBufferSizePredictorFactory you specify, you will see a message is split into multiple MessageEvents. It's because TCP/IP is not a message-oriented protocol but a stream-oriented one. Please read the user guide that explains how to write a proper decoder that deals with this common issue.
Related
Problem using mosquitto broker with netcat
I trying to use mosquitto broker for an IoT application. I have a embedded hardware, actually not much documented/exampled on the internet. I've succesfully implemented an TCP client on this hardware, and now i can send/listen messages throught any port i want via TCP and i listen via netcat. But when i tried to connect mosquitto, it doesn't accept the literal language. I digged on the internet. The broker take messages like shown below, but even this one not working. I can not found any documentation. I even tried to watch Wireshark packages, and i can not find any pattern. Any help will appreciated. $ echo -en "\x10\x0d\x00\x04MQTT\x04\x00\x00\x00\x00\x01a" |nc localhost 1883|hd 00000000 20 02 01 00 | ...| 00000004
Had a similar usecase like you and this is how I managed to decode this message and create my own connect request. echo -en "\x10\x0d\x00\x04MQTT\x04\x00\x00\x00\x00\x01a" |nc localhost 1883 The above message is broken down as follows: nc localhost 1883, opens a tcp socket to the mqtt port 1883 (to the broker (on localhost) listening to port 1883) \x10\x0d\x00\x04MQTT\x04\x00\x00\x00\x00\x01a is the connect packet sent to the socket that was opened. This connect packet can be broken down as follows: \x10: MQTT connect packet (Constitutes the control field with the 1st 4 bits representing the command type "0001" and the 2nd 4 bits the control flag. \x0d: Remaining length: is the total length of both the variable header and the payload. This needs to be set after the whole payload is complete. *Variable header = Protocol Name + Protocol level + Connect Flag byte + Keep Alive Payload = ClientId, username, password, e.t.c, each entry is provided in the format lengthOfEntry + Entry e.g for a clientId and username and password, this gives: lengthOfClientId + ClientId + lengthOfUsername + Username + lengthOfPassword + Password. *Note: the length is always provided as two bytes. \x00\x04MQTT: Represents the protocol name "MQTT". The first two bytes 0x00 and 0x04 are the protocol length and MQTT is then the protocol. These give a total of 6 bytes. \x04: is the protocol level. From MQTT specification, the value of the Protocol Level field for the version 3.1.1 of the protocol is 4 (0x04) \0x00: is the connect flag byte. each byte is represented as follows: UsernameFlag|PasswordFlag|Will Retain|Will|QoS|Will Flag|CleanSession|Reserved. 0x00 means none of the flags are set and persistent sessions will be used. Hence the payload doesn't require a username or password as will be seen in the payload bytes. \0x00\0x00: two bytes that represent the keepAlive time. In this case 0 is provided which means that the server is not required to disconnect the client on grounds of inactivity. keepalive mechanism is turned off The next bytes are already the payload. Since the connect flag was set such that no username or password is required, then we only have to provide the client id. In the above example the clientid is "a". This is of length 1 byte. \0x00\0x01: represents the length of the clientId. since we have the character "a" as the clientId we have just the length 1. The length is always given in 16bit (2 bytes). a: the clientId. If we count all the bytes, we come up to a total of 13 bytes which gives us our remaining le0gnth as 0x0d. If you need to add a username and password, set the connect flag accordingly 0b11000000 = 0xC0. The username and password are added to the payload right after the clientId in the order lengthofClientId, clientId, lengthOfUSername, Username, LengthofPassword, Password. The reminaining length needs to be adjusted to reflect this. Tip: The variable header has a fixed number of bytes of 10. The payload length always depends on the data provided. the length of each element is always 2 bytes. so for the above case, we have a clientid of 1 byte and the length which take 2 bytes to give a total of 10 + 2 + 1 = 13. I hope this helps you out and answers your question. Useful links: https://openlabpro.com/guide/mqtt-packet-format/ http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718024
How to receive CAN data from 29 bit CAN id using TP layer in CAPL
My requirement is to receive/send more than 8 bytes from 29 bit CAN id. I'm using TP layer for this. Is there any library functions for receiving the can data which is from 29 bit id? Ex: 0x1CDA00FE is tester and 0x1CDAFE00 is server.
non CAN-FD protocol on Data Link Layer does not support payload greater than 8 bytes. Payload in TP greater than 8 bytes is handled as consecutive frames, and usually are incorporated in a packaging protocol, like UDS.
If this is a diagnostic message ID, and you have diag layer (UDS/KWP) configured on this message, there is a complete diag library in CAPL to send/receive diag requests on the configured Diag Layer.
But to do this "by hand" (I do not know what protocol uses your message IDs to send consecutive frames, like in UDS) :
Receiving
message 0x1CDA00FE msgContainer; /*Create/ define an arbitrary instance of CAN message element*/
Here is how you capture the first iteration of the consecutive frames sent (all of them will be 8 bytes, but you will wait for them as individual 8 byte messages until the intended TP layer payload is completely sent to you). So you basically repeat this snippet 6 times (if your TP payload is 6*8=48bytes long), and do with the msgContainer (message with your payload) whatever you want, until the next one comes.
testwaitformessage(0x1CDA00FE,1000); //wait for the message ID to arrive in 1s
testGetWaitEventMsgData(msgContainer); /*Capture the message content from tail to toe in the msgContainer*/
write("The Byte(0) selector of the captured message for example selects the first byte of the 8 byte payload: 0x%X",msgContainer.byte(0)); // see message selectors for more options
Sending:
byte qword QPayload=0x11223344x55667788; /*You can do with byte array also, I don't fancy to write fors now*/
message 0x1CDAFE00 messagetoSend;
messagetoSend.qword(0)=QPayload;
output(messagetoSend);
/*Prepare another payload to send as the next consecutive frame*/
...
Twilio queue overflow error: how large is the queue?
Twilio's Message resource has a "status" property that indicates whether a SMS message is "queued", "sending", "failed", etc. If a Message instance failed to deliver, one possible error message is "Queue overflow". In the Twilio documentation, the description for this error case is: "You tried to send too many messages too quickly and your message queue overflowed. Try sending your message again after waiting some time." Is the queue referenced in error code 30001 an instance of this resource? https://www.twilio.com/docs/api/rest/queue Or is the queue (in the case of a 30001 error code) something that Twilio maintains on their end? If Twilio does throttling behind the scenes (queueing SMS messages per sending phone number), what is the size of that queue? How much would we have to exceed the rate limit (per phone number) before the queue overflow referenced in error code 30001 occurs?
Emily, message queue is not related to the queue resource you linked to above and it is something maintained on our end. Twilio can queue up to 4 hours of SMS. This means, we can send out 1 sms per second, if there are more than 14,400 messages in the queue, all the messages queued up after will fail with 30001 error queue overflow and will not be sent. This info is for Long Code numbers. The link above explains processing for other scenarios. A few suggestions to avoid the error: Keep messages to at most 160 characters if possible. But if not possible, calculate how many SMS messages each message will be (if you are not sure you can always send 1 test message and see how much you are charged for that message). Based on the assumption that your messages is 160 characters, throttle the sending rate to 3600 messages per hour (1 messaage/sec * 60 sec/min * 60 min/hr). Please let me know if you've got any other questions.
Each of the Twilio phone numbers(senders) has a separate queue in which 14400(4 hr x 60 min x 60 sec) message segments can be queued. 1 segment is sent in one second. What is a message segment? A message segment is not a complete message but a part of a message. Normally SMS is sent in terms of message segments and all message segments are combined on the user’s mobile to create the actual SMS. Twillio message segment config: 1 character = 8 bits(1 byte) GSM encoding = 7 bit per character UCS-2 encoding = 16 bit per character Data Header = 6 bytes per segment Summary: Each character takes 8 bits, If GSM encoding is used, each character will take 7 bits or if UCS-2 encoding is used, each char will take 16 bits. In the case of multiple segments, 6 bytes per segment will be used for data headers(responsible for combining all segments of the same SMS on user mobile) Character per Message Segment: GSM encoding when single segment = (140 char bytes x 8 bits)/ 7 bits = 160 characters UCS-2 encoding when single segment = (140 char bytes x 8 bits)/ 16 bits = 70 characters GSM encoding when multiple segment = ((140 char bytes - 6 header bytes) x 8 bits)/ 7 bits = 154 characters UCS-2 encoding when multiple segment = ((140 char bytes - 6 header bytes) x 8 bits)/ 16 bits = 67 characters Based on what encoding is used(check via Twilio Admin) for your message, you can calculate how many SMS can be in the queue at a time. References: https://support.twilio.com/hc/en-us/articles/115002943027-Understanding-Twilio-Rate-Limits-and-Message-Queues https://www.twilio.com/blog/2017/03/what-the-heck-is-a-segment.html
Reassembling packets in a Lua Wireshark Dissector
I'm trying to write a dissector for the Safari Remote Debug protocol which is based on bplists and have been reasonably successful (current code is here: https://github.com/andydavies/bplist-dissector). I'm running into difficultly with reassembling packets though. Normally the protocol sends a packet with 4 bytes containing the length of the next packet, then the packet with the bplist in. Unfortunately some packets from the iOS simulator don't follow this convention and the four bytes are either tagged onto the front of the bplist packet, or onto the end of the previous bplist packet, or the data is multiple bplists. I've tried reassembling them using desegment_len and desegment_offset as follows: function p_bplist.dissector(buf, pkt, root) -- length of data packet local dataPacketLength = tonumber(buf(0, 4):uint()) local desiredPacketLength = dataPacketLength + 4 -- if not enough data indicate how much more we need if desiredPacketLen > buf:len() then pkt.desegment_len = dataPacketLength pkt.desegment_offset = 0 return end -- have more than needed so set offset for next dissection if buf:len() > desiredPacketLength then pkt.desegment_len = DESEGMENT_ONE_MORE_SEGMENT pkt.desegment_offset = desiredPacketLength end -- copy data needed buffer = buf:range(4, dataPacketLen) ... What I'm attempting to do here is always force the size bytes to be the first four bytes of a packet to be dissected but it doesn't work I still see a 4 bytes packet, followed by a x byte packet. I can think of other ways of managing the extra four bytes on the front, but the protocol contains a lookup table thats 32 bytes from the end of the packet so need a way of accurately splicing the packet into bplists. Here's an example cap: http://www.cloudshark.org/captures/2a826ee6045b #338 is an example of a packet where the bplist size is at the start of the data and there are multiple plists in the data. Am I doing this right (looking other questions on SO, and examples around the web I seem to be) or is there a better way?
TCP Dissector packet-tcp.c has tcp_dissect_pdus(), which
Loop for dissecting PDUs within a TCP stream; assumes that a PDU
consists of a fixed-length chunk of data that contains enough information
to determine the length of the PDU, followed by rest of the PDU.
There is no such function in lua api, but it is a good example how to do it.
One more example. I used this a year ago for tests:
local slicer = Proto("slicer","Slicer")
function slicer.dissector(tvb, pinfo, tree)
local offset = pinfo.desegment_offset or 0
local len = get_len() -- for tests i used a constant, but can be taken from tvb
while true do
local nxtpdu = offset + len
if nxtpdu > tvb:len() then
pinfo.desegment_len = nxtpdu - tvb:len()
pinfo.desegment_offset = offset
return
end
tree:add(slicer, tvb(offset, len))
offset = nxtpdu
if nxtpdu == tvb:len() then
return
end
end
end
local tcp_table = DissectorTable.get("tcp.port")
tcp_table:add(2506, slicer)
websocket client packet unframe/unmask
I am trying to implement latest websocket spec. However, i am unable to get through the unmasking step post successful handshake. I receive following web socket frame: <<129,254,1,120,37,93,40,60,25,63,71,88,92,125,80,81,73, 51,91,1,2,53,92,72,85,103,7,19,79,60,74,94,64,47,6,83, 87,58,7,76,87,50,92,83,70,50,68,19,77,41,92,76,71,52, 70,88,2,125,90,85,65,96,15,14,20,107,31,14,28,100,27,9, 17,122,8,72,74,96,15,86,68,37,68,18,76,48,15,28,93,48, 68,6,73,60,70,91,24,122,77,82,2,125,80,81,85,45,18,74, 64,47,91,85,74,51,21,27,20,115,24,27,5,37,69,80,75,46, 18,68,72,45,88,1,2,40,90,82,31,37,69,76,85,103,80,94, 74,46,64,27,5,60,75,87,24,122,25,27,5,47,71,73,81,56, 21,27,93,48,88,76,31,57,77,74,11,55,73,68,73,115,65,81, 31,104,26,14,23,122,8,75,68,52,92,1,2,110,24,27,5,53, 71,80,65,96,15,13,2,125,75,83,75,41,77,82,81,96,15,72, 64,37,92,19,93,48,68,7,5,62,64,93,87,46,77,72,24,40,92, 90,8,101,15,28,83,56,90,1,2,108,6,13,21,122,8,82,64,42, 67,89,92,96,15,93,19,56,28,8,65,101,31,94,16,105,28,10, 20,56,30,14,65,56,27,93,71,106,16,11,17,63,25,4,17,57, 73,89,17,59,29,88,29,106,24,27,5,46,65,72,64,54,77,69, 24,122,66,93,93,49,5,12,8,109,15,28,76,59,90,93,72,56, 76,1,2,41,90,73,64,122,8,89,85,50,75,84,24,122,25,15, 23,105,25,5,19,106,26,14,20,111,25,27,5,53,77,85,66,53, 92,1,2,110,26,13,2,125,95,85,65,41,64,1,2,108,27,10,19, 122,7,2>> As per base framing protocol defined here (https://datatracker.ietf.org/doc/html/draft-ietf-hybi-thewebsocketprotocol-17#section-5.2) i have: fin:1, rsv:0, opcode:1, mask:1, length:126 Masked application+payload data comes out to be: <<87,58,7,76,87,50,92,83,70,50,68,19,77,41,92,76,71,52,70,88,2,125,90,85,65,96, 15,14,20,107,31,14,28,100,27,9,17,122,8,72,74,96,15,86,68,37,68,18,76,48,15, 28,93,48,68,6,73,60,70,91,24,122,77,82,2,125,80,81,85,45,18,74,64,47,91,85, 74,51,21,27,20,115,24,27,5,37,69,80,75,46,18,68,72,45,88,1,2,40,90,82,31,37, 69,76,85,103,80,94,74,46,64,27,5,60,75,87,24,122,25,27,5,47,71,73,81,56,21, 27,93,48,88,76,31,57,77,74,11,55,73,68,73,115,65,81,31,104,26,14,23,122,8,75, 68,52,92,1,2,110,24,27,5,53,71,80,65,96,15,13,2,125,75,83,75,41,77,82,81,96, 15,72,64,37,92,19,93,48,68,7,5,62,64,93,87,46,77,72,24,40,92,90,8,101,15,28, 83,56,90,1,2,108,6,13,21,122,8,82,64,42,67,89,92,96,15,93,19,56,28,8,65,101, 31,94,16,105,28,10,20,56,30,14,65,56,27,93,71,106,16,11,17,63,25,4,17,57,73, 89,17,59,29,88,29,106,24,27,5,46,65,72,64,54,77,69,24,122,66,93,93,49,5,12,8, 109,15,28,76,59,90,93,72,56,76,1,2,41,90,73,64,122,8,89,85,50,75,84,24,122, 25,15,23,105,25,5,19,106,26,14,20,111,25,27,5,53,77,85,66,53,92,1,2,110,26, 13,2,125,95,85,65,41,64,1,2,108,27,10,19,122,7,2>> While the 32-bit masking key is: <<37,93,40,60,25,63,71,88,92,125,80,81,73,51,91,1,2,53,92,72,85,103,7,19,79,60, 74,94,64,47,6,83>> As per https://datatracker.ietf.org/doc/html/draft-ietf-hybi-thewebsocketprotocol-17#section-5.2 : j = i MOD 4 transformed-octet-i = original-octet-i XOR masking-key-octet-j however, i doesn't seem to get my original octet sent from client side, which is basically a xml packet. Any direction, correction, suggestions are greatly appreciated.
I think you've mis-read the data framing section of the protocol spec. Your interpretation of the first byte (129) is correct - fin + opcode 1 - final (and first) fragment of a text message. The next byte (254) implies that the body of the message is masked and that the following 2 bytes provide its length (lengths of 126 or 127 imply longer messages whose length's can't be represented in 7 bits. 126 means that the following 2 bytes hold the length; 127 mean that its the following 4 bytes). The following 2 bytes - 1, 120 - imply a message length of 376 bytes. The following 4 bytes - 37,93,40,60 - are your mask. The remaining data is your message which should be transformed as you write, giving the message <body xmlns='http://jabber.org/protocol/httpbind' rid='2167299354' to='jaxl.im' xml:lang='en' xmpp:version='1.0' xmlns:xmpp='urn:xmpp:xbosh' ack='1' route='xmpp:dev.jaxl.im:5222' wait='30' hold='1' content='text/xml; charset=utf-8' ver='1.1 0' newkey='a6e44d87b54461e62de3ab7874b184dae4f5d870' sitekey='jaxl-0-0' iframed='true' epoch='1324196722121' height='321' width='1366'/>