Different wireshark decode result for a same http2 frame? - wireshark
I decode a same pcap file on different terminal both using wireshark. While I have different output.
Stream: HEADERS, Stream ID: 1537, Length 7, 200 OK
Length: 7
Type: HEADERS (1)
Flags: 0x04, End Headers
0... .... .... .... .... .... .... .... = Reserved: 0x0
.000 0000 0000 0000 0000 0110 0000 0001 = Stream Identifier: 1537
[Pad Length: 0]
Header Block Fragment: 88c00f0d023236
[Header Length: 78]
[Header Count: 3]
Header: :status: 200 OK
Header: content-type: application/json
Header: content-length: 26
enter image description here
I check the hex steam and index for second different header are the same. So what causes the different?
HTTP/2 header's are compressed with HPACK. This uses a number of techniques to compress the headers, including referencing to a predefined static table, using a dynamic table, and also using Huffman encoding instead of plain ASCII.
The HPACK static table has 61 commonly used headers and values. It's quicker to say "Use static entry 8", rather than :status: 200.
The dynamic table is an extension of the static table but is built up over the connection. So new headers seen for the first time are added to from index 62. It's a lot quicker to say "See header 62, that I sent you previously", than it is to say "user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36" again, and again, and again
However the dynamic table is shifted along with each new header - that is, the next new header is added with index 62, and the old header in spot 62 now becomes 63. This allows the oldest headers to eventually drop off the table after they reach the maximum size.
All this means it's not unexpected that two header blocks decode to different values, as the dynamic table is constantly moving. In fact it would be rare for two HPACK headers to decode to the exact same thing (implying that either only static look ups were used, or the dynamic header had not changed between the two requests).
To prove this is the issue, let's decode this header block:
88c00f0d023236
We need to look at this in binary as HPACK doesn't always use discreet 8-bit blocks (especially when using Huffman encoding):
10001000110000000000111100001101000000100011001000110110
There's various types of HPACK header fields:
We start with a 1 so we know the first header field is an Indexed Header Field Representation so the next 7 bits give the index value (0001000 or 8) which, as it's less than 62, is in the HPACK static table and looking that up we see it is :status: 200.
Next we have 11000000, which is also an Indexed Header Field Representation (starts with a 1) for index value 1000000 or 64. That is beyond the 61 pre-defined static headers so is in the dynamic table. As discussed above this will likely be different with each request. This is your c0 value and so explains it. In the first request the dynamic table must have content-type: application/json, in position 64, while in the second it must have status: 503 in position 64 (though one thing I don't understand is why the second example has two status codes?).
After that, just to show the next header is what's expected and that I'm not making all this up, we have 00001111.... It begins with 0000 so it's a Literal Header Field Without Indexing (meaning this header should NOT be added to the dynamic table since it's not likely to be needed again). The next 4 digits are 1111 which is index 15 + the next 8 bits (00001101 = 13), so it's header 15 + 13 = 29 which is in the static table as content-length (which is why it makes sense why they didn't add it to the table, since content-length values are likely to be highly variable between requests). Next we look at the value of this header (00000010). This starts with a 0 so we know it's not Huffman encoded and it is length 2. So we look at the next 2 octets which will be in ASCII encoding as not Huffman: 00110010 (or 50 in decimal), which looking up an ASCII table is the number 2, and then 00110110 (or 54 in decimal) which is the number 6 in ASCII. So we end up with content-length: 26, which is indeed what we see.
So both examples start with :status: 200 and end with content-length: 26, neither of which use the dynamic table, so they can be the same.
The thing that's different is the bit in the middle, and in both cases it refers to whatever is in dynamic table position number 64 at that time. And that can be different at different times. Hence different values for the same encoded HPACK header.
Related
Lua: How to gzip a string (gzip, not zlib) in memory?
Given a string, how can I compress it in memory with gzip? I'm using Lua.
It sounds like an easy problem, but there is a huge list of libraries. So far, all that I tried were either dead or I could produce only zlib compressed strings. In my use case, I need gzip compression, as it is expected by the receiver.
As a test, if you dump the compressed string to a file, zcat should be able to decompress it.
I'm using OpenResty, so any Lua library should be fine.
(The only solution that I got working so far is to dump the string in a file, call os.execute("gzip /tmp/example.txt") and read it back. Unfortunately, that is not a practical solution.)
It turns out that zlib is not far away from gzip. The difference is that gzip has an additional header. To get this header, you can use lua-zlib like this: local zlib = require "zlib" -- input: string -- output: string compressed with gzip function compress(str) local level = 5 local windowSize = 15+16 return zlib.deflate(level, windowSize)(str, "finish") end Explanation: The second parameter of deflate is the window size. It makes sure that a gzip header is written. If you omit the parameter, you get a zlib compressed string. level is the gzip compression level (1=worst to 9=best) Here is the documentation of the deflate (source: lua-zlib documentation): function stream = zlib.deflate([ int compression_level ], [ int window_size ]) If no compression_level is provided uses Z_DEFAULT_COMPRESSION (6), compression level is a number from 1-9 where zlib.BEST_SPEED is 1 and zlib.BEST_COMPRESSION is 9. Returns a "stream" function that compresses (or deflates) all strings passed in. Specifically, use it as such: string deflated, bool eof, int bytes_in, int bytes_out = stream(string input [, 'sync' | 'full' | 'finish']) Takes input and deflates and returns a portion of it, optionally forcing a flush. A 'sync' flush will force all pending output to be flushed to the return value and the output is aligned on a byte boundary, so that the decompressor can get all input data available so far. Flushing may degrade compression for some compression algorithms and so it should be used only when necessary. A 'full' flush will flush all output as with 'sync', and the compression state is reset so that decompression can restart from this point if previous compressed data has been damaged or if random access is desired. Using Z_FULL_FLUSH too often can seriously degrade the compression. A 'finish' flush will force all pending output to be processed and results in the stream become unusable. Any future attempts to print anything other than the empty string will result in an error that begins with IllegalState. The eof result is true if 'finish' was specified, otherwise it is false. The bytes_in is how many bytes of input have been passed to stream, and bytes_out is the number of bytes returned in deflated string chunks.
translate virtual address to physical address
The following page table is for a system with 16-bit virtual and physical addresses and with 4,096-byte pages. The reference bit is set to 1 when the page has been referenced. Periodically, a thread zeroes out all values of the reference bit.All numbers are provided in decimal. I want to convert the following virtual addresses (in hexadecimal) to the equivalent physical addresses. Also I want to set the reference bit for the appropriate entry in the page table. • 0xE12C • 0x3A9D • 0xA9D9 • 0x7001 • 0xACA1 I know the answers are but I want to know how can I achieve these answers: 0xE12C → 0x312C 0x3A9D → 0xAA9D 0xA9D9 → 0x59D9 0x7001 → 0xF001 0xACA1 → 0x5CA1 I found and tried This but it did not help me much.
It is given that virtual address is 16 bit long.Hence, there are 2^16 addresses in the virtual address space. Page Size is given to be 4 KB ( there are 4K (4 * (2 ^ 10) )addresses in a page), so the number of pages will be ( 2^16 ) / ( 2 ^ 12 ) = 2 ^ 4. To address each page 4 bits are required. The most significant 4 bits in the virtual address will denote the page number being referred and the remaining 12 bits will be the page offset. One thing to remember is page size (in the virtual address space ) is always same as the frame size in the main memory. Hence the last 12 bits will remain same in the physical address as that of the virtual address. To get the frame address in the main memory just use the first 4 bits. Example: Consider the virtual address 0xACA1 Here A in ACA1 denotes the page number ( 10 ) and corresponding frame no is 5 ( 0101) hence the resulting physical address will be → 0x5CA1.
To translate a virtual address to a physical address (applies ONLY to this homework question), we need to know 2 things: Page Size Number of bits for virtual address In this example: 16-bit system, 4KB page size and physical memory size is 64KB. First of all we need to determine the number of needed bits to act as offset inside page. log2(Page-Size) = log2(4096) = 12 bits for offset Out of the 16 bits for virtual address, 12 are for offset, that means each process has 2^4 = 16 virtual pages. Each entry in page table stores the corresponding frame accommodating the page. For example: Now lets translate! First of all for ease of work lets convert 0xE12C to binary. 0xE12C = (1110 0001 0010 1100) in base 2 1110 = 14 in decimal Entry 14 in P.T => Page frame 3. Lets concatenate it to the 12 offset bits Answer: (0011 0001 0010 1100) = 0x312C Another example: 0x3A9D 0x3A9D = 0011 1010 1001 1101 0011 = 3 PageTable[3] = 10 10 in decimal = 1010 in binary 1010 1010 1001 1101 in binary = 0xAA9D
To help you solve this question, we need to get our details right: 16 bit of virtual address space = 2^16 = 65,536 address space 16 bit of physical address space = 2^16 = 65,536 address space 4096 Byte page size determines the offset, which is Log(4096) / Log (2) = 12 bit. This means, 2^12 for Page size As per #Akash Mahapatra, the offset from virtual address is directly mapped to the offset onto physical address As such, we now have: 2^16 (16bit) for virtual address - 2^12 (12bit) for offset = 4-bit for pages, or rather total number of pages available. I won't repeat the calculation for physical since it's the same numbers. 2^4 (4bit) for pages = 16, which correlates to the number of table entries above! We're getting there... be patient! :) Memory Address 0xE12C in hex notation is also known to be holding 16-bit of address. (Since it's stated in the question.) Let's butcher the address now... We first remove '0x' from the info. We can convert E12C to binary notation like #Tony Tannous, but I am going to apply a little short-cut. I simply use a ratio. Well, the address is notated in 4 characters above, and since 16/4 = 4, I can define the first letter as virtual address, while the other 3 are offset address. With the information, 'E' in hexadecimal format, I need to convert to Decimal = 14. Then I look at your table provided, and I found page frame '3'. Page frame 3 is noted in decimal format, which then need to be converted back to Hexadecimal format... Duh!... which is 3! So, the Physical address mapping of the virtual memory location of 0xE12C can be found at 0x312C on the physical memory. You will then go back to the table, and refer to the reference bit column and put a '1' to the row 14. Apply the same concept for these - 0x3A9D → 0xAA9D 0xA9D9 → 0x59D9 0x7001 → 0xF001 0xACA1 → 0x5CA1 If you notice, the last 3 digits are the same (which determines the offset). And the 1st of the 4-digits are mapped according to the table: table entry 3 -> page frame 10 -> hex notation A table entry A (10) -> page frame 5 -> hex notation 5 table entry 7 -> page frame 15 -> hex notation F table entry A (10) -> page frame 5 -> hex notation 5 Hope this explanation helps you and others like me! :)
CocoaAsyncSocket processing hex values
I'm using CocoaAsyncSocket to process a fixed length header message. My node implementation is as follows var output = "\xA5\xA5" + jsonStr; socket.write(output); I intend to read the header first (\xA5\xA5) and then process the body of the message. The problem that I have is that I've noticed that A5 takes 4 bytes to be represented in iOS. So when I read 2 bytes I receive c2a5. CocoaAsync reading code socket?.readDataToLength(2, withTimeout: -1, tag: 1) What am I doing wrong here ?. I would like to keep a fixed length header on my server side. Any help is much appreciated.
websocket client packet unframe/unmask
I am trying to implement latest websocket spec. However, i am unable to get through the unmasking step post successful handshake. I receive following web socket frame: <<129,254,1,120,37,93,40,60,25,63,71,88,92,125,80,81,73, 51,91,1,2,53,92,72,85,103,7,19,79,60,74,94,64,47,6,83, 87,58,7,76,87,50,92,83,70,50,68,19,77,41,92,76,71,52, 70,88,2,125,90,85,65,96,15,14,20,107,31,14,28,100,27,9, 17,122,8,72,74,96,15,86,68,37,68,18,76,48,15,28,93,48, 68,6,73,60,70,91,24,122,77,82,2,125,80,81,85,45,18,74, 64,47,91,85,74,51,21,27,20,115,24,27,5,37,69,80,75,46, 18,68,72,45,88,1,2,40,90,82,31,37,69,76,85,103,80,94, 74,46,64,27,5,60,75,87,24,122,25,27,5,47,71,73,81,56, 21,27,93,48,88,76,31,57,77,74,11,55,73,68,73,115,65,81, 31,104,26,14,23,122,8,75,68,52,92,1,2,110,24,27,5,53, 71,80,65,96,15,13,2,125,75,83,75,41,77,82,81,96,15,72, 64,37,92,19,93,48,68,7,5,62,64,93,87,46,77,72,24,40,92, 90,8,101,15,28,83,56,90,1,2,108,6,13,21,122,8,82,64,42, 67,89,92,96,15,93,19,56,28,8,65,101,31,94,16,105,28,10, 20,56,30,14,65,56,27,93,71,106,16,11,17,63,25,4,17,57, 73,89,17,59,29,88,29,106,24,27,5,46,65,72,64,54,77,69, 24,122,66,93,93,49,5,12,8,109,15,28,76,59,90,93,72,56, 76,1,2,41,90,73,64,122,8,89,85,50,75,84,24,122,25,15, 23,105,25,5,19,106,26,14,20,111,25,27,5,53,77,85,66,53, 92,1,2,110,26,13,2,125,95,85,65,41,64,1,2,108,27,10,19, 122,7,2>> As per base framing protocol defined here (https://datatracker.ietf.org/doc/html/draft-ietf-hybi-thewebsocketprotocol-17#section-5.2) i have: fin:1, rsv:0, opcode:1, mask:1, length:126 Masked application+payload data comes out to be: <<87,58,7,76,87,50,92,83,70,50,68,19,77,41,92,76,71,52,70,88,2,125,90,85,65,96, 15,14,20,107,31,14,28,100,27,9,17,122,8,72,74,96,15,86,68,37,68,18,76,48,15, 28,93,48,68,6,73,60,70,91,24,122,77,82,2,125,80,81,85,45,18,74,64,47,91,85, 74,51,21,27,20,115,24,27,5,37,69,80,75,46,18,68,72,45,88,1,2,40,90,82,31,37, 69,76,85,103,80,94,74,46,64,27,5,60,75,87,24,122,25,27,5,47,71,73,81,56,21, 27,93,48,88,76,31,57,77,74,11,55,73,68,73,115,65,81,31,104,26,14,23,122,8,75, 68,52,92,1,2,110,24,27,5,53,71,80,65,96,15,13,2,125,75,83,75,41,77,82,81,96, 15,72,64,37,92,19,93,48,68,7,5,62,64,93,87,46,77,72,24,40,92,90,8,101,15,28, 83,56,90,1,2,108,6,13,21,122,8,82,64,42,67,89,92,96,15,93,19,56,28,8,65,101, 31,94,16,105,28,10,20,56,30,14,65,56,27,93,71,106,16,11,17,63,25,4,17,57,73, 89,17,59,29,88,29,106,24,27,5,46,65,72,64,54,77,69,24,122,66,93,93,49,5,12,8, 109,15,28,76,59,90,93,72,56,76,1,2,41,90,73,64,122,8,89,85,50,75,84,24,122, 25,15,23,105,25,5,19,106,26,14,20,111,25,27,5,53,77,85,66,53,92,1,2,110,26, 13,2,125,95,85,65,41,64,1,2,108,27,10,19,122,7,2>> While the 32-bit masking key is: <<37,93,40,60,25,63,71,88,92,125,80,81,73,51,91,1,2,53,92,72,85,103,7,19,79,60, 74,94,64,47,6,83>> As per https://datatracker.ietf.org/doc/html/draft-ietf-hybi-thewebsocketprotocol-17#section-5.2 : j = i MOD 4 transformed-octet-i = original-octet-i XOR masking-key-octet-j however, i doesn't seem to get my original octet sent from client side, which is basically a xml packet. Any direction, correction, suggestions are greatly appreciated.
I think you've mis-read the data framing section of the protocol spec. Your interpretation of the first byte (129) is correct - fin + opcode 1 - final (and first) fragment of a text message. The next byte (254) implies that the body of the message is masked and that the following 2 bytes provide its length (lengths of 126 or 127 imply longer messages whose length's can't be represented in 7 bits. 126 means that the following 2 bytes hold the length; 127 mean that its the following 4 bytes). The following 2 bytes - 1, 120 - imply a message length of 376 bytes. The following 4 bytes - 37,93,40,60 - are your mask. The remaining data is your message which should be transformed as you write, giving the message <body xmlns='http://jabber.org/protocol/httpbind' rid='2167299354' to='jaxl.im' xml:lang='en' xmpp:version='1.0' xmlns:xmpp='urn:xmpp:xbosh' ack='1' route='xmpp:dev.jaxl.im:5222' wait='30' hold='1' content='text/xml; charset=utf-8' ver='1.1 0' newkey='a6e44d87b54461e62de3ab7874b184dae4f5d870' sitekey='jaxl-0-0' iframed='true' epoch='1324196722121' height='321' width='1366'/>
TCP/IP Client / Server commands data
I have a Client/Server architecture (C# .Net 4.0) that send's command packets of data as byte arrays. There is a variable number of parameters in any command, and each paramater is of variable length. Because of this I use delimiters for the end of a parameter and the command as a whole. The operand is always 2 bytes and both types of delimiter are 1 byte. The last parameter_delmiter is redundant as command_delmiter provides the same functionality. The command structure is as follow: FIELD SIZE(BYTES) operand 2 parameter1 x parameter_delmiter 1 parameter2 x parameter_delmiter 1 parameterN x ............. ............. command_delmiter 1 Parameters are sourced from many different types, ie, ints, strings etc all encoded into byte arrays. The problem I have is that sometimes parameters when encoded into byte arrays contain bytes that are the same value as a delimiter. For example command_delmiter=255.. and a paramater may have that byte inside of it. There is 3 ways I can think of fixing this: 1) Encode the parameters differently so that they can never be the same value as a delimiter (255 and 254) Modulus?. This will mean that paramaters will become larger, ie Int16 will be more than 2 bytes etc. 2) Do not use delimiters at all, use count and length values at the start of the command structure. 3) Use something else. To my knowledge, the way TCP/IP buffers work is that SOME SORT of delimiter has to be used to seperate 'commands' or 'bundles of data' as a buffer may contain multiple commands, or a command may span multiple buffers.. So this BinaryReader / Writer seems like an obvious candidate, the only issue is that the byte array may contain multiple commands ( with parameters inside). So the byte array would still have to be chopped up in order to feel into the BinaryReader. Suggestions? Thanks.
The standard way to do this is to have the length of the message in the (fixed) first few bytes of a message. So you could have the first 4 bytes to denote the length of a message, read those many bytes for the content of the message. The next 4 bytes would be the length of the next message. A length of 0 could indicate end of messages. Or you could use a header with a message count. Also, remember TCP is a byte stream, so don't expect a complete message to be available every time you read data from a socket. You could receive an arbitrary number of bytes at ever read.