This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
need help on how to encode words using huffman code
Suppose I have the following Huffman coded symbols
A - 0
B - 10
C - 110
D - 111
and that you want to encode the sequence
A B A A C D A D B B
then I get the binary code like this:
01000110 11101111 010
(01000110) = 0x46
(11101111) = 0xEF
010=????
if there is no 010 in this code, I can save these byte into a file.
now how should I process this 010? Save it as 00000010? that doesn't work.
You'll need some header for your encoded data (a byte should be enough for this purpose, but you may need more, depending on what you actually need) and you can store how many padding bits you have in the last data byte. So in your example with 010 your header byte would contain 5 because you have 5 bits at the end of the last byte that you need to ignore.
Finding what values are stored in the bits that are actually useful is something that you need to handle bit by bit, since you can have overlapping codes where a code may be split between two bytes - so some bits may be at byte N and some bits may overlap onto byte N+1.
You can include an end byte that marks that you've finished, then perhaps another byte after that that says how many items were stored in the last data byte. Then you can sort out the remaining characters from the last byte with knowledge of how many bits are actually significant.
Related
I'm having trouble converting to 16 bit int values from raw data I'm receiving over Ethernet.
For example, I might receive this:
\x00\x0A\x00\x00\x00\x09\x01\x10\x00\x01\x00\x10\x02\x00\x00
I need to take two of these raw data bytes and convert them to a 16 bit unsigned value.
So far I've tried with tonumber() but I can't find a way to make it combine the 2 bytes, I've seen some examples of using string.gsub() on here to do the conversions but these all deal with an an ASCII representation of the raw data.
TIA
Use string.byte() on a single character to turn it into its numerical value, then just multiply the more significant one by 256 (or if you're on Lua 5.3 or newer, shift it left by 8 bits), then add them.
If you're on Lua 5.3 or newer, try also string.unpack. You can select the byte order with < and >:
s="\x00\x0A\x00\x00\x00\x09\x01\x10\x00\x01\x00\x10\x02\x00\x00\x00"
print("<",">")
for i=1,#s,2 do
print((string.unpack("<i2",s,i)),(string.unpack(">i2",s,i)))
end
I'm trying to implement a deflate compressor and I have to decide whether to
compress a block using the static huffman code or create a dynamic one.
What is the rationale behind the length associated with the static code?
(this is the table included in the rfc)
Lit Value Bits
--------- ----
0 - 143 8
144 - 255 9
256 - 279 7
280 - 287 8
I thought static code was more biased towards plain ascii text, instead it
looks like it prefers by a tiny bit the compression of the rle length
What is a good heuristic to decide whether to use static code?
I was thinking to build a distribution of probabilities from a sample of the
input data and calculate a distance (maybe EMD?) from the probabilities derived
from the static code.
I would guess that the creator of the code took a large sample of literals and lengths from compressed data, likely including executables along with text, and found typical code lengths over the large set. They were then approximated with the table shown. However the author passed away many years ago, so we'll never know for sure.
You don't need a heuristic. Once you have done the work to find matching strings, it is comparatively very fast to compute the number of bits in the block for both a dynamic and static representation. Then simply pick the smaller one. Or the static one if equal (decodes faster).
I don't know about rationale, but there was a small amount of irrationale in choosing the static code lengths:
In the table in your question, the maximum static code number there is 287, but the DEFLATE specification only allows up to code 285, meaning code lengths have wastefully been assigned to two invalid codes. (And not even the longest ones either!) It's a similar story with the table for distance codes, with 32 codes having lengths assigned, but only 30 valid.
So there are some easy improvements that could have been made, but that said, without some prior knowledge of the data, it's not really possible to produce anything that's massively more efficient generally. The "flatness" of the table (no code longer than 9 bits) reduces the worst-case performance to 1 extra bit per byte of uncompressable data.
I think the main rationale behind the groupings is that by keeping group sizes to a multiple of 8, it's possible to tell which group a code belongs to by looking at the 5 most significant bits, which also tells you its length, along with what value to add to immediately get the code value itself
00000 00 .. 00101 11 7 bits + 256 -> (256..279)
00110 000 .. 10111 111 8 bits - 48 -> ( 0..144)
11000 000 .. 11000 111 8 bits + 78 -> (280..287)
11001 0000 .. 11111 1111 9 bits - 256 -> (144..255)
So in theory you could set up a lookup table with 32 entries to quickly read in the codes, but it's an uncommon case and probably not worth optimising for.
There are only really two cases (with some overlap) where Fixed Huffman blocks are likely to be the most efficient:
where the input size in bytes is very small, Static Huffman can be more efficient than Uncompressed, because Uncompressed uses a 32-bit header, while Fixed Huffman needs only a 7-bit footer, plus 1 bit potential overhead per byte.
where the output size is very small (ie. small-ish, highly compressible data), Static Huffman can be more efficient than Dynamic Huffman - again because Dynamic Huffman uses a certain amount of space for an additional header. (A practical minimum header size is difficult to calculate, but I'd say at least 64 bits, probably more.)
That said, I've found they are actually helpful from a developer's perspective, because it's very easy to implement a Deflate-compatible function using Static Huffman blocks, and to iteratively improve from there to get more efficient algorithms working.
I was reading BMP file in hex editor while discovered something odd. Two first letters "BM" are written in order, however the next word(2B), which is means file size, is 36 30 in hex. Actual size is 0x3036. I've noticed that other numbers are stored the same way.
I'm also using MARS MIPS emulator which can display memory by words. String in.bmp is stored as b . n i / \0 p m.
Why data isn't stored continuously?
It depends not on the data itself but on how you store this data: per byte, per word (2 bytes, usually), or per long (4 bytes -- again, usually). As long as you store data per byte you don't see anything unusual; data appears "continuous". However, with longer units, you are subject to endianness.
It appears your emulator is assuming all words need to have their bytes reversed; and you can see in your example that this assumption is not always valid.
As for the BM "magic" signature: it's not meant to be read as a word value "BM", but rather as "first, a single byte B, then a single byte M". All next values are written in little-endian order, not only 'exchanging' your 36 and 30 but also the 2 zeroes 'before' (or 'after') (the larger values in the BMP header are of 4 bytes long type).
I'm parsing an Jpeg/JFIF file and I noticed that after the SOI (0xFF D8) I parse the different "streams" starting with 0xFFXX (where XX is a hexadecimal number) until I find the EOI (0XFFD9). Now the structure of the diffrent chunks is:
APP0 marker 2 Bytes
Length 2 Bytes
Now when I parse the a chunk I parse until i reach the length written in the 2 Bytes of the length field. After that I thought I would immediately find another Marker, followed by a length for the next chunk. According to my parser that is not always true, there might be data between the chunks. I couldn't find out what that data is, and if it is relevant to the image. Do you have any hints what this could be and how to interpret those bytes?
I'm lost and would be happy if somebody could point me in the correct direction. Thanks in advance
I've recently noticed this too. In my case it's an APP2 chunk which is the ICC profile which doesn't contain the length of the chunk.
In fact so far as I can see the length of the chunk needn't be the first 2 bytes (though it usually is).
In JFIF all 0xFF bytes are replaced with 0xFF 0x00 in the data section, so it should just be a matter of calculating the length from that. I just read until I hit another header, however I've noticed that sometimes (again in the ICC profile) there are byte sequences which don't make sense such as 0xFF 0x6D, so I may still be missing something.
If something is stored at 0x1001 0000 the next thing is stored at 0x1001 0004. And if I'm correct the memory pieces in a 32-bit architecture are 32 bits each. So would 0x1001 0002 point to the second half of the 32 bits?
First of all, memory addresses in MIPS architecture are not incremented by 4. MIPS uses byte addressing, so you can address any byte from memory (see e.g. lb and lbu to read a single byte, lh and lhu to read a half-word).
The fact is that if you read words which are 32 bits length (4 bytes, lw), then two consecutive words will be 4 bytes away from each other. In this case, you would add 4 to the address of the first word to get the address of the next word.
Beside this, if you read words you have to align them in multiples of 4, otherwise you will get an alignment exception.
In your example, if the first word is stored in 0x10010000 then the next word will be in 0x10010004 and of course the first half/second half would be in 0x1001000 and 0x1001002 (the ordering will depend on the endianness of the architecture).
You seem to have answered this one yourself! 32 bits make 4 bytes, so if you're e.g. pushing to a stack, where all elements are pushed as the same size, each next item will be 4 bytes ahead (or before) the next.