I managed extracted a single sector from a cd-rom bin file using a hex editor.
According to cd-rom specifications the sector should have the following format:
* 16 bytes for synchronization and header
* 8 bytes for the sub-header
* 2048 bytes for user data
* 4 bytes for the EDC which happens to be a CRC32 applied to user data and sub-header.
I found detailed information on this matter on page 55 of the following document: https://ia800408.us.archive.org/4/items/cdi_may94_r2/cdi_may94_r2.pdf
I want to manually modify user data in the sector, but in order to do that I must recalculate the EDC as well. (I assume ECC is ignored if EDC results are positive)
However, before modifying anything I want to check that I agree with my sector on the given CRC32. But, I can't. And that's the problem.
Here is my 2352 byte sector:
https://drive.google.com/open?id=1CYAInG8TYMyRMOgR1zeooUrLK5CR7cmM
The EDC section has the following CRC32 result: 92 54 48 44 (hex)
But when I recalculate the CRC32 myself using HxD hex-editor I get: 55 32 CA 62
Why am I getting a different result?
I tried changing input/output reflection in the custom CRC32 section, but nothing works. I've been stuck on this for an hour and couldn't find anything on google.
I'm going to answer my own question since I finally found the answer:
The CRC32 used a different polynomial.
The typical forward/reverse values are: 0x04C11DB7 / 0xEDB88320
The forward/reverse values my cd-rom used: 0x8001801B / 0xD8018001
So if you wanted to recalculate the CRC32 manually in HxD hex-editor you would
need to use a manual CRC with the following settings:
*Polynomial: 0x8001801B
*Initial value: 0
*Output XOR: 0
*Input & Output reflections checked.
Though CDmage can do all this automatically.
In the end all of this didn't help me because the emulator didn't give a flying goldfish over the EDC/ECC bytes. I should have known that.
After slightly changing a few text related bytes in my sector I noticed that the disk image couldn't load anymore, and thus, immediately assumed that EDC/ECC bytes were responsible. The real cause probably lies in the software somewhere.
Related
I'm trying to implement a deflate compressor and I have to decide whether to
compress a block using the static huffman code or create a dynamic one.
What is the rationale behind the length associated with the static code?
(this is the table included in the rfc)
Lit Value Bits
--------- ----
0 - 143 8
144 - 255 9
256 - 279 7
280 - 287 8
I thought static code was more biased towards plain ascii text, instead it
looks like it prefers by a tiny bit the compression of the rle length
What is a good heuristic to decide whether to use static code?
I was thinking to build a distribution of probabilities from a sample of the
input data and calculate a distance (maybe EMD?) from the probabilities derived
from the static code.
I would guess that the creator of the code took a large sample of literals and lengths from compressed data, likely including executables along with text, and found typical code lengths over the large set. They were then approximated with the table shown. However the author passed away many years ago, so we'll never know for sure.
You don't need a heuristic. Once you have done the work to find matching strings, it is comparatively very fast to compute the number of bits in the block for both a dynamic and static representation. Then simply pick the smaller one. Or the static one if equal (decodes faster).
I don't know about rationale, but there was a small amount of irrationale in choosing the static code lengths:
In the table in your question, the maximum static code number there is 287, but the DEFLATE specification only allows up to code 285, meaning code lengths have wastefully been assigned to two invalid codes. (And not even the longest ones either!) It's a similar story with the table for distance codes, with 32 codes having lengths assigned, but only 30 valid.
So there are some easy improvements that could have been made, but that said, without some prior knowledge of the data, it's not really possible to produce anything that's massively more efficient generally. The "flatness" of the table (no code longer than 9 bits) reduces the worst-case performance to 1 extra bit per byte of uncompressable data.
I think the main rationale behind the groupings is that by keeping group sizes to a multiple of 8, it's possible to tell which group a code belongs to by looking at the 5 most significant bits, which also tells you its length, along with what value to add to immediately get the code value itself
00000 00 .. 00101 11 7 bits + 256 -> (256..279)
00110 000 .. 10111 111 8 bits - 48 -> ( 0..144)
11000 000 .. 11000 111 8 bits + 78 -> (280..287)
11001 0000 .. 11111 1111 9 bits - 256 -> (144..255)
So in theory you could set up a lookup table with 32 entries to quickly read in the codes, but it's an uncommon case and probably not worth optimising for.
There are only really two cases (with some overlap) where Fixed Huffman blocks are likely to be the most efficient:
where the input size in bytes is very small, Static Huffman can be more efficient than Uncompressed, because Uncompressed uses a 32-bit header, while Fixed Huffman needs only a 7-bit footer, plus 1 bit potential overhead per byte.
where the output size is very small (ie. small-ish, highly compressible data), Static Huffman can be more efficient than Dynamic Huffman - again because Dynamic Huffman uses a certain amount of space for an additional header. (A practical minimum header size is difficult to calculate, but I'd say at least 64 bits, probably more.)
That said, I've found they are actually helpful from a developer's perspective, because it's very easy to implement a Deflate-compatible function using Static Huffman blocks, and to iteratively improve from there to get more efficient algorithms working.
I have one CAN standard 2.0A frame which contain 8 Bytes of DATA.
e.g
CAN Frame Data "00 CA 22 FF 55 66 AA DF" (8 Bytes)
Now I want to check how many stuff bits would be add in this CAN frame(bit stuffing). standers formula to calculate the Worst case bit stuffing scenario is as following:
64+47+[(34+64-1)/4] ->64 :: Data bits and 47 :: overhead bits 2.0A
How to calculate real stuffed bits in this sample CAN message ??
Any comment, suggestion would be warmly welcome.
There is no way to mathematically "calculate" the stuffed bits. You need to construct the frame (on bit level), traverse the bits, and count.
You could read more about bit stuffin at the link below.
https://en.wikipedia.org/wiki/CAN_bus#Bit_stuffing
Basic principle:
1. Contruct the can frame on bit level
2. Start at frame start bit. When 5 consecutive bits of same polarity is found than insert a bit of opposite polarity.
3. Continue to CRC delimeter (CRC delimeter is excluded)
OK, this question sounds simple but I am taken by surprise. In the ancient days when 1 Megabyte was a huge amount of memory, Intel was trying to figure out how to use 16 bits to access 1 Megabyte of memory. They came up with the idea of using segment and offset address values to generate a 20 bit address.
Now, 20 bits gives 2^20 = 1,048,576 locations that can be addressed. Now assuming that we access 1 byte per address location we get 1,048,576/(1024*1024) = 2^20/2^20 Megabytes = 1 Megabyte. Ok understood.
The confusion comes here, we have 16 bit data bus in the ancient 8086 and can access 2 bytes at a time rather than 1, this equate 20 bit address to being able to access a total of 2 Megabyte of data right? Why do we assume that each address only has 1 byte stored in it when the data bus is 2 bytes wide? I am confused here.
It is very important to consider the bus when trying to understand this. This is probably more of an electrical question than a software one, but here is the answer:
For 8086, when reading from ROM, The least significant address line (A0) is not used, reducing the number of address lines to 19 right then and there.
In the case where the CPU needs to read 16 bits from an odd address, say, bytes at 0x3 and 0x4, it will actually do two 16-bit reads: One from 0x2 and one from 0x4, and discard bytes 0x2 and 0x5.
For 8-bit ROM reads, the read on the bus is still 16-bits but the unneeded byte is discarded.
But for RAM there is sometimes a need to write just a single byte, this gets a little more complex. There is an extra output signal on the processor called BHE# (Bus high enable). The combination of A0 and BHE# are used to determine if the write is an 8 or 16-bits wide, and whether or not it is at an odd or even address.
Understanding these two signals is key to answering your question. Stating it simply as possible:
8-bit even access: A0 OFF, BHE# OFF
8-bit odd access: A0 ON, BHE# ON
16-bit access (must be even): A0 OFF, BHE# ON
And we cannot have a bus cycle with A0 ON and BHE# OFF because an odd access to the even byte of the bus is meaningless.
Relating this back to your original understanding: You are completely correct in the case of memory devices. A 1 megabyte 16-bit memory chip will indeed only have 19 address lines, to that chip, 16 bits is a byte, and in effect, they do not physically have an A0 address input.
... almost. 16-bit writable memory devices have two extra signals (BHE# and BLE#) which are connected to the CPU's BHE# and A0 respectively. This so they know to ignore part of the bus when an 8-bit access is under way, making them hybrid 8/16 bit devices. ROM chips do not have these signals.
For the hardware unenlightened, this is a fairly complex area we're touching on here, and it does get very complex indeed in terms of performance considerations and in large systems with mixed 8 and 16 bit hardware.
It's is all explained in fantastic detail in the 8086 datasheet
It's because a byte is the 'atom' in memory addressing and the code must be able to access all the individual bytes in the address space. really a matter of software and compatibility with 8-bit existing software back then.
This too may interest you: How a single byte of memory is accessed by CPU in a 32-bit memory and 32-bit processor
I am new to CUDA and currently optimize an existing application for molecular dynamics. What it does is that it takes array of double4 with coordinates and computes forces based on the neighborlist. I wrote a kernel with the following lines:
double4 mPos=d_arr_xyz[gid];
while(-1!=(id=d_neib_list[gid*MAX_NEIGHBORS+i])){
Calc(gid,mPos,AA,d_arr_xyz,id);i++;
}
then Calc takes d_arr_xyz[id] and calculates force. That gives 1 read of double4 + 65 reads of (int +double4) inside every call of Calc (65 is average number of neighbors (not equal to -1) in d_neib_list for each particle).
Is it possible to reduce those reads? Neighborlists for different particles, i.e. d_arr_xyz[gid] and d_arr_xyz[id] do not correalte, so I cannot use shared memory for the block of threads to cache d_arr_xyz.
What I see is that if somehow to load the whole list int*MAX_NEIGHBORS into shared memory in one or few large transactions, that will remove 65 separate reads of int.
So the question is: is it possible to do it so that those 65 reads of int will be translated into several large transactions. I read in the documentation that reads can be even 128 bytes long. What exactly should I write so that assembler will make 1 large call?
Update:
Thank you for your replies. From the answer from user talonmies below, I changed the code replacing dimensions x and y for the neighbors matrix. Now consecutive threads load consecutive int[gid], I guess that may result in a 128 byte read. The program works 8% faster.
All memory transactions are issued (where possible) on a per warp basis. So the 128 byte transaction you are asking about is when all 32 threads in a warp issue a memory load instruction which can be serviced in a single "coalesced" transaction. A single thread can't issue large memory transactions, only a warp of 32 threads can, and only when the memory coalescing requirements of whichever architecture you run the code on can be satisfied.
I couldn't really follow your description of what you code is actually doing, but from first principles alone, the answer would appear to be no.
I have a Wi-Fi capture (.pcap) that I'm analysing and have run across what appear to me to be inconsistencies between the 802.11 spec and Wireshark's interpretation of the data. Specifically what I'm trying to pull apart is the 2-byte 802.11 Frame Control field.
Taken from http://www4.ncsu.edu/~aliu3/802.bmp, the format of the Frame Control field's subfields are as follows:
And below is a Wireshark screen cap of the packet that has me confused:
So as per the Wireshark screenshot, the flags portion (last 8 bits) of the Frame Control field is 0x22, which is fine. How the Version/Type/Subtype being 0x08 matches up with Wireshark's description of the frame is what has me confused.
0x08 = 0000 1000b, which I thought would translate to Version = 00, Type = 00 (which I thought meant management not data frame) and Subtype = 1000 (which I thought would be a beacon frame). So I would expect this frame to be a management frame and more specifically, a beacon frame. Wireshark however reports it as a Data frame. The second thing that is confusing me is where Wireshark is even pulling 0x20 from in the line Type/Subtype: Data (0x20).
Can anyone clarify my interpretation of the 802.11 spec/Wireshark capture for me and why the two aren't consistent?
The data frame in you example is 0x08 because of the layout of that byte of the frame control (FC). 0x08 = 00001000
- The first 4 bits (0000) are the subtype. 0000 is the subtype of this frame
- The next 2 bits (10) is the type, which is 2 decimal and thus a data type frame
- The last 2 bits (00) are the version, which is 0
The table below translates the hex value of the subtype-type-version byte of the FC for several frame types. A compare of the QoS data to the normal data frame might really help get this down pat. Mind you the table might have an error or two, as I just whipped it up.
You are right that 1000 is a beacon frame, you just were looking at the wrong bits.
You have a radiotap header, you can get the dec representation of the type like so from the pcap API:
int type = pkt_data[20] >> 2;
This is a common error, and has certainly bitten me several times.
It is down to the Byte Ordering.
When you have a multi-byte number to represent, the question arises as to Which byte do you put/send first ?
Natural (human) byte order is to put the big part first, then the smaller parts after it, Left-to-right, also called Big Endian. Note that the Bits in each byte are never the wrong way around from a programmers' point of view.
e.g. 1234 decimal requires 2 bytes, 04D2 hex.
Do you write/send 04 D2, or D2 04 ?
The first is Big-endian, the second is Little-endian.
To confuse it more, the mechanisms involved may use different byte-orders.
There is the Network Byte Order, in this case Little-endian, the Architecture byte order (can be different for each CPU architecture) and the data may be in a buffer, so it will vary depending on whether you read the buffer top-to-bottom, or bottom-to-top.
It doesn't help that the explanation of which bits do what can also be 'backwards', as in your original post.
I am using wireshark version-2.4.3 on windows. My capture file of dataframes is like below.
Frame control field = 0x0842 i.e., in binary format 0000 1000 0100 0010
Framecontrol flag field = 0x42.i.e., in binary format 0100 0010
So, as per my understanding the LSB 8bits in a framecontrol field will correspond to flags.
MSB 8bits will correspond to subtype, type, version i.e. in my case 0000-subtype & 10-type & 00-version.
Which is data frame of subtype 0.
It might be the error with wireshark in your case. It should dispaly frame control field as 0x0822 instead of 0x2208.
Flags field is properly displayed as 0x22.
In My case I am using wireshark-2.4.3 and display of frame control field is correct 0x0842 where flags is 0x42.
My_capture_file: