I have a row length of 499 bytes. How many bytes per row does SE use to store the row in the datafile?
How many rows will fit in a 2K page?
There is one byte of overhead. Each row is terminated by either a newline (not deleted) or a null byte (deleted).
However, SE uses C-ISAM and C-ISAM does not use pages for the data. Consecutive rows are simply stored consecutively. The index file uses pages, but not the data file.
Related
I understand why a multi-level page table saves memory. In the classic 32-bit example, the size of each Page Table Entry (PTE) in a single-level table is 32 bit (4 Bytes). The fact that 20 of the 32 bits are used for page number means there are 2^20 (~1M) possible entries in this table. The total size of a single-level table is therefore 4 Bytes * 1M = 4MB.A two-level page table requires less memory because it does not need all of the second-level pages at the same time.
However, in a lot of examples for the calculations of two-level page table on the internet, people still use 4 Bytes to indicate the size of each PTE. Since we are only accessing 10 of the 32 bits at the first level, shouldn't the size of these PTEs be smaller than 4 Bytes?
The virtual address is broken to 10-10-12 first 10 bits are an index in the first level page table, there, you will find the address of the second level page table, which you index with the seconds 10 bits to find the start of the physical frame which you index with the remaining 12 bits.
Actually each entry contains the number of the virtual page (there are 2^20 such pages) and 12 flag bits.
I am given a system with 64-bit virtual address space. with page size of 2KB.
Also it is given that the physical memory is of the size 16GB.
I need to calculate the following parameters:
number of page entries (number of lines in the page table), how many bits are needed for the page offset, how many bits are needed for the virtual page number (VPN), and how many bits are needed for the physical page number (PPN).
So, first I concluded that the size of the virtual memory is 2^64 bytes, and that means there are 2^53 entries in the page table.
From the size of a page I concluded that 11 bits are needed for the page offset.
From here I'm not so sure.
Since each virtual address is of the size 64 it, then the VPN is of the size 64 - 11 = 53 bits.
Since the physical memory is of the size 2^34 bytes, then a physical address if of 34 bits. Which means the PPN is of the size 34 - 11 = 23 bits.
Are my calculations correct? and also is my thinking correct?
Help would be appreciated
Some of your results are correct. PPN is 23 bits, VPN is 53 bits.
But all the stuff concerning the page tables is wrong.
A page table contains a set of physical page adresses. Hence as a PPN is 23 bits, one needs 4 bytes (the power of 2 above 23) to describe a PP. If pages are 2k bytes, you can store 2^9 PP adresses par page.
As VPN are 53bits, and each table can resolve 9 bits, the translation can be done by 6 consecutive tables.
If you are not familiar with multilevel pages, there are many good tuturials. See for instance https://en.wikipedia.org/wiki/Page_table
What is certain is that the PT size is NOT 2^53!! First because 2^55 is an insane amount of memory (~10^16). And second, because the total number of PP is 2^23, so why use a table 1 billion times larger... (and this is why we use multilevel page tables)
I've done some reading but I'm not entirely sure about one thing, for example how much memory would this use in JVM 64 bit(sorry if stupid question, but I'm a bit confused and don't know much about this):
MyObject[] myArray; - I know an array takes up 24 bytes, but how much will each element in this array take? is every element an object reference, meaning 8 byte per element? If not, how do I know how many bytes each element in this array needs?
Normally, that is when using heap sizes of less than 32 GB, the 64-bit JVM uses compressed oops which store object pointers as a 32-bit integer (scaled by three bits when used, since all objects are aligned to 8 bytes; see the link for details), so each element would actually only use 4 bytes.
If you use more than 32 GB of heap or otherwise turn off compressed oops, however, then each element will indeed use 8 bytes.
Also, I suspect that your statement on the array header being 24 bytes is wrong. To begin with, when compressing oops, the class reference in the header is also compressed, and the identity-hash-code and array length fields are 32-bit to begin with, so I suspect it is more likely to use 12 bytes. Even when using full-length oops, it should still only take 16 bytes. I can't find any hard source verifying either, however. In general, however, it should be said that Hotspot does not even use a fixed-size object header but one that varies in size depending on various circumstances of the object. This article describes some of those circumstances.
That is on the Hotspot JVM, at least. Since the JLS doesn't specify any primitive sizes, it could, theoretically, be anything on any given JVM, though 8 bytes are, of course, the most likely implementation choice.
Here is good information on how to calculate the memory usage of a Java array
For Example
let's consider a 10x10 int array. Firstly, the "outer" array has its 12-byte object header followed by space for the 10 elements. Those elements are object references to the 10 arrays making up the rows. That comes to 12+4*10=52 bytes, which must then be rounded up to the next multiple of 8, giving 56. Then, each of the 10 rows has its own 12-byte object header, 4*10=40 bytes for the actual row of ints, and again, 4 bytes of padding to bring the total for that row to a multiple of 8. So in total, that gives 11*56=616 bytes. That's a bit bigger than if you'd just counted on 10*10*4=400 bytes for the hundred "raw" ints themselves.
I was reading BMP file in hex editor while discovered something odd. Two first letters "BM" are written in order, however the next word(2B), which is means file size, is 36 30 in hex. Actual size is 0x3036. I've noticed that other numbers are stored the same way.
I'm also using MARS MIPS emulator which can display memory by words. String in.bmp is stored as b . n i / \0 p m.
Why data isn't stored continuously?
It depends not on the data itself but on how you store this data: per byte, per word (2 bytes, usually), or per long (4 bytes -- again, usually). As long as you store data per byte you don't see anything unusual; data appears "continuous". However, with longer units, you are subject to endianness.
It appears your emulator is assuming all words need to have their bytes reversed; and you can see in your example that this assumption is not always valid.
As for the BM "magic" signature: it's not meant to be read as a word value "BM", but rather as "first, a single byte B, then a single byte M". All next values are written in little-endian order, not only 'exchanging' your 36 and 30 but also the 2 zeroes 'before' (or 'after') (the larger values in the BMP header are of 4 bytes long type).
Give an ETS table with data, the info/1 function returns various properties for the table, including a size value which is specific to the number of rows rather than the physical size.
Is there any way to calculate the amount of memory in bytes occupied by an ETS table ?
ets:new( mytable, [bag, named_table, compressed]),
ets:insert( mytable, { Key, Value } ),
....
ets:info ( mytable ).
TL;DR:
ETS table allocated memory size in bytes:
ets:info(Table,memory) * erlang:system_info(wordsize).
To elaborate a bit, ets:info(Table,memory) gives you the words allocated to data in an ETS table
(same for Mnesia. You can view al that info in the TV application. The same attribute for DETS tables is in bytes).
A word is nothing more than the 'natural' data unit of a particular CPU architecture. What that represents depends on your architecture: 32-bit or 64-bit (or use erlang:system_info(wordsize) to get the correct word size immediately)
On a 32-bit system, a word is 4 bytes (32 bits).
On a 64-bit system, a word is 8 bytes (64 bits).
Also note that a ETS table initially spans 768 words, to wich you must add the size of each element, 6 words + size of Erlang data. It's not really clear if those are the words "allocated to data" ets:info specifies.
Calculating the exact size is a bit of a hassle: ETS tables have their own independent memory management system, which is optimized and garbage collected, and can vary depending on table type (set, bag, duplicate_bag). As an experiment, an empty table returns, in my environment, 300 words "allocated to data". If I add 11 tuples, size increases to 366 words. No Idea to how those relate to the initial 768 words, or why the size only increases by 11*6 words, when it should have been 11*6 + 11*1 (11 atoms), according to definition.
Still, a naive estimate, taking the initial table size and the words allocated to data, for example 22086 words, results in 768*8 + 22.086*8 = 182.832 bytes (178.54 KiB).
Of course, the bigger the data, the less those "structural" words matter, so you could only use the "words allocated to data" number, returned by ets:info, to estimate your table's size in memory.
Edit: There are two other functions that let you audit ETS memory usage:
erlang:memory/1: erlang:memory(ets) returns the memory size, in bytes, allocated to ETS.
ets:i/0: an overview of all active ETS tables (a bit like viewing system tables in TV, but with type and memory data).
As a small test, an newly created empty table increased memory use with 312 words (2.44 KiB), a lot less than the 768 number in the manual (perhaps it's CPU architecture related, I have no idea), while ETS itself reported 299 words (2.33 KiB) allocated to the data.
That's only 13 words (104 bytes) of structural overhead away (or so it seems, it remains nebulous) from the increase erlang:memory/1 reported, so ets:info/2 is fairly accurate after all.
After the insertion of a simple tuple consisting of 2 atoms, erlang:memory/1 reported a 8 word memory allocation increase, just like the documentation said it would (new ETS record: 6 words + size of data - 2 in this case : 1 word per atom).
you can read the document about ets.
you can ues this to get the memory allocated to the table.
ets:info ( mytable, memory).
{memory, integer() >= 0
The number of words allocated to the table.