I have a 32-bit system. 2^32 bytes of memory. Let's say 4KB per page: 2^12 pages.
How does one determine the required number of pages? 2^32 / 2^12 = 2^20? So we have 20 bits for paging and 12 bits for offset?
Anyone have an example of how I could understand paging?
Your calculation is correct for a simple paging system, but things get more complicated for systems which allow variable sized pages (i.e. Pentium). Just be wary of the difference between virtual addressing and physical addressing... although your architecture might allow for 2^32 addressable bytes of memory, your actual hardware may have less or more memory.
If you want an example, just look up some university notes on operating systems... there are some lecs on memory management here.
We can Set page memory as 1.5 % multiple of our totaal available physical memory
ie if we have 4 gb of RAM then page file size can be upto 6 gb
Related
If a computer memory has addresses that are n bits long, what is the maximum number of memory cells that computer can have?
Can you even determine this based solely on the number of bits in the addresses?
I'm looking for what a formula would be to calculate the maximum number of memory cells based on the bit size of the memory cells.
The number of cells that can be accessed on a N bit computer is generally 2 to the Nth power. That does not imply that it is all RAM memory. Some of it may be consumed by the operating system. The processor may use some of it for memory mapped IO. Some of it may be reserved for future use. I had a PC back in the 80s, that also had banked memory. Basically your program could call an operating system function, and it swapped out one bank of memory for another. This was the era of 16 bit processors. 64k addressable space was the maximum you could have.
LINK 1: If size of the physical memory is 2^32-1, then what is the size of virtual memory?
the above link gives me an answer but i still do have some doubts.
pls answer in the way the questions posted here so that i will not be confused.....
1.Virtual memory is also called as Demand Paging whenever a page fault occurs
the operating system swaps the required page from the virtual memory. the virtual memory
here mean the harddisk or secondary storage. So how much space can be allocated for a
porcess in virutal memory? can this size(the space allocated for each process in the
Virtual memory) exceeds the size of our RAM size? i mean if our RAM is 4GB then what is
the maximum size of the virtual memory you can have for a process?can we have 4GB of
virtual memory for every process or can we have more than 4GB for every process?
(if it needs)
2.is the Virtual memory size fixed or dynamic? How much space is allocated for this memory
and in the above link it is told that 2^48 is the size of virtual memory in 64 bit machine
why is it only 2^48 and how can once can say a number like that?
thank you
If size of the physical memory is 2^32-1, then what is the size of virtual memory?
The size of the virtual address space is independent of the size of the physical address space. There is no answer.
So how much space can be allocated for a porcess in virutal memory?
That depends upon hardware limits, system parameters, and process quotas.
can this size(the space allocated for each process in the Virtual memory) exceeds the size of our RAM size?
Yes and it frequently does.
i mean if our RAM is 4GB then what is the maximum size of the virtual memory you can have for a process?
It can be anything. The rams size does not control.
can we have 4GB of virtual memory for every process or can we have more than 4GB for every process?
Both
is the Virtual memory size fixed or dynamic?
Dynamic
How much space is allocated for this memory and in the above link it is told that 2^48 is the size of virtual memory in 64 bit machine why is it only 2^48 and how can once can say a number like that?
It could be a hardware limit for a specific processor.
Paging is the way that virtual addresses are converted into physical addresses. This is done via page tables.
On x86 in Long Mode (64 bit mode), the page tables allow for 48 bit virtual address spaces (as in, 2^48 max size). This limitation is due to the design of x86's long mode page tables. Paging uses a few bits at a time from pointers to determine where to go next in the page tables. Basically, page tables are a relatively shallow b-tree style tree that let you look up the physical address corresponding to a virtual address.
To convert virtual addresses to physical addresses Long Mode page tables (for small pages) first extract 9 bits from the virtual address, then 9 more, then 9 more, then 9 more to find the right page, and use the low 12 bits to find the precise byte being accessed, for 48 bits total.
(For large and huge pages, x86 skips the last 1 and 2 steps of paging respectively, to find the address of the large or huge page, and the unused low 21 or 30 bits are used to find the precise byte in that page)
Virtual address spaces aren't necessarily dynamic, depending on what is meant by dynamic. The address space is always 48 bits (so long as you aren't switching between modes, like from long mode to protected mode with paging enabled (I.e. 32 bit mode)). Virtual address spaces are almost always sparse, as in most canonical (valid) addresses don't point don't point anything useful. The page tables don't have mappings for most addresses (accesses to those addresses generate page faults, which on Linux are often bounced back to userspace as the SIGSEGV you know and love).
That said, virtual memory can be dynamic in that when a page fault occurs the kernel could map in that page. To implement swap, OSes will use extra space on disk to give the illusion of more RAM by writing infrequently used pages back to disk, and lazily pulling pages back into RAM.
Fun fact, page tables have no restriction preventing the same physical page from being mapped in multiple times. You could build a monstrous page table with every virtual address pointing into exactly the same page (which is crazy), but doable. This means address spaces aren't necessarily sparse, just very likely to be. (Note that this page table would be huge. I'm sure someone has done the calculation, but my first guess would be order of terabytes)
X86 and x64 processors allow for 1GB pages when the PDPE flag is set on the cpu. In what application would this be practical or required and for what reason?
Hugepage would help in cases where you have a large memory footprint and memory access pattern spans large distance (across 4K pages).
It not only reduces TLB miss but also saves OS mm system page tables size.
A very good example is packet processing. In high throughput network applications (1Gbps or more), packets are normally stored in a packet buffer pool (i.e. pooling technique). For example, every packet buffer is 2KB in size and the pool contains 512 buffers. Access pattern of this packet buffer pool might not be sequential (buffer indexed at 1,2,3,4,5...) but rather random over time (1,104,407,45,905...). Since normal page size is 4K, normal TLB won't help here since each packet access would incur a TLB miss and there is a lot of different buffers sitting on different pages.
In contrast, if you put the pool in a 1GB hugepage, then all packet buffers share the same hugepageTLB entry thus avoiding misses.
This is used in DPDK (Data Plane Development Kit) where the packet
rate is very high that cycles wasted on TLB miss is not negligible.
Hugepage support is required for the large memory pool allocation used
for packet buffers (the HUGETLBFS option must be enabled in the
running kernel as indicated the previous section). By using hugepage
allocations, performance is increased since fewer pages are needed,
and therefore less Translation Lookaside Buffers (TLBs, high speed
translation caches), which reduce the time it takes to translate a
virtual page address to a physical page address. Without hugepages,
high TLB miss rates would occur with the standard 4k page size,
slowing performance.
http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#bios-setting-prerequisite-on-x86
Another example from Oracle:
...almost 6.8 GB of memory used for page tables when hugepages were not
configured...
...after hugepages were allocated and used by the Oracle database. The page table overhead was reduced to slightly less than 23 MB
http://www.databasejournal.com/features/oracle/understanding-hugepages-in-oracle-database.html
Related links:
https://en.wikipedia.org/wiki/Object_pool_pattern
--Edit--
However, hugepage should be used carefully. Above I mentioned that memory pool would benefit from 1GB hugepage. However, if you have an access pattern even across 1GB page boundary, then it might not help. There is an excellent blog on this:
http://www.pvk.ca/Blog/2014/02/18/how-bad-can-1gb-pages-be/
Imagine an application that uses huge amounts of memory—Molecular modeling. Weather prediction—especially if it has no user interaction.
Large pages:
(1) reduce the amount of page table overhead memory
(2) increases the amount of memory that can be stored in the MMU cache. (The same number of cache entries references more memory).
I have LabView installed on my Dell ws with 8 cores and 16GB DDRM, driving 4 24" monitors.If I create a video processor or compositor of most any type, with a 1024px x 1024px 'drawing' display, LabView reserves 1.5GB before I even began to composite. It was built from C and C++. I often store image details in 3D arrays of 256 x 256 x 256 of U32 integers that hold each RGB pixel color, plus the alpha channel for opacity or masking. That's 64MB per each layer of buffered video. If I need to remember 128 layers, thats 8GB right there. LabView is a programming langauge structured much like a CAD program. If I need 8GB for a series of video (HDTV) buffers, that is what it will give me, with a few seconds wait for malloc to do its work. If I created a 8GB 3D array for a database, it would be no different, even if I did it in MySQL (not as an array). To me, having many gigabytes of ram to play with is the norm, not an exception.
For a 4G ram, there is 4 * 1024 * 1024 * 1024 * 8 = 2^(32+3) bits. My question is how could a 32-bit PC can access a 4G memory. What I can think of this is "a byte is the storage unit, one can not store a data in a bit". Is this correct?
Another question is: in such PC, does a pointer always have size 32 bit? It seems reasonable for me, because we have 2^32 storage units to store the data. But in this answer and the next with their remarks, this is said to be wrong. If it is wrong, why?
Individual bits are accessed by reading the address of the byte containing it, modifying the byte and writing back if necessary.
In some architectures the smallest addressable unit is double word, in which case no single byte can be accessed "as is". Theoretically one could design an architecture that would address 16 GB of memory with 32-bits of unique addresses. And similar things happened years ago, when the addressable units of a Hard Drive were limited to bare 2^28 units of 512 byte sectors or so.
It's not completely wrong to say that PC's have 32-bit pointers. That's just a bit old information, as the newer models are internally 64-bit systems and can access depending on the OS up to 2^48 bytes of memory. Currently most existing PCs are 32-bit and nothing can be done about it.
Well, StuartLC remainded about paging. Even in the current 32-bit systems, one can use 48-bits of addressing using old age segment registers. (Can't remember if there was a restriction of segment registers low three bits being zero...) But anyway that would allow 2^45 bytes of individual addresses, out of which just a small fraction could ever be in the main memory simultaneously. If an OS supporting that addressing mode was developed, then probably full 64 bits would be allocated for the pointer. Just like it is today with 64-bit processors.
My question is how could a 32-bit PC can access a 4G memory
You may be confusing address bus (addressable memory) and the size of the processor registers. This superuser post details the differences.
Paging is a technique commonly used to allow memory to be addressed beyond the size of the OS's capabilities, e.g. see PAE
does a pointer always have size 32 bit
No, not necessarily - e.g. on 16 bit DOS and Windows, and also pointers could be relative to a segment.
Can one can not store a data in a bit?
Yes, you can, e.g. in C, bit packing in structs can be done, albeit at the cost of performance and portability.
Today performance is more important, and compilers will typically try and align data to its machine word size, for performance reasons.
I was reading http://duartes.org/gustavo/blog/post/motherboard-chipsets-memory-map and in specific, the following section:
In a motherboard the CPU’s gateway to
the world is the front-side bus
connecting it to the northbridge.
Whenever the CPU needs to read or
write memory it does so via this bus.
It uses some pins to transmit the
physical memory address it wants to
write or read, while other pins send
the value to be written or receive the
value being read. An Intel Core 2
QX6600 has 33 pins to transmit the
physical memory address (so there are
2^33 choices of memory locations) and
64 pins to send or receive data (so
data is transmitted in a 64-bit data
path, or 8-byte chunks). This allows
the CPU to physically address 64
gigabytes of memory (2^33 locations *
8 bytes) although most chipsets only
handle up to 8 gigs of RAM.
Now the math above states that since there are 33 pins for addressing, 2^33 * 8 bytes = 64 GB. All good, but now I get a bit confused. Let's say I install a 64 bit OS, I'll be able to address 64 GB total or 2^64Gb * 8 = 2^64GB (which is much more)? Also, assuming I'm using the same cpu above on a 32 bit cpu, I can address only 4 GB still (2^32 bits = 4Gb * 8 = 4GB)?
I think the physical vs "OS Allowable" is getting me confused.
Thanks!
You're confusing a bunch of things:
The size of a pointer limits the amount of virtual memory a user process can access. Not all of these will actually be usable by your process (it is traditional to reserve the "high" 1 or 2 GB for use by the kernel).
Not all virtual address bits are valid. The original AMD64 implementation effectively uses 48-bit sign-extended addresses (i.e. addresses in the range [0x0000800000000000,0xFFFF7FFFFFFFFFFF] are invalid). This exists largely to limit page tables to 4 levels, which decreases the cost of a page fault; you need 6-level page tables to address the full 2^64 bits, assuming 4K pages. For comparison, i386 has 2-level page tables.
Not all virtual addresses need to correspond to physical addresses at any given time. This is the whole point of virtual memory: you can address memory which doesn't "physically" exist, and the OS pages it in for you.
Not all physical addresses correspond to virtual addresses. They might not be mapped, for one, but it's also possible to have more physical memory than you can address. PAE supports up to 64 GB of physical addresses, and was common on servers before AMD64. While an indivial process can't address 64 GB, it means you can run a lot of multi-gigabyte processes without swapping all the time.
And finally: There's no point having more physical addresses than your RAM slots can handle. I have a D945GCLF2 board which supports AMD64, but only 2 GB of RAM. There's no point having extra physical address lines which can't be used anyway. (I'm handwaving over memory-mapped devices and the funky two-DIMMs-one-slot thing which I forget the name of.)
Also, note a few other things:
For memory-mapped I/O (in the hardware sense), the CPU needs to address individual bytes. It can't just do a 64-bit access. This seems to have been glossed over.
Modern processors include the memory controller on the CPU instead using the traditional northbridge and FSB (see HyperTransport and QuickPath).
Yes, number of bits in physical and virtual addresses can be different. Say, here is what 64-bit Linux says about the cores here (cat /proc/cpuinfo):
...
processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 280
stepping : 2
cpu MHz : 2392.623
cache size : 1024 KB
...
bogomips : 4784.41
TLB size : 1088 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp
There are a few things to consider about the physical address wires:
Each physical address wire ("pin") references a front-side-bus-word, not a byte address. If the CPU fetches 64-bit words, then the physical address wires will be aligned to that 8-byte boundary. Therefore, address lines A0-A2 are not wired because they would always be zero. Thus, the byte address range of the physical wires is increased by the width of the front-side bus.
The virtual memory system can maintain a map of 64-bit virtual addresses to n-bit physical addresses. In practice, the OS maintains a "physical max address" value which the VM mappings do not exceed.
Some memory architectures allow memory bank paging, where off-CPU hardware increases the effective physical memory address range by re-using some physical addresses for different "banks" of memory.
Imagine that in a 64-bit OS some of the wires to address memory don't go anywhere. The OS understands that this is pretty confusing, so it takes the standard 64-bit address and uses virtual memory mapping to make you believe that you're living in a flat 64-bit space.
The chipset limit is a big factor -- the hardware on the motherboard has to be able to pass the addresses from the CPU to the RAM. So the 8GB limit will apply unless you have a motherboard designed to handle more.
For reference, current 64-bit CPUs have the upper x-number-of bits (somewhere between 8 and 24 bits) of the address space wired together, as 64 bits is simply too much address space for now (you'd need 8 billion 2GB modules to take up that much address space). AMDs, for example, have a 48-bit limit (IIRC) on address space in a single segment. Which is more than enough, but nowhere near the theoretical max.
The main difference between a 64-bit and 32-bit OS is that one simply regards the primitive datatype (e.g. a word) as being wider. If the CPU can only physically address 2^33 locations, that won't change just because you're using a 64-bit OS. On the other hand, using a 32-bit OS will generally limit your addressable memory since 32-bit pointers can't represent all the possible values that your CPU could use to address memory (in your example, a 32-bit pointer is one bit short).
Long story short, your addressable memory is limited by both the pointer width (an OS restriction) and the data address bus width (a physical restriction). Some architectures have clever ways of getting around the OS pointer width by using two pointers, one to address a "bank" of memory and another to locally address within the bank. These schemes have sort of fallen out vogue lately, though.
Also, modern OSes generally use a virtual memory subsystem that translates logical addresses into their corresponding physical ones. With caching, the actual physical location of the memory could be in one (or several!) components along a memory heirarchy (e.g. processor cache, main memory, hard disk, etc.) Don't know how I completely forgot to mention VM, but it definitely would help your understanding to investigate it.
I believe that if you have a 64 bit operating system you can (theoretically) address 2^64 * 8 bytes = 16 EB (exabytes), but you will be limited by the hardware to 2^33 * 8 bytes = 64 GB. If you have a 32 bit OS you will not be able to utilize the full hardware capacity since the OS is the limiting factor, only being able to express 2^32 different addresses. I might be off but that's my current understanding.
I think you are getting confused by the fact the memory store 8 bytes at the same time , but an address (at the CPU level) refer to 1 byte (and not a bunch of 8). So with 32 bits you can "refer" to 2^32 bytes = 4GB. If you prefer +8 on pointer correspond to +1 on the number of the "physical" line.
You can then have access to more memory using pagination (not sure if it still used in modern computer).
To do an analogy with a library, you (or the CPU) can enumerate 32^2 books, but the librarian (the chipset) deals with shelves of book. So what is for you book #10, is book #2 or of the shelf #2 but you never see the shelves number. That's the job of the librarian to go to the good shelf and bring you the good book.
For me (another program on the same computer) book #10 could be a different one : book #2 of the shelf 100002 (because my page start at shelf 10000)
We can both refer to 32^2 different book, but they are not the same (and the library can have much more than that).
(Change have changed lots since I studied computer, so what I'm saying can be not 100 % accurate, but I think the idea is there)