In the context of multiprocessor operating systems, what is meant by the following quote?
The paging mechanisms on different processors must be coordinated to
enforce consistency when several processors share a page or segment
and to decide on page replacement.
The reuse of physical pages is the biggest problem of concern; that
is, it must be guaranteed that a physical page can no longer be
accessed with its old contents before the page is put to a new use.
The following section is specifically what I do not understand:
The reuse of physical pages is the biggest problem of concern; that
is, it must be guaranteed that a physical page can no longer be
accessed with its old contents before the page is put to a new use.
I would greatly appreciate it if someone could please take the time to clarify this concept for a newbie.
Let's say that you have a physical page frame that has been previously mapped to a process as a logical page. That means that the process's page tables reference that page.
If, as part of the virtual memory management, the operating system takes that page frame away from the process (to give it to anther) it must ensure that the process's page tables do not reference the page frame.
Otherwise, you would have two (or more) processes reading and writing to the same page independently from each other and causing general havoc.
Your quote refers to the fact that the process of allocating and deallocating page frames becomes more complex in a multi-processor system.
One complexity not mentioned is that it is possible for multiple processes to map to the same page frame. When such a page frame is being deallocated, the page table for ALL the referencing processes most be updated to remove their references to it.
Related
If I have a process which needs 6KB of RAM and the page size is 4KB, I need to allocate two pages. Can another PROCESS access the remaining 2KB for himself, so that two processes share same page table?
Can two processes share same page, in case that the other page isn't "full"?
In theory it is possible, but there are significant issues:
As #Peter says, allowing both processes to share the same page would bypass traditional process memory protections, as for most processors access protection granularity would be no smaller than a whole page.
The two processes would have to coordinate in some way about who gets what part of that shared page. This could range from
simple coordination that says, process 1 gets the first half and process 2 gets the 2nd half — but this becomes silly when process 1 or process 2 needs more memory, since at that point probably would have been better off simply having their own pages.
the processes communicate with each other to formalize a split between who gets what of that page. Such communication would typically be some kind of synchronization, which is a bottleneck for many situations.
Consider multiple threads in the same process — some modern runtime systems, e.g. for Java and C#, provide a separate per-thread heap so that simple memory allocations do not require synchronization with other threads.
Having one page that is not full represents less than a page of waste per process, which is not very high overhead, so not really a problem that needs solving, given the issues of security and coordination.
Effectively, the operating systems already share the whole of physical memory between processes albeit at a page granularity, so there is sharing (just not intra-page sharing) and the amount of waste is bounded.
I know that Operating systems usually keep Page Tables to map a chunk of virtual memory to a chunk of physical memory.
My question is, does the CPU load the whole chunk when it's loading a given byte?
lets say I have:
ld %r0, 0x4(%r1)
Assuming my page size is 4 KB, does the the CPU load all 4KB at once or it manages to
Load only a byte given the offset properly?
Is the page size mandated by the hardware or configurable by software and the OS?
Edit:
Figured that page size is mandated by hardware:
The available page sizes depend on the instruction set architecture, processor type, and operating (addressing) mode. The operating system selects one or more sizes from the sizes supported by the architecture
your question touches so many levels of cpu/memory architecture ... without knowing the exact cpu-architecture/memory-architecture/version you have in mind: while the cpu-command targets only one byte, it will trigger the memory-controller to locate the whole physical-page and load that one (atleast, prefetch might kick in) completely to second/first-level cache. your data is transfered after filling the cache.
On a typical modern CPU, yes, it loads the whole page.
It couldn't really work any other way, since there are only two states in the page tables for a given page: present and not present. If the page is present, it must be mapped to some page in physical memory. If not present, every access to that page produces a page fault. There is no "partially present" state.
In order for it to be safe for the OS to mark the page present, it has to load the entire page into physical memory and update the page tables to point the virtual page to the physical page. If it only loaded a single byte or a smaller amount, the application might later try to access some other byte on the same page that hadn't been loaded, and it'd read garbage. There's no way for the CPU to generate another page fault in that case to let the OS fix things up, unless the page were marked not present, in which case the original access wouldn't be able to complete either.
The page size is fixed in hardware, though some architectures offer a few different choices that the OS can select from. For instance, recent x86-64 CPUs allow pages to be either 4 KB, 2 MB or 1 GB. The OS can mix-and-match these at runtime; there are bits in the page tables to indicate the size of each page.
According to "Windows Internals, Part 1" (7th Edition, Kindle version):
Pages in a process virtual address space are either free, reserved, committed, or shareable.
Focusing only on the reserved and committed pages, the first type is described in the same book:
Reserving memory means setting aside a range of contiguous virtual addresses for possible future use (such as an array) while consuming negligible system resources, and then committing portions of the reserved space as needed as the application runs. Or, if the size requirements are known in advance, a process can reserve and commit in the same function call.
Both reserving or committing will initially get you entries in the VADs (virtual address descriptors), but neither operation will touch the PTE (page table entries) structures. It used to cost PTEs for reserving before Windows 8.1, but not anymore.
As described above, reserved means blocking a range of virtual addresses, NOT blocking physical memory or paging file space at the OS level. The OS doesn't include this in the commit limit, therefore when the time comes to allocate this memory, you might get a surprise. It's important to note that reserving happens from the perspective of the process address space. It's not that there's any physical resource reserved - there's no stamping of "no vacancy" against RAM space or page file(s).
The analogy with plots of land might be missing something: take reserved as the area of land surrounded by wooden poles, thus letting others now that the land is taken. But how about committed ? It can't be land on which structures (eg houses) have already been build, since those would require PTEs and there's none there yet, since we haven't accessed anything. It's only when touching committed data that the PTEs will get built, which will make the pages available to the process.
The main problem is that committed memory - at least in its initial state - is functionally very much alike reserved memory. It's just an area blocked within VADs. Try to touch one of the addresses, and you'll get an access violation exception for a reserved address:
Attempting to access free or reserved memory results in an access violation exception because the page isn’t mapped to any storage that can resolve the reference
...and an initial page fault for a committed one (immediately followed by the required PTE entries being created).
Back to the land analogy, once houses are build, that patch of land is still committed. Yet this is a bit peculiar, since it was still committed when the original grass was there, before the very first shovel was excavated to start construction. It resembled the same state as that of a reserved patch. Maybe it would be better to think of it like terrain eligible for construction. Eg you have a permit to build (albeit you might never build as much as a wall on that patch of land).
What would be the reasons for using one type of memory versus the other ? There's at least one: the OS guarantees that there will be room to allocate committed memory, should that ever occur in the future, but doesn't guarantee anything for reserved memory aside from blocking that process' address space range. The only downside for committed memory is that one or more paging files might need to be extended in size as to be able to make the commit limit take into account the recently allocated block, so should the requester demand the use of part of all the data in the future, the OS can provide access to it.
I can't really think how the land analogy can capture this detail of "guarantee". After all, the reserved patch also physically existed, covered by the same grass as a committed one in its pristine state.
The stack is another scenario where reserved and committed memory are used together:
When a thread is created, the memory manager automatically reserves a predetermined amount of virtual memory, which by default is 1 MB.[...] Although 1 MB is reserved, only the first page of the stack will be committed [...]
along with a guard page. When a thread’s stack grows large enough to touch the guard page, an exception occurs, causing an attempt to allocate another guard. Through this mechanism, a user stack doesn’t immediately consume all 1 MB of committed memory but instead grows with demand."
There is an answer here that deals with why one would want to use reserved memory as opposed to committed . It involves storing continuously expanding data - which is actually the stack model described above - and having specific absolute address ranges available when needed (although I'm not sure why one would want to do that within a process).
Ok, what am I actually asking ?
What would be a good analogy for the reserved/committed concept ?
Any other reason aside those depicted above that would mandate the
use of reserved memory ? Are there any interesting use cases when
resorting to reserved memory is a smart move ?
Your question hits upon the difference between logical memory translation and virtual memory translation. While CPU documentation likes to conflate these two concepts, they are different in practice.
If you look at logical memory translation, there are are only two states for a page. Using your terminology, they are FREE and COMMITTED. A free page is one that has no mapping to a physical page frame and a COMMITTED page has such a mapping.
In a virtual memory system, the operating system has to maintain a copy of the address space in secondary storage. How this is done depends upon the operating system. Typically, a process will have its mapping to several different files for secondary storage. The operating system divides the address space into what is usually called a SECTION.
For example, the code and read only data could be stored virtually as one or more SECTIONS in the executable file. Code and static data in shared libraries could each be in a different section that are paged to the shared libraries. You might have a map to a shared filed to the process that uses memory that can be accessed by multiple processes that forms another section. Most of the read/write data is likely to be in a page file in one or more sections. How the operating system tracks where it virtually stores each section of data is system dependent.
For windows, that gives the definition of one of your terms: Sharable. A sharable section is one where a range of addresses can be mapped to different processes, at different (or possibly the same) logical addresses.
Your last term is then RESERVED. If you look at the Windows' VirtualAlloc function documentation, you can see that (among your options) you can RESERVE or COMMIT. If you reserve you are creating a section of VIRTUAL MEMORY that has no mapping to physical memory.
This RESERVE/COMMIT model is Windows-specific (although other operating systems may do the same). The likely reason was to save disk space. When Windows NT was developed, 600MB drives the size of washing machine were still in use.
In these days of 64-bit address spaces, this system works well for (as you say) expanding data. In theory, an exception handler for a stack overrun can simply expand the stack. Reserving 4GB of memory takes no more resources than reserving a single page (which would not be practicable in a 32-bit system—see above). If you have 20 threads, this makes reserving stack space efficient.
What would be a good analogy for the reserved/committed concept ?
One could say RESERVE is like buying options to buy and COMMIT is exercising the option.
Any other reason aside those depicted above that would mandate the use of reserved memory ? Are there any interesting use cases when resorting to reserved memory is a smart move ?
IMHO, the most likely places to RESERVE without COMMITTING are for creating stacks and heaps with the former being the most important.
I was wondering about this because it's a potential security hole if process A can malloc 50 megs of data that is not zero'd out and that chunk of memory turns out to include what had been physical pages from process B and still contain process B's data.
Is malloc'd data zeroed in objective c?
Mostly Yes. There's a zero-page writer that is part of the memory manager which provides a process with zero'd pages. The memory manager will call memory_object_data_unavailable to tell the kernel to supply zero-filled memory for the region.
If the process calls free and then mallocs again, the page is not re-zero'd. Zeroization only occurs when a new page is demanded. In fact, the page is probably not returned to the system upon free. The process retains the page for its own use due to the runtime. Related, see Will malloc implementations return free-ed memory back to the system?
If a page is returned to the system under a low-memory condition, the the page will be re-zero'd even if the process formerly held the page. The memory manager does not account for last owner of a page. It just assumes a new page needs to be zero'd to avoid an information leak across processes.
Note Microsoft calls it the zero-page writer. Darwin has the same component, but I don't recall seeing it named. Also see Mac OS X Internals: A Systems Approach by Singh. Its a bit dated, but it provides a lot of system information. Chapter 8, Memory, is the chapter of interest.
Singh's book goes into other details, like cases where a page is demanded but does not need to be zeroized. In this case, there was some shared data among processes, and a new page was allocated to the process under a Copy-on-Write (COW) scheme. Effectively, the new page was populated from existing data rather than zero's. The function of interest is memory_object_data_request.
Linux has an interesting discussion of the zero page at Some ado about zero. Its interesting reading about a topic that seems mundane on the surface.
There is chance were a heavy weight application that needs to be launched in a low configuration system.. (Especially when the system has too less memory)
Also when we have already opened lot of application in the system & we keep on trying opening new new application what would happen?
I have only seen applications taking time to process or hangs up for sometime when I try operating with it in low config. system with low memory and old processors..
How it is able to accomodate many applications when the memory is low..? (like 128 MB or lesser..)
Does it involves any paging or something else..?
Can someone please let me know the theory behind this..!
"Heavyweight" is a very vague term. When the OS loads your program, the EXE is mapped in your address space, but only the code pages that run (or data pages that are referenced) are paged in as necessary.
You will likely get horrible performance if pages need to constantly be swapped as the program runs (aka many hard page faults), but it should work.
Since your commit charge is near the commit limit, and the commit limit will likely have no room to grow, you will also likely recieve many malloc()/VirtualAlloc(..., MEM_COMMIT)/HeapAlloc()/{Local|Global}Alloc() failures so you need to watch the return codes in your program.
Some keywords for search engines are: paging, swapping, virtual memory.
Wikipedia has an article called Paging (Redirected from Swap space).
There is often the use of virtual memory. Virtual memory pages are mapped to physical memory if they are used. If a physical page is needed and no page is available, another is written to disk. This is called swapping and that explains why crowded systems get slow and memory upgrades have positive effects on performance.