I am using kafka-streams and the off heap memory usage grows up to the physical limits of the machine. However when running kafka-streams in docker memory usage grows past the limits of the container so the container gets OOM killed.
My assumption is that rocksdb is allocating the off heap space. -Xmx can be used to limit the heap usage but I can not find anything similar for the rockdb off heap usage.
How does rocksdb detect the physical memory limit and is there a way to simulate this limit in a container?
This is an issue of memory fragmentation.
You can either tune the glibc memory allocator by setting the environment variable MALLOC_ARENA_MAX=2 or change memory allocator from glibc to jemalloc.
I'm encountring a critical problem. working on a Java RCP app (desktop) which is frequently crashing on my machine (not on the machine of my collegue).
I set the xmx and the xms as the config of my collegue app and i run a diagnostic on memory but the app still crashes.
according to the charts of memory diagnostics, at the moment of the crash, the non heap memory is increasing (because of the increase of loaded classes). the RAM of the computer of my collegue is about 3 times superior to the RAM on my machine . After setting the heap memoy (xmx & xms) like on the other machine, i'm suspecting a heap memory problem(i'm using the last version of java 8).
There is any way to know the limit of the heap memory?
Does the value of the ram affect the non-heap memory?
I have a high end machine with 32 GB and I want to effectively utilize the available RAM for Java Process. Practically, I have seen I can run at max 3 JBoss instances (or Java Process) with Max 3GB Heap Size in one box still I have 20GB free space not utilized. Is it possible to create Java Program that can effectively directly work with RAM like C++, not relying on Java Memory Model for creating objects on JVM and also not relying on GC to reclaim memory. I mean Java Program directly working with RAM
I read somewhere that pinned memory in CUDA is scarce source. What is upper bound on pinned memory? In windows, in linux?
Pinned memory is just physical RAM in your system that is set aside and not allowed to be paged out by the OS. So once pinned, that amount of memory becomes unavailable to other processes (effectively reducing the memory pool available to rest of the OS).
The maximum pinnable memory therefore is determined by what other processes (other apps, the OS itself) are competing for system memory. What processes are concurrently running in either Windows or Linux (e.g. whether they themselves are pinning memory) will determine how much memory is available for you to pin at that particular time.
Under Windows Server 2003, Enterprise Edition, SP2 (/3GB switch not enabled)
As I understand it, and I may be wrong, the maximum addressable memory for a process is 4GB.
Is that 2GB of private bytes and 2GB of virtual bytes?
Do you get "out of memory" errors when the private byte limit or virtual byte limit is reached?
It is correct that the maximum address space of a process is 4GB, in a sense. Half of the address space is, for each process, taken up by the operating system. This can be changed with the 3GB switch but it might cause system instability. So, we are left with 2GB of addressable memory for the process to use on its own. Well, not entirely. It turns out that a part of this space is taken up by other stuff such as DLLs an other common code. The actual memory available to you as a programmer is around 1.5GB - 1.7GB.
I'm not sure about how you can handle accidentally going above this limit but I know of games which crash in large multiplayer maps for this reason. Another thing to note is that a 32bit program cannot use more than the 2GB address space on a 64bit system unless they enable the /LARGEADDRESSAWARE:YES linker flag.
Mark Russinovich started a series of posts on this..
Pushing the Limits of Windows: Physical Memory
While 4GB is the licensed limit for 32-bit client SKUs, the effective limit is actually lower and dependent on the system's chipset and connected devices. The reason is that the physical address map includes not only RAM, but device memory as well, and x86 and x64 systems map all device memory below the 4GB address boundary to remain compatible with 32-bit operating systems that don't know how to handle addresses larger than 4GB. If a system has 4GB RAM and devices, like video, audio and network adapters, that implement windows into their device memory that sum to 500MB, 500MB of the 4GB of RAM will reside above the 4GB address boundary.
You can only access 2Gb of memory in total (without the 3Gb switch) on 32bit Windows platforms.
You could run multiple 32bit VMs on a 64bit OS so that each app has access to as much memory as possible if your machine has more than 4Gb.
A lot of people are just starting to hit these barriers, I guess it's easier if your app is in .net or Java as the VMs happily go up to 32Gb of memory on 64bit os.
On 32 bits, if there is enough physical memory and disk space for virtual memory, memory runs out around 3GB since the kernel reserves the address space above 0xC0000000 for itself. On a 64 bits kernel running a 64 bits application, the limit is at 8TB.
For more details, check out MSDN - Memory Limits for Windows Releases
Maximum addressable memory for a 32bit machine is 4GB, for a 64bit machine you can address loads more. (Although some 32bit machines have extension systems for accessing more, but I don't think this is worth bothering with or considering for use).
You get out of memory errors when the virtual limit is reached. On Windows Server 2003, task manager tells you the limit on the performance tab labelled 'Commit Charge Limit'.