I am running a data analysis code in docker using pandas on MacOS.
However, the program gets killed on high memory allocation in a data frame (I know because it gets killed when my program is loading a huge dataset).
Without the container, my program runs alright on my laptop.
Why is this happening and how can I change this?
Docker on MacOS is running inside a Linux VM which has an explicit memory allocation. From the docs:
MEMORY
By default, Docker for Mac is set to use 2 GB runtime memory,
allocated from the total available memory on your Mac. You can
increase the RAM on the app to get faster performance by setting this
number higher (for example to 3) or lower (to 1) if you want Docker
for Mac to use less memory.
Those instructions are referring to the Preferences dialog.
My Java application runs on Windows Server 2008 R2 and JDK 1.6
When I monitor it with JConsole, the committed virtual memory increases continually over time, while the heap memory usage stays below 50MB.
The max heap size is 1024MB.
The application creates many small, short-lived event objects over time. The behavior is as if each heap allocation is counted against the committed virtual memory.
When the committed virtual memory size approaches 1000MB, the application crashes with a native memory allocation failure.
java.lang.OutOfMemoryError: requested 1024000 bytes for GrET in C:\BUILD_AREA\jdk6\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
My conclusion is the virtual address space of 2GB has been exhausted.
Why does JConsole show the committed virtual address space growing over time even though the heap is not growing over time?
This is odd behavior but I have been able to reproduce it 100%. I'm currently testing Neo4j 2.0.1 Enterprise on my laptop and desktop machines. Laptop has 8GB RAM i7 4600U and Desktop has 16GB RAM i7 4770k. (Both machines are running Windows 8.1 x64 ENT and the same version of Java (latest as of FEB 19, 2014).
On first boot of each, when I run an expensive (or not so expensive) query, I can see the memory allocation go up (as expected for the cache). When starting the server, initial allocation is around 200-250mb, give or take. After a few expensive queries, it goes up to about 2GB, which is fine, I want this memory allocation. However, I have a batch script that stops the service, clears out the database and restarts the service (to start fresh when testing different development methods).
After 3 or 4 restarts, I noticed that the memory will NEVER climb above 400mb. Processor usage sits around 30-40% during the expensive query, but memory never increases. I will then get Unknown Error messages in the console when running other expensive queries. This is the same query that after a full reboot of the system, would bring the memory usage up to 2GB.
I'm not sure what could be causing this, or if there is a way to make sure that memory usage continues to be allocated, even on service restart. Rebooting a production server doesn't seem like a viable option, unless running in HA.
I have a high end machine with 32 GB and I want to effectively utilize the available RAM for Java Process. Practically, I have seen I can run at max 3 JBoss instances (or Java Process) with Max 3GB Heap Size in one box still I have 20GB free space not utilized. Is it possible to create Java Program that can effectively directly work with RAM like C++, not relying on Java Memory Model for creating objects on JVM and also not relying on GC to reclaim memory. I mean Java Program directly working with RAM
Under Windows Server 2003, Enterprise Edition, SP2 (/3GB switch not enabled)
As I understand it, and I may be wrong, the maximum addressable memory for a process is 4GB.
Is that 2GB of private bytes and 2GB of virtual bytes?
Do you get "out of memory" errors when the private byte limit or virtual byte limit is reached?
It is correct that the maximum address space of a process is 4GB, in a sense. Half of the address space is, for each process, taken up by the operating system. This can be changed with the 3GB switch but it might cause system instability. So, we are left with 2GB of addressable memory for the process to use on its own. Well, not entirely. It turns out that a part of this space is taken up by other stuff such as DLLs an other common code. The actual memory available to you as a programmer is around 1.5GB - 1.7GB.
I'm not sure about how you can handle accidentally going above this limit but I know of games which crash in large multiplayer maps for this reason. Another thing to note is that a 32bit program cannot use more than the 2GB address space on a 64bit system unless they enable the /LARGEADDRESSAWARE:YES linker flag.
Mark Russinovich started a series of posts on this..
Pushing the Limits of Windows: Physical Memory
While 4GB is the licensed limit for 32-bit client SKUs, the effective limit is actually lower and dependent on the system's chipset and connected devices. The reason is that the physical address map includes not only RAM, but device memory as well, and x86 and x64 systems map all device memory below the 4GB address boundary to remain compatible with 32-bit operating systems that don't know how to handle addresses larger than 4GB. If a system has 4GB RAM and devices, like video, audio and network adapters, that implement windows into their device memory that sum to 500MB, 500MB of the 4GB of RAM will reside above the 4GB address boundary.
You can only access 2Gb of memory in total (without the 3Gb switch) on 32bit Windows platforms.
You could run multiple 32bit VMs on a 64bit OS so that each app has access to as much memory as possible if your machine has more than 4Gb.
A lot of people are just starting to hit these barriers, I guess it's easier if your app is in .net or Java as the VMs happily go up to 32Gb of memory on 64bit os.
On 32 bits, if there is enough physical memory and disk space for virtual memory, memory runs out around 3GB since the kernel reserves the address space above 0xC0000000 for itself. On a 64 bits kernel running a 64 bits application, the limit is at 8TB.
For more details, check out MSDN - Memory Limits for Windows Releases
Maximum addressable memory for a 32bit machine is 4GB, for a 64bit machine you can address loads more. (Although some 32bit machines have extension systems for accessing more, but I don't think this is worth bothering with or considering for use).
You get out of memory errors when the virtual limit is reached. On Windows Server 2003, task manager tells you the limit on the performance tab labelled 'Commit Charge Limit'.