I am running a data analysis code in docker using pandas on MacOS.
However, the program gets killed on high memory allocation in a data frame (I know because it gets killed when my program is loading a huge dataset).
Without the container, my program runs alright on my laptop.
Why is this happening and how can I change this?
Docker on MacOS is running inside a Linux VM which has an explicit memory allocation. From the docs:
MEMORY
By default, Docker for Mac is set to use 2 GB runtime memory,
allocated from the total available memory on your Mac. You can
increase the RAM on the app to get faster performance by setting this
number higher (for example to 3) or lower (to 1) if you want Docker
for Mac to use less memory.
Those instructions are referring to the Preferences dialog.
Related
I'm running docker desktop community 2.1.0.3 on MacOS Mojave. I've got 8GB of memory allocated to Docker, which already seems like a lot (that's half my RAM). Somehow even after exiting and then starting Docker for Mac again, which means no containers are running, docker is already exceeding the memory allocation by 1GB.
What is expected memory usage for docker with no containers running? Is there a memory leak in docker for mac or docker's hyperkit?
As #GabLeRoux has shared in a comment, the "Real Memory" usage is much lower than what you see in the "Memory" column in Activity Monitor.
This document thoroughly explains memory usage on Mac OS with Docker Desktop and information is excerpted from there.
To see the "Real Memory" used by Docker, right-click the column names in Activity Monitor and Select "Real Memory". The value in this column is what's currently physically allocated to com.docker.hyperkit.
Alternate answer: I reduced the number of CPUs and Memory Docker is allowed to use within the Docker Resources preferences. My computer is running faster and quieter now.
I just now put this in place, so time will tell if this solution works for me. Before it was making my computer max out on memory. Now it's significantly reduced.
Thank you for the note on real memory. I added that to my Activity Monitor.
UPDATE: It's been a few days now and my computer runs well below the max of memory and my fan runs at a minimum if at all.
I think you shouldn't be using swap while ram is not full, for ssd health and speed
For cloud run's memory usage from the docs (https://cloud.google.com/run/docs/configuring/memory-limits)
Cloud Run applications that exceed their allowed memory limit are terminated.
When you configure memory limit settings, the memory allocation you are specifying is used for:
Operating your service
Writing files to disk
Running binaries or other processes in your container, such as the nginx web server.
Does the size of the container count towards "operating your service" and counts towards the memory limit?
We're intending to use images that could already approach the memory limit, so we would like to know if the service itself will only have access to what is left after subtracting container size from the limit
Cloud Run PM here.
Only what you load in memory counts toward your memory usage. So for example, if you have a 2GB container but only execute a very small binary inside it, then only this one will count as used memory.
This means that if your image contains a lot of OS packages that will never be loaded (because for example you inherited from a.big base image), this is fine.
Size of the container image you deploy to Cloud Run does not count towards the memory limit. For example, if your container image is 3 GiB, you can still run on a 256 MiB memory environment.
Writing new files to local filesystem, or (obviously) allocating more memory within your app will count towards the memory usage of your container. (Perhaps also obvious, but worth mentioning) the operating system will "load" your container's entrypoint executable to memory (well, to execute it). That will count towards the available memory as well.
I have setup a JupyterHub and configured a pyspark kernel for it. When I open a pyspark notebook (under username Jeroen), two processes are added, a Python process and a Java process. The Java process is assigned 12g of virtual memory (see image). When running a test script on a range of 1B number it grows to 22g. Is that something to worry about when we work on this server with multiple users? And if it is, how can I prevent Java from allocating so much memory?
You don't need to worry about virtual memory usage, reserved memory is much more important here (the RES column).
You can control size of JVM heap usage using --driver-memory option passed to spark (if you use pyspark kernel on jupyterhub you can find it in environment under PYSPARK_SUBMIT_ARGS key). This is not exactly the memory limit for your application (there are other memory regions on JVM), but it is very close.
So, when you have multiple users setup, you should learn them to set appropriate driver memory (the minimum they need for processing) and shutdown notebooks after they finish work.
I'm encountring a critical problem. working on a Java RCP app (desktop) which is frequently crashing on my machine (not on the machine of my collegue).
I set the xmx and the xms as the config of my collegue app and i run a diagnostic on memory but the app still crashes.
according to the charts of memory diagnostics, at the moment of the crash, the non heap memory is increasing (because of the increase of loaded classes). the RAM of the computer of my collegue is about 3 times superior to the RAM on my machine . After setting the heap memoy (xmx & xms) like on the other machine, i'm suspecting a heap memory problem(i'm using the last version of java 8).
There is any way to know the limit of the heap memory?
Does the value of the ram affect the non-heap memory?
This is odd behavior but I have been able to reproduce it 100%. I'm currently testing Neo4j 2.0.1 Enterprise on my laptop and desktop machines. Laptop has 8GB RAM i7 4600U and Desktop has 16GB RAM i7 4770k. (Both machines are running Windows 8.1 x64 ENT and the same version of Java (latest as of FEB 19, 2014).
On first boot of each, when I run an expensive (or not so expensive) query, I can see the memory allocation go up (as expected for the cache). When starting the server, initial allocation is around 200-250mb, give or take. After a few expensive queries, it goes up to about 2GB, which is fine, I want this memory allocation. However, I have a batch script that stops the service, clears out the database and restarts the service (to start fresh when testing different development methods).
After 3 or 4 restarts, I noticed that the memory will NEVER climb above 400mb. Processor usage sits around 30-40% during the expensive query, but memory never increases. I will then get Unknown Error messages in the console when running other expensive queries. This is the same query that after a full reboot of the system, would bring the memory usage up to 2GB.
I'm not sure what could be causing this, or if there is a way to make sure that memory usage continues to be allocated, even on service restart. Rebooting a production server doesn't seem like a viable option, unless running in HA.