I'm running a jupyter/scipy-notebook Docker container.
I have not restricted the memory assigned to the container with the run command.
However, what I'm seeing issuing the docker stats command is that the container is limiting its memory usage to 2 GB (on 16 GB available!), even if doing complex calculations.
How is this possible?
Alter the resources (RAM) settings from Docker Desktop - MAC/Windows.
MAC - Docker Desktop
Preferences --> Advanced --> Change Ram Settings
Windows - Docker Desktop
Settings --> Resources --> Change the CPU / RAM / SWAP Settings
Reference: Compiled the solution from #samirko and #Andris Birkmanis. (Added Windows Solution)
I am running Docker on Mac OS and Jupyter crashed when trying to read over 600MB CSV file.
Following Andris Birkmanis instructions helped to tackle the issue by increasing the size of allocated memory for Docker.
If everything is going well, by default, docker shouldn't limit by default memory usage at all. So, your MEM USAGE / LIMIT doing docker stats [containerid] should be the same than your total memory (16Gb in your case), although it's not free but available.
Furthermore, there's no way to set by default a docker memory limit invoking dockerd,
So, the only thing I can purpose is specify memory limit in docker run
-m, --memory="" Memory limit (format: <number>[<unit>]). Number is a positive integer. Unit can be one of b, k, m, or g. Minimum is 4M.
--memory-swap="" Total memory limit (memory + swap, format: <number>[<unit>]). Number is a positive integer. Unit can be one of b, k, m, or g.
--memory-reservation="" Memory soft limit (format: <number>[<unit>]). Number is a positive integer. Unit can be one of b, k, m, or g.
--kernel-memory="" Kernel memory limit (format: <number>[<unit>]). Number is a positive integer. Unit can be one of b, k, m, or g. Minimum is 4M.
For more information, please check Docker documentation run-time options
Check your docker run --memory-reservation=10g ...and let's see.
Related
I am running Docker containers containing JVM (java8u31). These containers are deployed as pods in a kubernetes cluster. Often I get OOM for the pods and Kubernetes kills the pods and restarts it. I am having issues in finding the root cause for these OOMs as I am new to Kubernetes.
Here are the JVM parameters
-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Xms700M -Xmx1000M -XX:MaxRAM=1536M -XX:MaxMetaspaceSize=250M
These containers are deployed as stateful set and following is the resource allocation
resources:
requests:
memory: "1.5G"
cpu: 1
limits:
memory: "1.5G"
cpu: 1
so the total memory allocated to the container matches the MaxRam
If I use -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/etc/opt/jmx/java_pid%p.hprof that doesn't help because the pod is getting killed and recreated and started as soon as there is a OOM so everything within the pod is lost
The only way to get a thread or HEAP dump is to SSH into the pod which also I am not able to take because the pod is recreated after an OOM so I don't get the memory footprint at the time of OOM. I SSH after an OOM which is not much help.
I also profiled the code using visualVM, jHat but couldn't find substantial memory footprint which could lead to a conclusion of too much memory consumption by the threads running within the JVM or a probable leak.
Any help is appreciated to resolve the OOM thrown by Kubernetes.
When your application in a pod reaches the limits of memory you set by resources.limits.memory or namespace limit, Kubernetes restarts the pod.
The Kubernetes part of limiting resources is described in the following articles:
Kubernetes best practices: Resource requests and limits
Resource Quotas
Admission control plugin: ResourceQuota
Assign Memory Resources to Containers and Pods
Memory consumed by Java application is not limited to the size of the Heap that you can set by specifying the options:
-Xmssize Specifies the initial heap size.
-Xmxsize Specifies the maximum heap size.
Java application needs some additional memory for metaspace, class space, stack size, and JVM itself needs even more memory to do its tasks like garbage collection, JIT optimization, Off-heap allocations, JNI code.
It is hard to predict total memory usage of JVM with reasonable precision, so the best way is to measure it on the real deployment with usual load.
I would recommend you to set the Kubernetes pod limit to double Xmx size, check if you are not getting OOM anymore, and then gradually decrease it to the point when you start getting OOM. The final value should be in the middle between these points.
You can get more precise value from memory usage statistics in a monitoring system like Prometheus.
On the other hand, you can try to limit java memory usage by specifying the number of available options, like the following:
-Xms<heap size>[g|m|k] -Xmx<heap size>[g|m|k]
-XX:MaxMetaspaceSize=<metaspace size>[g|m|k]
-Xmn<young size>[g|m|k]
-XX:SurvivorRatio=<ratio>
More details on that can be found in these articles:
Properly limiting the JVM’s memory usage (Xmx isn’t enough)
Why does my Java process consume more memory than Xmx
The second way to limit JVM memory usage is to calculate heap size based on the amount of RAM(or MaxRAM). There is a good explanation of how it works in the article:
The default sizes are based on the amount of memory on a machine, which can be set with the -XX:MaxRAM=N flag.
Normally, that value is calculated by the JVM by inspecting the amount of memory on the machine.
However, the JVM limits MaxRAM to 1 GB for the client compiler, 4 GB for 32-bit server compilers, and 128 GB for 64-bit compilers.
The maximum heap size is one-quarter of MaxRAM .
This is why the default heap size can vary: if the physical memory on a machine is less than MaxRAM , the default heap size is one-quarter of that.
But even if hundreds of gigabytes of RAM are available, the most the JVM will use by default is 32 GB: one-quarter of 128 GB. The default maximum heap calculation is actually this:
Default Xmx = MaxRAM / MaxRAMFraction
Hence, the default maximum heap can also be set by adjusting the value of the - XX:MaxRAMFraction=N flag, which defaults to 4.
Finally, just to keep things interesting, the -XX:ErgoHeapSizeLimit=N flag can also be set to a maximum default value that the JVM should use.
That value is 0 by default (meaning to ignore it); otherwise, that limit is used if it is smaller than MaxRAM / MaxRAMFraction .
The initial heap size choice is similar, though it has fewer complications. The initial heap size value is determined like this:
Default Xms = MaxRAM / InitialRAMFraction
As can be concluded from the default minimum heap sizes, the default value of the InitialRAMFraction flag is 64.
The one caveat here occurs if that value is less than 5 MB —or, strictly speaking, less than the values specified by -XX:OldSize=N (which defaults to 4 MB) plus -XX:NewSize=N (which defaults to 1 MB).
In that case, the sum of the old and new sizes is used as the initial heap size.
This article gives you a good point to start tuning your JVM for web-oriented application:
Java VM Options You Should Always Use in Production
If you are able to run on Java 11 (or 10) instead of 8, the memory limit options have been much improved (plus the JVM is cgroups-aware). Just use -XX:MaxRAMPercentage (range 0.0, 100.0):
$ docker run -m 1GB openjdk:11 java -XshowSettings:vm -XX:MaxRAMPercentage=80 -version
VM settings:
Max. Heap Size (Estimated): 792.69M
Using VM: OpenJDK 64-Bit Server VM
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment (build 11.0.1+13-Debian-2)
OpenJDK 64-Bit Server VM (build 11.0.1+13-Debian-2, mixed mode, sharing)
That way, you can easily specify 80% of available container memory for the heap, which wasn't possible with the old options.
Thanks #VAS for your comments. Thanks for the kubernetes links.
After few tests I think that its not a good idea to specify XMX if you are using -XX:+UseCGroupMemoryLimitForHeap since XMX overrides it. I am still doing some more tests & profiling.
Since my requirement is running a JVM inside a docker container. I did few tests as mentioned in the posts by #Eugene. Considering every app running inside a JVM would need HEAP and some native memory, I think we need to specify -XX:+UnlockExperimentalVMOptions, XX:+UseCGroupMemoryLimitForHeap, -XX:MaxRAMFraction=1 (considering only the JVM running inside the container, at the same time its risky) -XX:MaxRAM (I think we should specify this if MaxRAMFraction is 1 so that you leave some for native memory)
Few tests:
As per below docker configuration, the docker is allocated 1 GB considering you only have the JVM running inside the container. Considering docker's allocation to 1G and I also want to allocate some to the process/native memory, I think I should use MaxRam=700M so that I have 300 MB for native.
$ docker run -m 1GB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XX:MaxRAM=700M -XshowSettings:vm -version
VM settings:
Max. Heap Size (Estimated): 622.50M
Ergonomics Machine Class: server
Using VM: OpenJDK 64-Bit Server VM
Now specifying XX:MaxRAMFraction=1 might be killing:
references: https://twitter.com/csanchez/status/940228501222936576?lang=en
Is -XX:MaxRAMFraction=1 safe for production in a containered environment?
Following would be better, please note I have removed MaxRAM since MaxRAMFraction > 1 :
$ docker run -m 1GB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 -XshowSettings:vm -version
VM settings:
Max. Heap Size (Estimated): 455.50M
Ergonomics Machine Class: server
Using VM: OpenJDK 64-Bit Server VM
This gives rest of the 500M for native e.g. could be used for MetaSpace by specifying -XX:MaxMetaspaceSize:
$ docker run -m 1GB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 -XX:MaxMetaspaceSize=200M -XshowSettings:vm -version
VM settings:
Max. Heap Size (Estimated): 455.50M
Ergonomics Machine Class: server
Using VM: OpenJDK 64-Bit Server VM
Logically and also as per the above references, it makes sense to specify -XX:MaxRAMFraction >1. This also depends on the application profiling done.
I am still doing some more tests, will update these results or post. Thanks
recently I've come also across similar issue
java 11.0.11+9 + kubernetes running docker containers in pod
similar config as op
resources:
requests:
memory: "1G"
cpu: 400m
limits:
memory: "1G"
with -XX:MaxRAMPercentage=60.0
Our service uploads and downloads a lot of data. Therefore direct memory is being used and in this issue I've found that MaxDirectMemorySize is equal to heapsize. So if we calculate the memory usage it could go behind limit 1G (1G * 0.6 * 2). In this case we've increased memory to 1.5G and changed -XX:MaxRAMPercentage=35.0 so we have enough space for heap + direct memory and even for some OS related tasks. Be cautious when you set up MaxRAMPercentage or Xmx in container environment.
We're running Docker containers of NiFi 1.6.0 in production and have to come across a memory leak.
Once started, the app runs just fine, however, after a period of 4-5 days, the memory consumption on the host keeps on increasing. When checked in the NiFi cluster UI, the JVM heap size used hardly around 30% but the memory on the OS level goes to 80-90%.
On running the docker starts command, we found that the NiFi docker container is consuming the memory.
After collecting the JMX metrics, we found that the RSS memory keeps growing. What could be the potential cause of this? In the JVM tab of cluster dialog, young GC also seems to be happening in a timely manner with old GC counts shown as 0.
How do we go about identifying in what's causing the RSS memory to grow?
You need to replicate that in a non-docker environment, because with docker, memory is known to raise.
As I explained in "Difference between Resident Set Size (RSS) and Java total committed memory (NMT) for a JVM running in Docker container", docker has some bugs (like issue 10824 and issue 15020) which prevent an accurate report of the memory consumed by a Java process within a Docker container.
That is why a plugin like signalfx/docker-collectd-plugin mentions (two weeks ago) in its PR -- Pull Request -- 35 to "deduct the cache figure from the memory usage percentage metric":
Currently the calculation for memory usage of a container/cgroup being returned to SignalFX includes the Linux page cache.
This is generally considered to be incorrect, and may lead people to chase phantom memory leaks in their application.
For a demonstration on why the current calculation is incorrect, you can run the following to see how I/O usage influences the overall memory usage in a cgroup:
docker run --rm -ti alpine
cat /sys/fs/cgroup/memory/memory.stat
cat /sys/fs/cgroup/memory/memory.usage_in_bytes
dd if=/dev/zero of=/tmp/myfile bs=1M count=100
cat /sys/fs/cgroup/memory/memory.stat
cat /sys/fs/cgroup/memory/memory.usage_in_bytes
You should see that the usage_in_bytes value rises by 100MB just from creating a 100MB file. That file hasn't been loaded into anonymous memory by an application, but because it's now in the page cache, the container memory usage is appearing to be higher.
Deducting the cache figure in memory.stat from the usage_in_bytes shows that the genuine use of anonymous memory hasn't risen.
The signalFX metric now differs from what is seen when you run docker stats which uses the calculation I have here.
It seems like knowing the page cache use for a container could be useful (though I am struggling to think of when), but knowing it as part of an overall percentage usage of the cgroup isn't useful, since it then disguises your actual RSS memory use.
In a garbage collected application with a max heap size as large, or larger than the cgroup memory limit (e.g the -Xmx parameter for java, or .NET core in server mode), the tendency will be for the percentage to get close to 100% and then just hover there, assuming the runtime can see the cgroup memory limit properly.
If you are using the Smart Agent, I would recommend using the docker-container-stats monitor (to which I will make the same modification to exclude cache memory).
Yes, NiFi docker has memory issues, shoots up after a while & restarts on its own. On the other hand, the non-docker works absolutely fine.
Details:
Docker:
Run it with 3gb Heap size & immediately after the start up it consumes around 2gb. Run some processors, the machine's fan runs heavily & it restarts after a while.
Non-Docker:
Run it with 3gb Heap size & it takes 900mb & runs smoothly. (jconsole)
I have a C program running in an alipine docker container. The image size is 10M on both OSX and on ubuntu.
On OSX, when I run this image, using the 'docker stats' I see it uses 1M RAM and so in the docker compose file I allocate a max of 5M within my swarm.
However, on Ubuntu 16.04.4 LTS the image is also 10M but when running it uses about 9M RAM, and I have to increase the max allocated memory in my compose file.
Why is there a such a difference in RAM usage between OSX and Ubuntu?
Even though we have different OSs, I would have thought once you are running inside a framework, then you would behave similarly on different machines. So I would have thought there should be comparable memory usage.
Update:
Thanks for the comments. So 'stats' may be inaccurate, and there are differences so best to baseline on linux. As an aside, but I think interesting, the reason for asking this question is to understand the 'under the hood' in order to tune my setup for a large number of deployed programs. Originally, when I tested I tried to allocate the smallest amount of maximum RAM on ubuntu, this resulted in a lot of disk thrashing something I didn't see or hear on my Macbook, (no hard disks!).
Some numbers which are completely my setup but also I think are interesting.
1000 docker containers, 1 C program each, 20M RAM MAX per container, Server load of 98, Server runs 4K processes in total, [1000 C programs total]
20 docker containers, 100 C programs each, 200M RAM MAX per container, Server load of 5 to 50, Server runs 2.3K processes in total, [2000 C programs total].
This all points at give your docker images a good amount of MAX RAM and it is nicer to your server to have fewer docker containers running.
I'm relatively new to docker. I'm trying to get the percent of cpu quota (actually) used by a container. Is there a default metric emitted by one of the endpoints or is it something that I will have to calculate with other metrics? Thanks!
docker stats --no-stream
CONTAINER ID NAME CPU % (rest of line truncated)
949e2a3724e6 practical_shannon 8.32% (truncated)
As mentioned in the comment from #asuresh4, above, docker stats appears to give the ACTUAL cpu utilization, not the configured values. The output here is from Docker version 17.12.1-ce, build 7390fc6
--no-stream means run stats once, not continuously as it normally does. As you might guess, you can also ask for stats on a single container (specify the container name or id).
In addition to CPU %, MEM USAGE / LIMIT, MEM %, NET I/O, and BLOCK I/O are also shown.
I'm trying to build a system which runs pieces of code in consistent conditions, and one way I imagine this being possible is to run the various programs in docker containers with the same layout, reserving the same amount of memory, etc. However, I can't seem to figure out how to keep CPU usage consistent.
The closest thing I can seem to find are "cpu shares," which, if I understand the documentation, limit cpu usage with respect to what other containers/other processes are running on the system, and what's available on the system. They do not seem to be capable of limiting the container to an absolute amount of cpu usage.
Ideally, I'd like to set up docker containers that would be limited to using a single cpu core. Is this at all possible?
If you use a newer version of Docker, you can use --cpuset-cpus="" in docker run to specify the CPU cores you want to allocate:
docker run --cpuset-cpus="0" [...]
If you use an older version of Docker (< 0.9), which uses LXC as the default execution environment, you can use --lxc-conf to configure the allocated CPU cores:
docker run --lxc-conf="lxc.cgroup.cpuset.cpus = 0" [...]
In both of those cases, only the first CPU core will be available to the docker container. Both of these options are documented in the docker help.
I've tried to provide a tutorial on container resource alloc.
https://gist.github.com/afolarin/15d12a476e40c173bf5f