I have two systems with different Xeon processors and different amounts of RAM. Both run on Ubuntu 16 and have the same docker version. My applications are dockerized.
I ran the same docker image on both the systems and the amount of memory they consumed on both the systems are 610 MB and 814 MB.
I'm trying to figure out why this difference occurs. Does having a faster CPU reduce the usage of memory? and if so then why does it take less memory?
Related
I'm running a docker environment with roundabout 25 containers for my private use. My system has 32GB Ram.
Especially RStudio Server and JupyterLab often need a lot of memory.
So I limited the memory to for both container at 26 GB.
This works good as long not both application storing dataframes in memory. If now RServer stores some GB and Jupyter also is filling memory to the limit my system crashes.
Is there any way to configure that these two container together are allowed to use 26GB ram max.
Or a relative limit like Jupyter is allowed to use 90% of free memory.
As I'm working now with large datasets it can happen all the time that (because I forget to close a kernel or something else) the memory can increase to the limit and I want just the container to crash and not the whole system.
And I don't want lower the limit for Jupyter further as the biggest dataset on its own need 15 GB of memory.
Any ideas?
On the Google Cloud Platform (GCP), I have the following specs:
Machine type: n1-standard-8 (8 vCPUs, 30 GB memory)
CPU platform: Intel Haswell
I am using Jupyter notebook to fit an SVM to large amounts of NLP data. This process is very slow, and according to the GCP I am only utilizing around 0.12% of CPUs
How do I increase CPU utilization?
As DazWilkin mentioned actually, you're using 12% (12/100). This corresponds to one vCPU. This is because -- IIRC -- Jupyter is a Python app and Python's single-threaded so you're stuck using one core. You could reduce the number of cores (the OS will use multiple cores, of course) to save yourself some money but you'll need to evaluate alternatives to use more cores.
I want to run multiple containers on a single host by providing limits on CPU & Memory. If my host has 1024 cpu shares & I assign them as 512 & 512 to two containers, it means that the first container can take as much as 1024 if second container is not using any cpu. But if both of them are using cpu, then both get limited to 512.
Is it also true for memory usage? Or somehow can I set it that way?
Here is the scenario:
I have 1024 Mb of RAM available for containers and I have two containers, I want each one to take 512 Mb of RAM but should be able to extend to more than 512 if other container is not using it. How is it possible?
In the case of memory you provide to Docker a fixed amount of memory (and swap) in bytes, kilobytes, megabytes,..., and that amount will limit the memory that container can allocate, no matter if the host has memory free or if it is being used by other process.
When limiting the memory it's important taking care of how Docker (or cgroups) limit the memory and swap of the container. From Docker v1.5 (and fixed in v1.6) Docker lets limit the memory and swap independently. Check Docker documentation to more details about this.
I have a high end machine with 32 GB and I want to effectively utilize the available RAM for Java Process. Practically, I have seen I can run at max 3 JBoss instances (or Java Process) with Max 3GB Heap Size in one box still I have 20GB free space not utilized. Is it possible to create Java Program that can effectively directly work with RAM like C++, not relying on Java Memory Model for creating objects on JVM and also not relying on GC to reclaim memory. I mean Java Program directly working with RAM
I went through documentation of Passenger to find out how many application instances it can run with respect to hardware configuration. Documentation only talks about RAM
The optimal value depends on your system’s hardware and the server’s average load. You should experiment with different values. But generally speaking, the value should be at least equal to the number of CPUs (or CPU cores) that you have. If your system has 2 GB of RAM, then we recommend a value of 30. If your system is a Virtual Private Server (VPS) and has about 256 MB RAM, and is also running other services such as MySQL, then we recommend a value of 2.
It says minimum value can be number of CPU/CPU Cores we have. I have a VPS with one VCPU & 1GB RAM & my service provider has an option to just upgrade the RAM. I'm wondering how far I can just keep upgrading only RAM? How important it is to upgrade number of CPUs?
Quick Answer
Depends on what resources are the bottleneck for your app.
Long answer
You'll need to factor in a few things:
How much CPU time does your app need?
How much RAM does any given instance of your app use at peak load?
Does your app spend a lot of time doing IO intensive tasks? (ie: db and file reads/writes, network communication)
There can be other things to factor in, but your bottlenecks will probably be one of the above. If RAM is your main bottleneck, by all means use your newly available RAM. However, if it turns out that your app is being slowed down by CPU availability or flooded IO, no amount of RAM is going to speed things up.
On the topic of CPU cores; my understanding is that the main Apache process that runs Passenger is a single threaded process. Apache spawns new threads to handle concurrency on an as-needed basis. Each additional CPU core theoretically allows you to spawn x*n threads, where x is the number of threads you can optimally run under a single CPU core and n is the number of CPU cores available to Apache.
Disclaimer: I'm not very well read on Passenger internals; though this logic usually holds true for other kinds of Apache configurations.