I am using the official elasticsearch docker image. Since ES requires a certain level of memory mapped regions, (as documented), I increased it using
docker-machine ssh <NAME> sudo sysctl -w vm.max_map_count=262144
I also read here that the memory allocated should be around 50% of the total system memory.
I am confused about how these two play together. How does allocating more memory mapped regions affect the RAM allocated. Is it the part of the RAM, or is it taken above the RAM allocation for elasticsearch?
To sum it up very shortly, the heap is used by Elasticsearch only and Lucene will use the rest of the memory to map index files directly into memory for blazing fast access.
That's the main reason why the best practice is to allocate half the memory to the ES heap to let the remaining half to Lucene. However, there's also another best practice to not allocate more than 32-ish GB of RAM to the ES heap (and sometimes it's even less than 30B).
So, if you have a machine with 128GB of RAM, you won't allocate 64GB to ES but still a maximum 32-ish GB and Lucene will happily gobble up all the remaining 96GB of memory to map its index files.
Tuning the memory settings is a savant mix of giving enough memory (but not too much) to ES and making sure Lucene can have a blast by using as much as the remaining memory as possible.
Related
As the image shows that, as the memory capacity increases the accessing time is also increasing.
Does it make sense that, accessing time is dependent on the memory capacity..???
No. The images show that technologies with lower cost in $ / GB are slower. Within a certain level (tier of the memory hierarchy), performance is not dependent on size. You can build systems with wider busses and so on to get more bandwidth out of a certain tier, but it's not inherently slower to have more.
Having more disks or larger disks doesn't make disk access slower, they're close to constant latency determined by the nature of the technology (rotating platter).
In fact, larger-capacity disks tend to have better bandwidth once they do seek to the right place, because more bits per second are flying under the read / write heads. And with multiple disks you can run RAID to utilize multiple disks in parallel.
Similarly for RAM, having multiple channels of RAM on a big many-core Xeon increases aggregate bandwidth. (But unfortunately hurts latency due to a more complicated interconnect vs. simpler quad-core "client" CPUs: Why is Skylake so much better than Broadwell-E for single-threaded memory throughput?) But that's a sort of secondary effect, and just using RAM with more bits per DIMM doesn't change latency or bandwidth, assuming you use the same number of DIMMs in the same system.
What do I get by running multiple nodes on a single host? I am not getting availability, because if the host is down, the whole cluster goes with it. Does it make sense regarding performance? Doesn't one instance of ES take as many resources from the host as it needs?
Generally no, but if you have machines with ridiculous amounts of CPU and memory, you might want that to properly utilize the available resources. Avoiding big heaps with Elasticsearch is a good thing generally since garbage collection on bigger heaps can become a problem and in any case above 32 GB you lose the benefit of pointer compression. Mostly you should not need big heaps with ES. Most of the memory that ES uses is through memory mapped files, which relies on the OS cache. So just because you aren't assigning memory to the heap doesn't mean it is not being used: more memory available for caching means you'll be able to handle bigger shards or more shards.
So if you run more nodes, that advantage goes away and you waste memory on redundant heaps, and you'll have nodes competing for resources. Mostly, you should base these decisions on actual memory, cache, and cpu usage of course.
It depends on your host and how you configure your nodes.
For example, Elastic recommends allocating up to 32GB of RAM (because of how Java compresses pointers) to elasticsearch and have another 32GB for the operating system (mostly for disk caching).
Assuming you have more than 64GB of ram on your host, let's say 128, it makes sense to have two nodes running on the same machine, having both configured to 32GB ram each and leaving another 64 for the operating system.
The following is the structure of RAM for the entire Hack Computer in Nand2Tetris:
Putting aside virtual memory, is this a good simplified model for how the entire RAM is set up on x86 computers? Is RAM really just made of clusters of memory regions each with their own stack, heap and instruction memory, stacked on top of each other in RAM?
Basically, is RAM just a collection of independent and separate memory regions of each process/program running? Or, does RAM consist of variables scattered randomly from different programs?
Hugely over-simplified, processes on a machine with Virtual Memory could all think they have a memory map similar to that of the Hack Virtual Machine (note: Virtual Memory != Virtual Machine).
However, individual chunks of each process' memory map might be mapped to some arbitrary physical memory, shuffled off to a swap file, not allocated until actually needed, shared with other processes, and more. And those chunks that are in RAM might be anywhere (and might move).
You may find this article to be a good starting point to understanding Virtual Memory: https://en.wikipedia.org/wiki/Virtual_memory
I typically use about 5GB of RAM on my workstation at work. I normally have to run a few instances of matlab at a time, each running simulink simulations. These use a total of about 4-6GB RAM. When these are active, windows dumps memory in RAM to the page file to free space for matlab.
The problem is when the simulations are over, 2-3GB stays in the page file and slows the systems DRAMATICALLY. This computer has AWFUL disk read and write performance.
Is there a way I can move the paged memory back over to ram to avoid this performance hit?
Right now I am required to restart my computer when I am done running the simulations to speed it up again.
I have 8GB RAM with a 12GB page file.
Check out
Is it possible to unpage all memory in Windows?
The answer given by #KerrekSB seems to include some code for doing it. But the long and short of it is that you need to walk the list of processes, then walk the list of memory allocations for those processes, reading as you go.
I want to run multiple containers on a single host by providing limits on CPU & Memory. If my host has 1024 cpu shares & I assign them as 512 & 512 to two containers, it means that the first container can take as much as 1024 if second container is not using any cpu. But if both of them are using cpu, then both get limited to 512.
Is it also true for memory usage? Or somehow can I set it that way?
Here is the scenario:
I have 1024 Mb of RAM available for containers and I have two containers, I want each one to take 512 Mb of RAM but should be able to extend to more than 512 if other container is not using it. How is it possible?
In the case of memory you provide to Docker a fixed amount of memory (and swap) in bytes, kilobytes, megabytes,..., and that amount will limit the memory that container can allocate, no matter if the host has memory free or if it is being used by other process.
When limiting the memory it's important taking care of how Docker (or cgroups) limit the memory and swap of the container. From Docker v1.5 (and fixed in v1.6) Docker lets limit the memory and swap independently. Check Docker documentation to more details about this.