I check the performance monitoring metrics on DolphinDB’s web interface after scheduled jobs are finished, but it shows that the node still uses a lot of memory:
I executed pnodeRun(objs) and pnodeRun(objs{true}) to release all variables in the cluster. Both returned empty, meaning there is no variable taking up memory. I also used pnodeRun(clearAllCache) to clear cached data, but the memory usage stayed unchanged. Is there another way to troubleshoot such issue?
If the memory usage of a node is close to maxMemSize, release all cached data on the node via clearAllCache(). If memory usage is still very high, check the following items:
Check if there are sessions consuming too much memory via
getSessionMemoryStat() and identify the corresponding users. Execute
objs(true) to identify the variables (including the variables shared
by other sessions) occupying large amount of memory in the current
session.
Check if there are background jobs or interactive jobs running longer
than expected via getRecentJobs() and getConsoleJobs(). The memory
usage of these jobs is not reflected in the result of
getSessionMemoryStat().
Check if the streaming data is consuming too much memory via
getStreamingStat().
See DolphinDB Memory Management for more information.
Related
I am running an Erlang application that often writes millions of records to the mnesia table for making a scheduler. When the time is due, the records get executed and removed from the table. The table is configured with {type, disk_copies}, {type, ordered_set}. I use transaction operations for writing and dirty operations for deleting records.
I have an experiment that writes 2 million records and then deletes all of them: the RAM memory was not reclaimed after it finished. There is a spike that twice increases the memory when I start to delete those records. For example, the beam memory starts as 75MB, and becomes after the experiment 410MB. I've used erlang:memory() to inspect the memory before and after, found that the memory was eaten by the process_used and binary but actually, I did not have any action with binary. If I use erlang:garbage_collect(Pid) for all running processes, the memory gets reclaimed, leaving 180MB.
Any suggestions for troubleshooting this issue would be highly appreciated. Thank you so much.
Answer from Rickard Green from Elrang OTP:
The above does not indicate a bug.
A process is not garbage collected unless it reaches certain limits, for example, it needs to allocate heap data and there is no free heap available. If a process stops executing, it does not matter how long time passes, it won't automatically garbage collect by itself unless it reaches one of these limits. A garbage collection can be forced by calling erlang:garbage_collect() though.
A process that has had a lot of live data (and by this have grown large) but at the time of the garbage collection has no live data wont shrink down to its original size immediately. It will instead get a relatively large heap. The heap space is free for usage by the process, but it is allocated from the system's point of view. The relatively large heap is selected in order to avoid triggering garbage collections unnecessarily frequent.
Not only your processes are effected when you execute. Also other processes might build up heap in order to serve your processes.
If you look at memory consumption via top or similar, it is also expected that memory usage will have increased after execution even if you are able to garbage collect every process down into its initial size. This due to memory allocators that place memory blocks into larger chunks of memory which cannot be removed until the whole memory chunk is free. More or less every memory allocation system that exist will have this characteristic.
I am running ignite 2.3 in embedded mode which means my default mode of data storage is off-heap. I have not enabled on heap cache.
The problem is when I run a large query, the data is stored ON the heap for a long time before it finally garbage collects. Is this expected? Why does it take so long for JVM to garbage collect this data.
I am concerned because as a result of my query, and the data it occupies on heap this will affect my application performance.
My CacheConfiguration are as follows:
data region max size: 12288000000L
CacheMode.LOCAL
indexedType
Ignite always stores data off-heap, on-heap caching is a feature that gives you ability to use Java heap as a cache for off-heap memory and configure eviction policies specific to this cache.
So, the data that you observe on Java heap with onHeapCacheEnabled=false is not Ignite cache data in your case, but a Java application memory footprint which is absolutely expected to exist. If you are experiencing application peformance issues that are connected with GC, check Preparing for Production and namely Garbage Collection Tuning sections of Ignite documentation for tuning tips or ask any specific questions here.
Regards.
I used ./bin/spark-shell to run some experiments and find out the following facts. When running jobs (transformation + action), I notice the memory usage in the top. For example, for 5G text file, I did a simple filter() and count(). After the job is done, there are 7g marked as res in the top window. I have 100g memory and set the executor memory to be 50g.
Is there anyone know what that 7g is?
Since Spark runs on the JVM, just because Spark may no longer have any references to some memory doesn't mean that that memory will be freed. Even if a garbage collection has been triggered, the JVM may not release the memory back to the OS (this is controlled by -XX:MaxHeapFreeRatio among other things).
When the cpu is executing a program, does it move all data through the memory pipeline? Then any piece of data would be moved from ram->cache->registers so all data that's executed goes in the cpu registers at some point. Or does it somehow select the code it puts in those faster memory types, or can you as a programmer select specific code you want to keep in, for example, the cache for optimization?
The answer to this question is an entire course in itself! A very brief summary of what (usually) happens is that:
You, the programmer, specify what goes in RAM. Well, the compiler does it on your behalf, but you're in control of this by how you declare your variables.
Whenever your code accesses a variable the CPU's MMU will check if the value is in the cache and if it is not, then it will fetch the 'line' that contains the variable from RAM into the cache. Some CPU instruction sets may allow you to prevent it from doing so (causing a stall) for specific low-frequecy operations, but it requires very low-level code to do so. When you update a value, the MMU will perform a 'cache flush' operation, committing the cached memory to RAM. Again, you can affect how and when this happens by low-level code. It will also depend on the MMU configuration such as whether the cache is write-through, etc.
If you are going to do any kind of operation on the value that will require it being used by an ALU (arithmetic Logic Unit) or similar, then it will be loaded into an appropriate register from the cache. Which register will depend on the instruction the compiler generated.
Some CPUs support Dynamic Memory Access (DMA), which provides a shortcut for operations that do not really require the CPU to be involved. These include memory-to-memory copies and the transfer of data between memory and memory-mapped peripheral control blocks (such as UARTs and other I/O blocks). These will cause data to be moved, read or written in RAM without actually affecting the CPU core at all.
At a higher level, some operating systems that support multiple processes will save the RAM allocated to the current process to the hard disk when the process is swapped out, and load it back in again from the disk when the process runs again. (This is why you may find 'Page Files' on your C: drive and the options to limit their size.) This allows all of the running processes to utilise most of the available RAM, even though they can't actually share it all simultaneously. Paging is yet another subject worthy of a course on its own. (Thanks to Leeor for mentioning this.)
We have some problems with Dart. It seems like after some period of time the garbage collector can't clear the memory in VM, so application hangs. Anyone with this issue? Are there any memory limits?
You should reuse your objects instead of creating new ones. You should use pool pattern:
http://en.wikipedia.org/wiki/Object_pool_pattern
Be careful about canvas and it's proper destruction.
Another GC performance papers:
http://blog.tojicode.com/2012/03/javascript-memory-optimization-and.html
http://qt-project.org/doc/qt-5/qtquick-performance.html
Are there any memory limits?
Yes. Dart apparently runs with a maximum sizes that can be configured at launch time:
How to run a dart program with big memory?
(The following applies to all garbage-collected languages ...)
If your application starts to run out of space (i.e. the heap is slowly filing with objects that the GC can't remove) then you may get into a nasty situation where the GC runs more and more frequently, and manages to reclaim less and less memory each time. Eventually you run out of memory, but before that happens the application gets really slow.
The solution is typically to do one or both of the following:
Find what is causing the memory to run out. It is typically not that you are allocating too many objects. Rather, the typical cause is that the unwanted objects are all still reachable ... via some data structure that your application has built.
Set the "quick death" tuning option for the GC .... if available. For example, Java garbage collectors can be configured to measure the time spent garbage collecting. (The GC overhead.) When the GC overhead exceeds a preset ratio, the Java virtual machine throws an OutOfMemoryError to "pull the plug".