Due to InfluxDB will take my RAM depleted over time. Becasue of the reading speed in my case is not important, now i dont want to store data in memory, just on disk
Can i do that? If yes then how?
It is not possible to operate InfluxDB without using memory. Data in InfluxDB is not held in memory indefinitely and will be persisted to disk. A health InfluxDB instance will appear to consume all of the memory on a machine, but the OS will be able to free that memory if need be.
Related
When running redis INFO command, I'm getting the following:
used_memory_rss_human:2.69G
I understand this is the amount of memory redis freed but was not released back to the OS
How can I release this memory back to the OS?
According to Redis Docs:
Redis will not always free up (return) memory to the OS when keys are
removed. This is not something special about Redis, but it is how most
malloc() implementations work. For example, if you fill an instance
with 5GB worth of data, and then remove the equivalent of 2GB of data,
the Resident Set Size (also known as the RSS, which is the number of
memory pages consumed by the process) will probably still be around
5GB, even if Redis will claim that the user memory is around 3GB. This
happens because the underlying allocator can't easily release the
memory. For example often most of the removed keys were allocated in
the same pages as the other keys that still exist. The previous point
means that you need to provision memory based on your peak memory
usage. If your workload from time to time requires 10GB, even if most
of the times 5GB could do, you need to provision for 10GB.
I am running ignite 2.3 in embedded mode which means my default mode of data storage is off-heap. I have not enabled on heap cache.
The problem is when I run a large query, the data is stored ON the heap for a long time before it finally garbage collects. Is this expected? Why does it take so long for JVM to garbage collect this data.
I am concerned because as a result of my query, and the data it occupies on heap this will affect my application performance.
My CacheConfiguration are as follows:
data region max size: 12288000000L
CacheMode.LOCAL
indexedType
Ignite always stores data off-heap, on-heap caching is a feature that gives you ability to use Java heap as a cache for off-heap memory and configure eviction policies specific to this cache.
So, the data that you observe on Java heap with onHeapCacheEnabled=false is not Ignite cache data in your case, but a Java application memory footprint which is absolutely expected to exist. If you are experiencing application peformance issues that are connected with GC, check Preparing for Production and namely Garbage Collection Tuning sections of Ignite documentation for tuning tips or ask any specific questions here.
Regards.
This is a rather simple question, but I haven't been able to pinpoint a clear answer in my searching.
If I have an NSArray, and add fifty 1MB UIImages to it, where does that 50MB get deducted from? Will the app be using 50MB more memory? Will it simply store it on the disk?
The same goes for Core Data where instead of using a persistent store I store it in memory. Would the size of the Core Data store take up exactly that much memory/RAM or would it live on the disk and be wiped when the app finishes executing?
I'm concerned whether or not I should be storing several dozen megabytes in UIImages in an NSArray, or if I should be using NSCache (I'd rather not as I'd prefer to never lose any of the images).
If I have an NSArray, and add fifty 1MB UIImages to it, where does
that 50MB get deducted from? Will the app be using 50MB more memory?
Yes.
Will it simply store it on the disk?
No. Arrays are stored in memory.
The same goes for Core Data where instead of using a persistent store
I store it in memory. Would the size of the Core Data store take up
exactly that much memory/RAM or would it live on the disk and be wiped
when the app finishes executing?
Yes, if you tell Core Data to story everything in memory, that's exactly what it'll do.
The line between "memory" and "disk" can get a little fuzzy if you consider that virtual memory systems can swap pages of real memory out to disk and read them back when they're needed. That's not an issue for iOS, however, as iOS doesn't provide a VM backing store and writeable memory is never swapped out.
I'm concerned whether or not I should be storing several dozen
megabytes in UIImages in an NSArray, or if I should be using NSCache
Those aren't your only options, of course. You could store the images in files and read them in as needed. You should be thoughtful about the way your app uses both memory and disk space, of course, but you also need to consider network use, battery use, and performance. Storing data on disk is often preferable to downloading it again because downloading takes time, may impact the user's data plan, and uses a lot more energy than just reading data from secondary storage.
We are trying to Integrate SQLite in our Application and are trying to populate as a Cache. We are planning to use it as a In Memory Database. Using it for the first time. Our Application is C++ based.
Our Application interacts with the Master Database to fetch data and performs numerous operations. These Operations are generally concerned with one Table which is quite huge in size.
We replicated this Table in SQLite and following are the observations:
Number of Fields: 60
Number of Records: 1,00,000
As the data population starts, the memory of the Application, shoots up drastically to ~1.4 GB from 120MB. At this time our application is in idle state and not doing any major operations. But normally, once the Operations start, the Memory Utilization shoots up. Now with SQLite as in Memory DB and this high memory usage, we don’t think we will be able to support these many records.
When I create the DB on Disk, the DB size sums to ~40MB. But still the Memory Usage of the Application remains very high.
Q. Is there a reason for this high usage. All buffers have been cleared and as said before the DB is not in memory?
Any help would be deeply appreciated.
Thanks and Regards
Sachin
You can use the vacuum command to free up memory by reducing the size of sqlite database.
If you are doing a lot of insert update operations then the db size may increase. You can use vaccum command to free up space.
SQLite uses memory for things other than the data itself. It holds not only the data, but also the connections, prepared statements, query cache, query results, etc. You can read more on SQLite Memory Allocation and tweak it. Make sure you are properly destroying your objects too (sqlite3_finalize(), etc.).
It is possible for an operating system to determine whether a page of memory is in DRAM or in swap; for example, simply try to access it and if a page fault occurs, it wasn't.
However, is the same thing possible with CPU cache?
Is there any efficient way to tell whether a given memory location has been loaded into a cache line, or to know when it does so?
In general, I don't think this is possible. It works for DRAM and the pagefile since that is an OS managed resource, cache is managed by the CPU itself.
The OS could do a tight timing loop of a memory read and try to see if it completes fast enough to be in the cache or if it had to go out to main memory - this would be very error prone.
On multi-core/multi-proc systems, there are cache coherency protocols that are used between processors to determine when to they need to invalidate each other's caches, I suppose you could have a custom device that would snoop this protocol that the OS would query.
What are you trying to do? If you want to force something into memory, current x86 processors support prefetching memory into the cache in a non-blocking way, for instance with Visual C++ you could use _mm_prefetch to fetch a line into the cache.
EDIT:
I haven't done this myself, so use at your own risk. To determine cache misses for profiling, you may be able to use some architecture-specific registers. http://download.intel.com/design/processor/manuals/253669.pdf, Appendix A gives "Performance Tuning Events". This can't be used to determine if an individual address is in the cache or when it is loaded in the cache, but can be used for overall stats. I believe this is what vTune (a phenomenal profiler for this level) uses.
If you try to determine this yourself then the very act of running your program could invalidate the relevant cache lines, hence rendering your measurements useless.
This is one of those cases that mirrors the scientific principle that you cannot measure something without affecting that which you are measuring.
X86
dont know how to tell if address IS in cache
BUT here is how to tell if address WAS in cache
rdtsc
save timestamp
mov eax,address
rdtsc read timestamp counter
calculate timestamp difference
if < threshold then was in cache
threshold has to be determined from documentation or empirically
some machines have cache hit/miss counters which would serve equally well