the taosd memory increased from 8% to 25% after I executed several queries in taos client, why the memory did't go down after the I get query results? Is there any way to free up memory in TDengine database?
According to the official document https://www.taosdata.com/en/documentation/getting-started, you can use "reset query cache" to free up memory.
To clean the schema of local cached tables, execute command RESET QUERY CACHE
Related
I am using Apache Ignite 2.8.0.
i am running my server without persistence.
i have some records in my cache. it shows "totalAllocatedSize":18869600.
Now i cleared my cache, again it shows as the same "totalAllocatedSize":18869600.(i don't have any records in my cache)
why it shows like this, actually i don't have any records in cache, so it need to be show as 0. but it shows the previous value i got when some records in my cache..
why it's behave like this? or How i will get my actual memory used right now?
Like many databases, Apache Ignite will not de-allocate memory it has already allocated. You can see that you have space available by decreased fillFactor metric.
I have created about 5 reports in Microsoft PowerBI using a SQL database created in Microsoft Azure.
The database has more than 50 million row.
Recently my reports have stopped refreshing. In case they refresh, the refresh time is long and is running really slow.
here is a screenshot of the error i'm having enter image description here
I contacted yesterday Microsoft PowerBI to check if the issue is from the software itself. I showed them my database and my reports and they told me that the DTU in my SQL database is reaching a maximum of 100% which is slowing down the response of the database during the refresh and preventing it from performing well. Here is a screenshot of my database performance enter image description here. Please note that this picture is showing only the maximum of the DTUs, the average is giving a 50% value
I'm not an expert in Azure and i need to know if the DTUs can really effect the performance of calling the data from the database to Powerbi.
Yes, the DTUs will affect the query performance for Azure SQL database. Off course it will effect the performance of calling the data from the database to PowerBi.
Reference Database transaction units (DTUs):
A database transaction unit (DTU) represents a blended measure of CPU, memory, reads, and writes.
For a single database at a specific compute size within a service tier, Microsoft guarantees a certain level of resources for that database (independent of any other database in the Azure cloud). This guarantee provides a predictable level of performance. The amount of resources allocated for a database is calculated as a number of DTUs and is a bundled measure of compute, storage, and I/O resources.
The ratio among these resources is originally determined by an online transaction processing (OLTP) benchmark workload designed to be typical of real-world OLTP workloads. When your workload exceeds the amount of any of these resources, your throughput is throttled, resulting in slower performance and time-outs.
When the DTUs reaching a maximum of 100%, it means the performance of the database has reached the resource limits.
You need to scale the Azure SQL database service price tier or do a performance turning.
For more details, please see: Monitoring and performance tuning. Azure SQL Database provides tools and methods you can use to monitor usage easily, add or remove resources (such as CPU, memory, or I/O), troubleshoot potential problems, and make recommendations to improve the performance of a database.
Hope this helps.
For demo purposes, I am running Neo4j in a low memory environment -- A laptop with 4GB of RAM, 1644MB is use for video memory, leaving only 2452 MB available for use.. It's also running SQL Server, our WCF services, and our clients.. So there's little memory for Neo4j.
I'm running LOAD CSV cypher scripts via REST from a C# service. There are more than 20 scripts, and theyt work well in a server environment. I've written code to paginate, so that they run in smaller batches. I've reduced the batch size very low ( 25 csv rows ) and a given script may do 300 batches, but I continue to get "Java heap space" errors at some point.
I've tried configuring Neo4j with a relatively large heap space ( 640MB ) which is all the available RAM size plus setting the cache_type to none, and it gets much further before I get the java heap space error. What I don't understand is in that case, why does it grow that much? Also until I restart the neo4j service, I get these java heap space errors quickly. The batch size doesn't seem to impact how much memory is used appreciably.
However, after doing that, and I run the application with these settings, the query performance becomes very slow due to the cache settings.
I am running this on a Windows 7 laptop with 4G RAM -- using Neo4j 2.2.1 Community Edition.
Thoughts?
Perhaps you can share your LOAD CSV statement and the other queries you run.
I think you just run into this:
http://markhneedham.com/blog/2014/10/23/neo4j-cypher-avoiding-the-eager/
So PROFILE or EXPLAIN your queries and make it not to use that much intermediate state. We can help if you share your statements.
And you should use PERIODIC COMMIT 100.
Something like:
heap=512M
dbms.pagecache.memory=200M
keep_logical_logs=false
cache_type=none
http://console.neo4j.org runs neo4j in memory putting up to 50 instances in a single gigabyte of memory. So it should be doable.
TokuMx though has benefits, we are running into issues. Recently we migrated to this engine and in process our clean up scripts are useless. We have transient data that we used clean every night and then reclaim disk via db.repairDatabase . However that command is not supported by TokuMX and as a result we are not able to reclaim the disk.
Is there an alternate way ?
It sounds like partitioned collections are the right abstraction for your application. Normal collections will suffer from the accumulation of MVCC garbage if you have a pattern of deleting large swaths of old data. With partitioned collections, you can drop a partition and reclaim all the space instantaneously.
Getting started with latest DSE, trying to setup an initial DSE solr cluster and wanting to make sure basic capacity needs are met. In following docs I have done some initial capacity testing following directions here:
http://www.datastax.com/documentation/datastax_enterprise/4.5/datastax_enterprise/srch/srchCapazty.html
My test single node setup is on AWS, m3.xl, 80GB raid0 for the two 40GB ssd's, latest DSE installed
I have inserted a total of 6MM example records and run some solr searches which would be similar to that which production would be running.
Have the following numbers for my 6MM records:
6MM Records
7.6GB disk (Cassandra + solr)
2.56GB solr index size
96.2MB solr field cache(totalReadableMemSize)
25.57MB solr Heap
I am trying to plan out an initial starter cluster, would like to plan for around 250MM records stored and indexed to start. Read load will be pretty minimal in the early days, so not too worried about read throughput to start.
Following the capacity planning doc page and some numbers for 250MM from 6MM looks like base requirements for dataset would be:
250MM Records
106GB solr index size
317GB disk (Cassandra + solr)
4GB solr field cache(totalReadableMemSize)
1.1GB solr Heap
So some questions looking for some guidance on and if I am understanding docs correctly:
Should I be targeting ~360GB+ storage to be safe and not exceed 80% disk capacity on average as data set grows?
Should I use nodes that can allocate 6GB for solr + XGB for Cassandra? (ie: if entire solr index for 250MM is around 6GB for heap and field cache, and I partition across 3 nodes with replication)
With ~6GB for solr, how much should I try to dedicate to Cassandra proper?
Anything else to consider with planning (will be running on AWS)?
UPDATED (11/6) - Notes/suggestions from phact
With Cass+Solr running together, will target prescribed 14GB for each node for base operation, moving to targeted 30GB memory nodes on AWS, leaving 16GB for OS, solr index, solr field cache
I added the solr index size to numbers above, if suggested target to keep most/all index in memory seems I might need to target AT LEAST 8 nodes to start, with 30GB memory per node.
Seems like a good amount of extra overhead for solr nodes for targeting index in memory, might have to re-consider approach
DSE heap on a Solr node
The recommended heap size for a DSE node running solr is 14gb. This is because Solr and Cassandra actually run in the same JVM. You don't have to allocate memory for Solr separately.
AWS M3.xl
m3.xl's with 15gb ram will be a bit tight with a 14gb heap. However, if your workload is relatively light, you can probably get away with a 12gb heap on your solr nodes.
OS page cache
You do want to make sure that you are able to at least fit your Solr indexes in OS page cache (memory left over after subtracting out your heap -- assuming this is a dedicated box). Ideally, you will also have room for Cassandra to store some of your frequently read rows in page cache.
A quick and dirty way of figuring out how big your indexes are is checking the size of your index directory on your file system. Make sure to forecast out/extrapolate if you're expecting your data to grow. You can also check the index size for each of your cores as follows:
http://localhost:8983/solr/admin/cores?action=STATUS&memory=true
Note - each node should hold it's index in memory, not the entire cluster's index.
Storage
Yes you do want to ensure your disks are not over utilized or you may face issues during compaction. In theory--worse case scenario--tiered compaction could require up to 50% of your disk to be free. This is not common though, see more details here.