I'm using telegraf with influxdb, and in the telegraf I'm using the statsd_input plugin.
The statsd_input.conf:
[[inputs.statsd]]
## Address and port to host UDP listener on
service_address = ":8126"
## The following configuration options control when telegraf clears it's cache
## of previous values. If set to false, then telegraf will only clear it's
## cache when the daemon is restarted.
## Reset gauges every interval (default=true)
delete_gauges = true
## Reset counters every interval (default=true)
delete_counters = true
## Reset sets every interval (default=true)
delete_sets = true
## Reset timings & histograms every interval (default=true)
delete_timings = true
## Percentiles to calculate for timing & histogram stats
percentiles = [90]
## separator to use between elements of a statsd metric
metric_separator = "."
## Parses tags in the datadog statsd format
## http://docs.datadoghq.com/guides/dogstatsd/
parse_data_dog_tags = true
## Statsd data translation templates, more info can be read here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md#graphite
# templates = [
# "cpu.* measurement*"
# ]
## Number of UDP messages allowed to queue up, once filled,
## the statsd server will start dropping packets
allowed_pending_messages = 10000
## Number of timing/histogram values to track per-measurement in the
## calculation of percentiles. Raising this limit increases the accuracy
## of percentiles but also increases the memory usage and cpu time.
percentile_limit = 1000
I'm trying to set the time precision to seconds. I tried to accomplish this in the telegram.conf file, but it's written in the notes that the precision setting does not affect the statsd plugin:
## By default, precision will be set to the same timestamp order as the
## collection interval, with the maximum being 1s.
## Precision will NOT be used for service inputs, such as logparser and statsd.
## Valid values are "ns", "us" (or "µs"), "ms", "s".
precision = ""
I haven't seen a setting of the precision in the statsd_input.conf file.
What is the correct way to accomplish this?
Unfortunately this isn't supported by influxdb. The workaround is to send the information using socket_listener with the correct timestamp.
This information is per the issues logged against influxdb's GitHub
Related
I am using envoyproxy/ratelimit (along with Istio) to setup a global rate limiting in my k8s cluster for a given service. The rate limit is based on a header (in my case the username) so that each username is limited by the number of RPS. The following configuration was used to achieve this:
domain: ratelimit
descriptors:
- key: USERNAME
rate_limit:
unit: second
requests_per_unit: 100
shadow_mode: true
Also, I used a EnvoyFilter (Istio CRD) to define which header will used.
The resulting metric does not show a label for a specific user, just for the entire descriptor:
ratelimit_service_rate_limit_within_limit{app="ratelimit",domain="ratelimit",instance="xxx",job="kubernetes-pods",key1="USERNAME",kubernetes_namespace="xxx",kubernetes_pod_name="ratelimit-xxx",pod_template_hash="xxx",security_istio_io_tlsMode="istio",service_istio_io_canonical_name="ratelimit",service_istio_io_canonical_revision="latest"}
So my question is: how can I get the metrics for a specific username? Considering my configuration is applied to all of them and not for a specific value.
Thanks to this PR you can now add a detailed_metric parameter to enable this behavior, as shown in this example.
After getting some metric buffer overflow warning messages, I am trying to understand better how the fundamental agent parameters interval, metric_batch_size, metric_buffer_limit and flush_interval impact each other.
Looking the specification, these 3 parameters are defined as:
interval : Default data collection interval for all inputs
metric_batch_size : Telegraf will send metrics to output in batch of at most metric_batch_size metrics.
metric_buffer_limit : Telegraf will cache metric_buffer_limit metrics for each output, and will flush this buffer on a successful write. This should be a multiple of metric_batch_size and could not be less than 2 times metric_batch_size .
flush_interval : Default data flushing interval for all outputs. You should not set this below interval . Maximum flush_interval will be flush_interval + flush_jitter
What I understand is that
Data is only written to the output at each flush_interval .
If not all the data can be written, it uses the buffer to store.
What I am not sure to understand is when will the buffered data be written to the output again ? Will it be at next flush_interval occurence ?
Thanks in advance for your help on this !
Answered and solved thanks to Jay_Clifford
Yes. Data will be sent at the next flush interval.
(cf. Telegraf community post)
I'm using ubuntu server and configure the odoo project. it has 8GB of ram and available memory is arround 6GB so i need to increase the odoo default memory. So please let me know how to increase?
Have you tried playing with some of Odoo's Advanced and Multiprocessing options?
odoo.py --help
Advanced options:
--osv-memory-count-limit=OSV_MEMORY_COUNT_LIMIT
Force a limit on the maximum number of records kept in
the virtual osv_memory tables. The default is False,
which means no count-based limit.
--osv-memory-age-limit=OSV_MEMORY_AGE_LIMIT
Force a limit on the maximum age of records kept in
the virtual osv_memory tables. This is a decimal value
expressed in hours, and the default is 1 hour.
--max-cron-threads=MAX_CRON_THREADS
Maximum number of threads processing concurrently cron
jobs (default 2).
Multiprocessing options:
--workers=WORKERS Specify the number of workers, 0 disable prefork mode.
--limit-memory-soft=LIMIT_MEMORY_SOFT
Maximum allowed virtual memory per worker, when
reached the worker be reset after the current request
(default 671088640 aka 640MB).
--limit-memory-hard=LIMIT_MEMORY_HARD
Maximum allowed virtual memory per worker, when
reached, any memory allocation will fail (default
805306368 aka 768MB).
--limit-time-cpu=LIMIT_TIME_CPU
Maximum allowed CPU time per request (default 60).
--limit-time-real=LIMIT_TIME_REAL
Maximum allowed Real time per request (default 120).
--limit-request=LIMIT_REQUEST
Maximum number of request to be processed per worker
(default 8192).
Also if you are using WSGI or something similar to run Odoo, these may also need some tuning.
I'm using Graphite+Statsd (with Python client) to collect custom metrics from a webapp: a counter for successful transactions. Let's say the counter is stats.transactions.count, that also has a rate/per/second metric available at stats.transactions.rate.
I've also setup Seyren as a monitor+alert system and successfully pulled metrics from Graphite. Now I want to setup an alert in Seyren if the number of successful transactions in the last 60 minutes is less than a certain minimum.
Which metric and Graphite function should I use? I tried with summarize(metric, '1h') but this gives me an alert each hour when Graphite starts aggregating the metric for the starting hour.
Note that Seyren also allows to specify Graphite's from and until parameters, if this helps.
I contributed the Seyren code to support from/until in order to handle this exact situation.
The following configuration should raise a warning if the count for the last hour drops below 50, and an error if it drops below 25.
Target: summarize(nonNegativeDerivative(stats.transactions.count),"1h","sum",true)
From: -1h
To: [blank]
Warn: 50 (soft minimum)
Error: 25 (hard minimum)
Note this will run every minute, so the "last hour" is a sliding scale. Also note that the third boolean parameter true for the summarize function is telling it to align its 1h bucket to the From, meaning you get a full 1-hour bucket starting 1 hour ago, rather than accidentally getting a half bucket. (Newer versions of Graphite may do this automatically.)
Your mileage may vary. I have had problems with this approach when the counter gets set back to 0 on a server restart. But in my case I'm using dropwizard metrics + graphite, not statsd + graphite, so you may not have that problem.
Please let me know if this approach works for you!
I have been playing around with heap size for JVM and the file store cache size in Neo4j. It seems like setting the memory mapped buffer to be handled by the OS does not have any effect on the system.
I tried setting the JVM heap quite large with a tiny cache and it was exactly as fast as a if the cache was large.
So my question is: How can I configure the system to allow me to control the cache? Is this an issue with the batching as it says that this uses the JVM heap?
I am using the following python script to fill up the database
neo4j.GraphDatabaseService("http://localhost:7474/db/data/")
f = open('indexslowdown_exp.txt','w')
f.write("Properties\t,\tSpeed\n")
total_time = timedelta(0)
name = 0
for y in range(0,1000):
batch = neo4j.WriteBatch(graph_db)
for x in range(0,100):
batch.create({"name":name})
name += 1
for x in range(0,100):
rand_node_A = random.randint(0,name-1)
rand_node_B = random.randint(0,name-1)
batch.append_cypher("START n=node("+str(rand_node_A)+"), m=node("+str(rand_node_B)+") CREATE (n)-[r:CONNECTED]->(m)")
start_time = datetime.now()
batch.submit()
end_time = datetime.now()
total_time = (end_time-start_time)
f.write(str(name)+" , "+str((total_time)/200)+"\n")
print "Inserting nodes: " + str(total_time)
f.close()
Neo4j.properties file:
use_memory_mapped_buffers=true
/# Default values for the low-level graph engine
neostore.nodestore.db.mapped_memory=1k
neostore.relationshipstore.db.mapped_memory=1k
neostore.propertystore.db.mapped_memory=2k
neostore.propertystore.db.strings.mapped_memory=1k
neostore.propertystore.db.arrays.mapped_memory=1k
# Enable this to be able to upgrade a store from an older version
#allow_store_upgrade=true
# Enable this to specify a parser other than the default one.
#cypher_parser_version=2.0
# Keep logical logs, helps debugging but uses more disk space, enabled for
# legacy reasons To limit space needed to store historical logs use values such
# as: "7 days" or "100M size" instead of "true"
keep_logical_logs=true
# Autoindexing
# Enable auto-indexing for nodes, default is false
#node_auto_indexing=true
# The node property keys to be auto-indexed, if enabled
#node_keys_indexable=name,age
# Enable auto-indexing for relationships, default is false
#relationship_auto_indexing=true
# The relationship property keys to be auto-indexed, if enabled
#relationship_keys_indexable=name,age
neo4j-wrapper:
wrapper.java.additional=-Dorg.neo4j.server.properties=conf/neo4j-server.properties
wrapper.java.additional=-Djava.util.logging.config.file=conf/logging.properties
wrapper.java.additional=-Dlog4j.configuration=file:conf/log4j.properties
#********************************************************************
# JVM Parameters
#********************************************************************
wrapper.java.additional=-XX:+UseConcMarkSweepGC
wrapper.java.additional=-XX:+CMSClassUnloadingEnabled
# Uncomment the following lines to enable garbage collection logging
wrapper.java.additional=-Xloggc:data/log/neo4j-gc.log
wrapper.java.additional=-XX:+PrintGCDetails
wrapper.java.additional=-XX:+PrintGCDateStamps
wrapper.java.additional=-XX:+PrintGCApplicationStoppedTime
wrapper.java.additional=-XX:+PrintTenuringDistribution
# Initial Java Heap Size (in MB)
wrapper.java.initmemory=200
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=400
#********************************************************************
# Wrapper settings
#********************************************************************
# Override default pidfile and lockfile
#wrapper.pidfile=../data/neo4j-server.pid
#wrapper.lockfile=../data/neo4j-server.lck
#********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
# using this configuration file has been installed as a service.
# Please uninstall the service before modifying this section. The
# service can then be reinstalled.
# Name of the service
wrapper.name=neo4j
# User account to be used for linux installs. Will default to current
# user if not set.
wrapper.user=
What are you most concerned with? Batch insertion performance? If so, MMIO will be most effective if you can fit the entire graph into memory, so if you can estimate the number of nodes and relationships, you can come up with a rough calculation of the size you need for those two stores.
Also given that you look to only be inserting primitives, you can likely estimate the size for the property store that you need. If you're going to store strings and arrays (of a larger type), you can increase the MMIO settings for those two stores, as well, but if you don't need them, set them low.
Approx. size of node store: # of nodes * 14 bytes (if you're using the latest Neo4j; 9 bytes if Neo4j is < 2.0)
Approx. size of relationship store: # of rels * 33 bytes
Remember: There's a near 1:1 correspondence between the store sizes on disk and in memory for the filesystem cache.
As well, a larger JVM heap doesn't necessarily mean greater performance; in fact, the MMIO sizes (depending on your value for the setting use_memory_mapped_buffers) may lie outside of the JVM heap. Regardless, a large JVM heap can also introduce longer GC pauses and other GC-related issues, so don't make it too big.
HTH
For your queries:
Use Parameters!
Use Labels & Indexes in 2.0.x
Try not to use random in performance test loops
Make sure to use streaming