I have two databases in Influxdb:
base1
base2
And metrics like these:
base1.example.allRequests.all.percentiles99 807 1607947555
base2.example.allRequests.all.percentiles99 807 1607947555
I have solution with influx.conf where I'm sending data on different ports.
http://localhost:2003 for base1.....
http://localhost:2004 for base2.....
[[graphite]]
…
bind-address = ":2003”
database = "base-1”
…
[[graphite]]
…
bind-address = ":2004”
database = "base-2”
…
However, I think there is a better solution for this case where data can be sent based on measurement not just using ports. Can someone please help me or suggest how to it?
Related
Let us assume I have various MQTT clients who send data within some topic, for instance for temperature sensors tele/temp/%devicename%/SENSOR in a JSON Format, such as
{"Time":"2020-03-24T20:17:04","DS18S20":{"Temperature":22.8},"TempUnit":"C"}}
My basic telegraf.conf looks as following
# Influxdb Output
[[outputs.influxdb]]
database = "telegraf"
# sensors
[[inputs.mqtt_consumer]]
name_override = "sensor"
topics = ["tele/temp/+/SENSOR"]
data_format = "json"
My problem is now that I fail to do basic operations on that json data.
I do not want to save the host and the topic. How can I drop fields?
The topic contains the %devicename%. How can I add it as tag?
I cannot use json_query, since the %devicename% is there and there is only one field.
How can I rename the field %devicename%_Temperature to "temperature"?
In general, I would like to have an easy way to keep only the measurement of a multitude of sensors in the following format
timestamp | temperature | device
2020-03-24T20:17:04 | 22.8 | DS18S20
Thanks a lot!
If you don't want to save the topic in InfluxDB set the following as part of your inputs_mqtt_consumer:
topic_tag=""
Adding the topic as a tag name can be done using processors.regex
[[processors.regex]]
[[processors.regex.tags]]
key = "topic"
pattern = ".*/(.*)/.*"
replacement = "${1}"
result_key = "devicename"
In this case the 2nd section of the topic would become the devicename.
I have a postgres db with a table networks which includes a column network of type cidr. I want to queue all networks which contain an IP address I provide. I couldnt find how to do this.
Currently I pull all networks, and then use the include? method IPAddr class provides:
Network.all.each{|row| pp row if row.network.include?("10.176.0.5")}
You can use postgres' >>= operator (includes or equal) or >>(includes) that is defined for cidr type:
Network.where(['network >> ?', "10.176.0.5"])
It is possible to drop all databases in influx?
or drop only databases starting with a given prefix?
Something like:
DROP DATABASE somedb*.*
?
Thanks in advance,
It is possible with a short python script:
from influxdb import InfluxDBClient
client = InfluxDBClient(INFLUXDB_ADDRESS,
INFLUXDB_PORT,
INFLUXDB_USER,
INFLUXDB_PASSWORD,
None)
pattern='somedb'
databases = client.get_list_database()
for db in databases:
if db.get('name').startswith(pattern):
client.drop_database(db.get('name'))
In the website Introduction to Py2neo 2.0
http://py2neo.org/2.0/intro.html
why do we need Pushing & Pulling ?
also graph.cypher.begin() and commit(). as below
tx = graph.cypher.begin()
for name in ["Alice", "Bob", "Carol"]:
tx.append("CREATE (person:Person {name:{name}})
RETURN person", name=name)
alice, bob, carol = [result.one for result in tx.commit()]
friends = Path(alice, "KNOWS", bob, "KNOWS", carol)
graph.create(friends)`
I used a small py2neo program as below, and it also works ( at least I can see it on localhost:7474 ) ? please explain the two different methods, thanks
alice = Node("Person", name="Alice")
bob = Node("Person", name="Bob")
alice_knows_bob = Relationship(alice, "KNOWS", bob)
graph.create(alice_knows_bob)
Pushing and pulling are explicitly drawn out into methods to allow full control over network traffic. Earlier versions of the library sent and received changes to and from the server as they were made on the client. This was inefficient as separate HTTP requests were made for each individual property change etc. The current push/pull API is modelled to some extent on git by passing synchronisation control to the application developer.
I am attempting to sort records in a model which has 500k rows in it. When I initially attempted this procedure I had just 200 records and used the following code and pulled out records 1-5 to list those that were most popular:
#mostpopular = Product.find(:all, :order => 'click_count DESC')
However, now I have a far larger dataset, this grinds my computer to a halt and I am looking to try to complete the search in a more efficient manner.
I have tried adjusting the code to #mostpopular = Product.order('click_count DESC').limit(10) but this still take a LONG time to complete...
Is there a more efficient way to pull out the top 10 most popular records from a large dataset?
Thanks for your time
The answer is most likely not in rails, it's in your database.
Write the query to the log, so you can see what query is being done:
logger.debug Product.find(:all, :order => 'click_count DESC').limit(10).to_sql
Once you have the SQL in your hand, head over to your database's console and ask it to show you query plan and statistics for that query. You don't say what database you're using, but in postgresql, you'd use the EXPLAIN command. I'll be you see that a row scan (aka sequence scan) is being done.
You may find that click_count is missing an index, and adding it fixes your trouble.
You simply need to add an index to improve the speed of the query. Add the following to a migration:
add_index :products, :click_count
Then run rake db:migrate
First to set sort buffer size, as per your system or server configuration. Also for this edit contents of my.ini file in MySQL root dir:
[client]
#password = your_password
port = 3306
socket = /tmp/mysql.sock
# The MySQL server
[mysqld]
port = 3306
socket = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 16K
max_allowed_packet = 1M
table_open_cache = 4
sort_buffer_size = 64K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
thread_stack = 128K
Improve performance by setting sort_buffer_size refer this link http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/
it will improve your application performance.
Product.find_by_sql("SELECT * FROM products ORDER BY click_count DESC LIMIT 10")
See the efficiency or time consume by above query.