I'm trying to find a time series database for the following scenario:
Some sensor on raspberry pi provides the realtime data.
Some application takes the data and pushes to the time series database.
If network is off (GSM modem ran out of money or rain or something else), store data locally.
Once network is available the data should be synchronised to the time series database in the cloud. So no missing data and no duplicates.
(Optionally) query database from Grafana
I'm looking for time series database that can handle 3. and 4. for me. Is there any?
I can start Prometheus in federated mode (Can I?) and keep one node on raspberry pi for initial ingestion and another node in the cloud for collecting the data. But that setup would instantly consume 64mb+ of memory for Prometheus node.
Take a look at vmagent. It can be installed at every device where metrics from local sensors must be collected (e.g. at the edge), and collect all these metrics via various popular data ingestion protocols. Then it can push the collected metrics to a centralized time series database such as VictoriaMetrics. Vmagent buffers the collected metrics on the local storage when the connection to a centralized database is unavailable, and pushes the buffered data to the database as soon as the connection is recovered. Vmagent works on Rasberry PI and on any device with ARM, ARM64 or AMD64 architecture.
See use cases for vmagent for more details.
Related
My team is looking for a solution (both in house or tools) for performance monitoring and operation management for 500 plus SSIS ETL loads ( with varied run frequencies- daily, weekly, monthly etc) and 100 plus data pipelines ( currently Hadoop is used as data lake storage layer but the plan is to migrate to data bricks hosted on AWS). Data pipes will increase as ML And AI needs evolve over time. The solution should be able to handle input from SSIS as well as from data pipelines. Primary goals for this solution are:
Generate a dashboard that shows data quality anomalies ( ETL source sent 100 but destination received only 90, For pipeline x -- avg data volume received is reduced by 90%).
Send alerts on failures. Can integrate with service now to create tickets for severe failure etc.
Allow the operation team to quickly figure out important performance metrics -- Execution run time, any slowness in a particular pipeline/bottlenecks.
Ability to customize the dashoards.
Right now we are thinking a SQL server based centralized logging table which will get metrics from ETL and pipelines and then expose this data to powerbi and create custom dashboards. Write some API to integrate with service now to create alerts. But this solution might be hard to scale.
Can someone suggest me some good Data monitoring tools which can serve our needs. My google search came up with these 3 tools :
Data Dog, Data fold, Dyna trace and ELK stack.
We are using influxdb at different industrial sites, where we log up to 10.000 values ranging from 1Hz to 1000Hz sample rate, from 3-5 different machines - resulting in something like 1GB data/hour.
The logging is handles by simple HTTP line-protocal calls to an Influxdb 1.8 server. Running on a xeon 2.5Ghz 10c 64Gb ram 6TB SSD raid 5 array.
Right now the values are stored in the same database with a measurement for each machine, with a retention policy of 20weeks with a shard duration of 1week.
The data is visualized through grafana mostly.
Many people query the database at once through multiple grafana dashboards - which can tend to be fairly slow when I retrieve large amounts of data. No cross measurement calculations are performed, it is only visual plots.
Will I get any read-speed benefits from doing multiple databases instead of a single database with multiple measurements?
When getting data from a database, do influx need to "open" files containing data from all measurements in order to find data from a specific measurement?
Actually I am developing system monitoring tools using different opens source such as InfluxDb, Telegraf, Grafana and SNMP(Simple Network Management Protocol).
I am enabling SNMP in the different linux or Redhat systems and fetching the systems statics through Telegraf and storing data into influxDB.
So here I am using telegraf as a agent master(collecting data from multiple systems) and SNMP as a agent and got CPUs, Ram/swap, Disk usage, Network packets etc through Mibs.
I am trying to monitor Average Disk Response time (wait+service time) as well.
I got UCD-DISKIO-MIB but its providing some row data.
So the question is how to calculate average response time using those data.
I would like to use InfluXDB wit my Raspberry/Openhab home automatisation.
I am just worried about db size/performance.
So my plan would be: log only 1 month on Pi, let it be cleaned automatically.
Clean I understand is easy with retention. (Automatically clear old data )
For long time anaylsis I want to collect all the data on a server.
Now question: How can I export the data on PI before retention into flatfile and afterwards import that data in a seperate InfluXDB on different server?
(Or even better: is there a way to do this in a sort of cluster mode?)
thanks a lot,
Chris
I use InfluxDB on a pi for sensor logs. I log now 4 records every 5 seconds for more than 3 months and performances on my pi are really good. I don't have now the file size but was not more than 10Mb
You can use InfluxDB in cluster mode but not sure it will answer yor question for data cleaning.
To exprt data, you can use InfluxDB API to get all series in the data base, then all data and flush that in a json file. You can use the API to load that file in another DB
I am using graphite server to capture my metrics data and bring down to graphs. I have 4 application servers which is load balancer setup. My aim is capture system data such as cpu usage, memory usage, disk load, etc., for all the 4 application servers. I setup an graphite environment in a separate server and i wanted to push the system data for all the applications servers to graphite and get it display as graphs. I don't know what needs to be done for feeding system data to graphite. My thinking was to install statsd in all application servers and feed the system data to graphite but looks like statsd does not support system data rather application data.
Can anyone help me to catch the right track. Thanks in advance.
Running collectd with a graphite agent would be an excellent start to gather the information your after.
There is an almost unlimited amount of ways to get your data into graphite.
You can find a list of tools that have known to work very well with graphite on the readthedocs.org page: http://graphite.readthedocs.org/en/0.9.10/tools.html
There is also an example script that gathers load average from the system in the carbon project: example-client.py