How can I get the size of the data returned from the database by a TFDConnection? i.e. Network traffic in bytes.
I want a driver agnostic solution as the app uses several data sources.
Related
I'm trying to find a time series database for the following scenario:
Some sensor on raspberry pi provides the realtime data.
Some application takes the data and pushes to the time series database.
If network is off (GSM modem ran out of money or rain or something else), store data locally.
Once network is available the data should be synchronised to the time series database in the cloud. So no missing data and no duplicates.
(Optionally) query database from Grafana
I'm looking for time series database that can handle 3. and 4. for me. Is there any?
I can start Prometheus in federated mode (Can I?) and keep one node on raspberry pi for initial ingestion and another node in the cloud for collecting the data. But that setup would instantly consume 64mb+ of memory for Prometheus node.
Take a look at vmagent. It can be installed at every device where metrics from local sensors must be collected (e.g. at the edge), and collect all these metrics via various popular data ingestion protocols. Then it can push the collected metrics to a centralized time series database such as VictoriaMetrics. Vmagent buffers the collected metrics on the local storage when the connection to a centralized database is unavailable, and pushes the buffered data to the database as soon as the connection is recovered. Vmagent works on Rasberry PI and on any device with ARM, ARM64 or AMD64 architecture.
See use cases for vmagent for more details.
Actually I am developing system monitoring tools using different opens source such as InfluxDb, Telegraf, Grafana and SNMP(Simple Network Management Protocol).
I am enabling SNMP in the different linux or Redhat systems and fetching the systems statics through Telegraf and storing data into influxDB.
So here I am using telegraf as a agent master(collecting data from multiple systems) and SNMP as a agent and got CPUs, Ram/swap, Disk usage, Network packets etc through Mibs.
I am trying to monitor Average Disk Response time (wait+service time) as well.
I got UCD-DISKIO-MIB but its providing some row data.
So the question is how to calculate average response time using those data.
I'm glossing over their documentation here :
http://www.rubydoc.info/github/github/statsd-ruby/Statsd
And there's methods for recording data, but I can't seem to find anything about retrieving recorded data. I'm adopting a projecting with an existing statsd addition. It's host is likely a defunct URL. Perhaps, is the host where those stats are recorded?
The statsd server implementations that Mircea links just take care of receiving, aggregating metrics and publishing them to a backend service. Etsy's statsd definition (bold is mine):
A network daemon that runs on the Node.js platform and listens for
statistics, like counters and timers, sent over UDP or TCP and sends
aggregates to one or more pluggable backend services (e.g.,
Graphite).
To retrieve the recorded data you have to query the backend. Check the list of available backends. The most common one is Graphite.
See also this question: How does StatsD store its data?
There are 2 parts to statsd: a client and a server.
What you're looking at is the client part. You will not see functionality related to retrieving the data as it's not there - it normally is on the server side.
Here is a list of statsd server implementations:
http://www.joemiller.me/2011/09/21/list-of-statsd-server-implementations/
Research and pick one that fits your needs.
Statsd originally started at etsy: https://github.com/etsy/statsd/wiki
I want to implement a demo application to listen data via TCP/IP.
Data Transmitter will transmit a series or ASCII char or a series of string all the time. It feeds data into TCP/IP address (eg. 127.0.0.1:22) This could be a GPS transmitter.
I want to implement a demo application for receiving data by clicking the start button and listening to the data via TCP/IP and display it accordingly.
Correct me if I am wrong, I don't think I can use Server/Client server for this purpose. I tried to create a client application with TIdTcpClient, it receives only one time data. I don't think Indy has a TCP listening component.
Thanks in advance.
If you wanna monitor network comunications between some device and some other program on your computer using of TIdTCPServer won't work. Why? Once Indy will read network data it will mark it as processed and delete it from network buffer. So that data probably won't even reach to the other program on your computer. Workaround for this is that you design your application to actually work similar as network bridge. Your application listens to the data on one port and then forwards that data on another port on which the other program is listening. But the main problem is that you have to make this to work both ways.
What you need is somekind of a component which is able to peek at the network data but don't interact with it. This is usually done on driver level.
Now if it is not abolutely necessary to have such functionality in your own software but you are only interested in getting the data I recomend you try Wireshark (http://www.wireshark.org/). Wireshark is a verry powerfull freware software which alows you to monitor all netwrok traffic on basically all protocols without causing any interuptions. In order for this software to work it instals special driver which serves for intercepting the network data.
Maybe you would want to use same driver in your application if this functionality needs to be in your application.
Based on your diagram I think that your implementation could also be based on a message-oriented middleware, using a message broker which receives the GPS transmitter or other data.
The message broker would then store the data internally and forward it to all interested clients which are connected. A typical messaging pattern in this case is a "Topic", which broadcasts the messages similar to a radio station.
So the middleware will ensure that the information will be collected (optionally also persisted to disk) and then guarantees the delivery to the receivers. This can be done even in a way where receivers which have been off-line for a while still receive the GPS messages created while they where not listening ('retroactive consumers').
There are many popular free open source message brokers, and most of them also can be used with Delphi.
I am using graphite server to capture my metrics data and bring down to graphs. I have 4 application servers which is load balancer setup. My aim is capture system data such as cpu usage, memory usage, disk load, etc., for all the 4 application servers. I setup an graphite environment in a separate server and i wanted to push the system data for all the applications servers to graphite and get it display as graphs. I don't know what needs to be done for feeding system data to graphite. My thinking was to install statsd in all application servers and feed the system data to graphite but looks like statsd does not support system data rather application data.
Can anyone help me to catch the right track. Thanks in advance.
Running collectd with a graphite agent would be an excellent start to gather the information your after.
There is an almost unlimited amount of ways to get your data into graphite.
You can find a list of tools that have known to work very well with graphite on the readthedocs.org page: http://graphite.readthedocs.org/en/0.9.10/tools.html
There is also an example script that gathers load average from the system in the carbon project: example-client.py