I am interested in monitoring my multi-agent apache flume setup. I have enabled the inbuilt ganglia server which provides me the flume metrics through JSON data. Now I am interested in viewing these info in graphs/charts. TO achieve this I am using ganglia web ui, I have these questions - Do I have to install gmond and gmetad to achieve it, if not then how I will use the existing ganglia info with the ganglia web ui ?
Thanks in advance.
You'll need both, IMHO. Moreover, I think Flume can communicate directly to a gmond by appending some stuff in JAVA_OPTS, see hortonworks docs.
You'll need gmetad because it stores your data in RRD files, and the web UI query on it to display graphs.
Graphite can do the job too.
Related
I have a requirement where in I need to monitor some custom services on aws nodes and collect metrics in timeseries. There are specifically two use cases. One being the monitoring of hardware resources like cpu, mem, disk util etc and the other being monitoring service specific metrics.
While reading up I came across collectd as one of the open source option. However I wanted to know how I can use collectd to monitor service specific metrics. Does collectd expose APIs which the service can use to log the metrics and if yes how performant is it.
I am new to collectd & would like to know if there are any other open source options as well.
The collectd agent can monitor custom metrics using read plugins:
Exec plugin for custom metrics fetched with bash scripts
cURL-JSON for metrics published in JSON format via HTTP
I have a cluster of servers monitored with ganglia and I just added a new application on one of them. The new application uses an SNMP handler to report about its activity.
I never used SNMP before and would like to gather most of SNMP metrics that I see in the MIB file to my rrd files used by Ganglia.
Would this be possible ?
I wanted to write a new ganglia module that would take into account the new metrics from the new application. So I tried to read the code of the SNMP handler of the application but cannot catch where it takes its information from.
What would be a good way to figure out this situation ?
Thank you!
I need to make freinds zabbix & other monitoring system.
My company uses Zabbix for monitoring. Our partner plans to use other system.
We need to exchange monitoring datas.
I'm interested in coopereation with the next systems: BMC Patrol, MS SCOM, NetCool, Portal.
What is the best way to integrate it?
Maybe via SNMP?
Replicate hosts and metrics into your Zabbix (use Zabbix trapper item type and setup also Allowed hosts value) and then just use some suitable zabbix-sender implementation and push data into Zabbix.
IMO it's terrible idea, because latency, syncing, ... Do you really need data (item values) or do you need only visualize data from different datasources in one graph?
Regarding BMC Patrol you can use History Loader/Propagator KM to export the monitoring data:
https://docs.bmc.com/docs/display/public/unixlinux912/PATROL+KM+for+History+Loader
or you can use the 'dump_hist' command to dump the history data from the agents:
https://docs.bmc.com/docs/display/pia9600/dump_hist+uility
Regarding Netcool events, you could get the information using different approaches, for example, depending on the version, you could get the events from the HTTP interface, as described below:
https://www.ibm.com/support/knowledgecenter/en/SSNFET_9.2.0/com.ibm.netcool_OMNIbus.doc_7.4.0/omnibus/wip/api/reference/omn_api_http_httpinterface.html
Or perhaps you could create a flat file gateway to read the events and write them on a file:
https://www.ibm.com/support/knowledgecenter/en/SSSHTQ/omnibus/gateways/flatfilegw/wip/concept/flatfilegw_intro.html
I'm trying to monitor my network usage with Graphite but I can't figure out how to do this, could you help me please?
In addition to this, I would like to monitor some other services, as nginx, mysql, etc,..
Thanks for your help!
Best.
Ofcourse there are multiple solutions possible. The solution I'm using is collectd. With collectd you can collect statistic data from plugins in rrd files. It has a lot of plugins like network, nginx and mysql.
It does not generate graphs by itself, but there are multiple ways to generate graphs. One of them is to send the collected data to graphite with the graphite plugin.
I am new with Apache Flume.
I understand that Apache Flume can help transport data.
But I still fail to see the ultimate benefit offered by Apache Flume.
If I can configure a software or make a software to send which data goes where, why I need Flume?
Maybe someone can explain a situation that shows Apache Flume's benefit?
Reliable transmission (if you use the file channel):
Flume sends batches of small events. Every time it sends a batch to the next node it waits for acknowledgment before deleting. The storage in the file channel is optimized to allow recovery on crash.
I think the biggest benefit that you get out of flume is extensiblity. Basically all components starting from source, interceptor and sink, everything is extensible.
We use flume and read data using custom kakfa source, data is in the form of JSON we parse it in custom kafka source and then pass it on to HDFS sink. It working reliably in 5 of nodes. We extended only kafka source, HDFS sink functionality we got out the box.
At the same time, being from the Hadoop ecosystem, you get great community support and multiple options to use the tools in different ways.