I am currently start working on Apache flume.I had installed the apache hadoop,java8 64 bit and Apache flume in same directory.
Now i need to monitoring the apache flume.
Is there any possibility to monitoring the Apache flume using REST API and JMX?
Prometheus JMX Exporter or Jolokia can be added to Flume JVM arguments
Related
I am trying to configure Spark 3.3.0 Thrift Server with Apache Ranger but I cannot find any resources or information for this setup.Any suggestions on how to implement this? Thanks very much!
I already have an STS (kerberos jdbc) turned on and Apache Ranger on a Kerberos authenticated cluster, but I don't know how to integrate STS with Apache Ranger. I have done my research on https://github.com/yaooqinn/spark-ranger, but It's an archived repository and the code doesn't support Spark 3.3.0. Any help would be appreciated.
i think i found the new plugin:
https://kyuubi.readthedocs.io/en/master/security/authorization/spark/install.html
it's working with apache spark 3.x thrift server
I have a prometheus server running on a K8s instance and telegraf on a different cluster. Is there some way to pull metrics from the prometheus server using telegraf? I know there is telegraf support for scraping metrics from prometheus clients but I am looking to get these metrics from the prometheus server.
Thanks
there is this thing inside data sources, called scrapers, its a tab, you just need to put the url of the server.
I am trying to configure this using cli, but i can only do it with gui
There is a prometheus remote write parser (https://github.com/influxdata/telegraf/tree/master/plugins/parsers/prometheusremotewrite), I think it will be included in the 1.19.0 release of Telegraf. If you want to try it out now you can use a nightly build. (https://github.com/influxdata/telegraf#nightly-builds)
Configure your prometheus remote write towards telegraf and configure the input plugin to listen for traffic on the port that you configured. For convenience sake, have the output plugin configured to file so you can see the metrics in a file, almost immediately
I am trying to understand Flume and referring to the official page of flume at flume.apache.org
In particular, referring to this section, I am bit confused in this.
Do we need to run the flume agent on the actual webserver or can we run flume agents in a different physical server and acquire data from webserver?
If above is correct, then how flume agent gets the data from webserver logs? How can webserver make its data available to the flume agent ?
Can anyone help understand this?
The Flume agent must pull data from a source, publish to a channel, which then writes to a sink.
You can install Flume agent in either a local or remote configuration. But, keep in mind that having it remote will add some network latency to your event processing, if you are concerned about that. You can also "multiplex" Flume agents to have one remote aggregation agent, then individual local agents on each web server.
Assuming a flume agent is locally installed using a Spooldir or exec source, it'll essentially tail any file or run that command locally. This is how it would get data from logs.
If the Flume agent is setup as a Syslog or TCP source (see Data ingestion section on network sources), then it can be on a remote machine, and you must establish a network socket in your logging application to publish messages to the other server. This is a similar pattern to Apache Kafka.
I enabled JMX in JBoss it works fine with jconsole. How to monitor JBoss with JMX in Zabbix.
my JBoss version: 6.4.9
Zabbix version: 3.2
Zabbix 3.2 does not support remoting-jmx, used by JBoss, without patching the product. You should upgrade to 3.4 which introduced support for configurable JMX endpoints.
Is it possible to monitor TITAN cassandra server with rexster remotely via JMX using something like VisualVM?
I have titan installed on the cloud and want to monitor it from my dev box. Is this possible.
I have read this
https://github.com/tinkerpop/rexster/wiki/Monitoring
but it seems that JMX MBeans are only available locally however I could be wrong
You can monitor Rexster JMX remotely with VisualVM, but it takes a bit of configuration and changes to rexster.sh as you need to include these environment variables:
-Dcom.sun.management.jmxremote.port=3333
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
You can read some more about how to do remote setup on the VisualVM site.
You mentioned that you are trying to monitor an instance in the cloud. You didn't mention the cloud provider, but I've had trouble doing this with EC2 in the past. Perhaps this post will help you out. While I've had issues with VisualVM remoting to EC2, I have successfully connected to Rexster via VisualVM from another EC2 instance without trouble so if all else fails that could be your workaround.