How to monitor Kafka broker using jmxtrans? - jmx

Question

Kafka 0.8.1.1 (kafka_2.8.0-0.8.1.1.tgz)
I am using jmxtrans to do JMX monitoring of a Kafka instance (which is running in docker). Unfortunately, kafka metrics are not being returned.
I have tried a few things to debug this and know that kafka is running correctly (I can produce/consume messages successfully) have concluded that jmxtrans does return JMX metrics (for example, java.lang:type=Memory, attribute=HeapMemoryUsage returns correct data) so the general kafka and JMX capability seems to be working. Also, I can access the metrics when I use jconsole -- the metrics seem to be captured with data in all relevant fields.
When I try jmxtrans using the following configuration, unfortunately, I do not get any information back (no data at all in fact). I believe the metrics are supposed to be captured based upon the kafka documentation ("kafka.server:type=BrokerTopicMetrics", attribute="MessagesInPerSec")
The following is the jmxtrans configuration that I used:
{
"servers" : [ {
"port" : "9999",
"host" : "10.0.1.201",
"queries" : [ {
"outputWriters" : [ {
"#class" : "com.googlecode.jmxtrans.model.output.StdOutWriter",
"settings" : {
}
} ],
"obj" : "kafka.server:type=BrokerTopicMetrics",
"attr" : [ "MessagesInPerSec" ]
} ],
"numQueryThreads" : 2
} ]
}
I am not sure why data is not returned. Maybe I setup an invalid jmxtrans configuration or perhaps I am specifying the metric improperly.
Any help is appreciated.

Answer 1

After a lot of experimentation, I have now resolved the question. For completeness, below is how I resolved the problem.
It appears that I specified the "obj" value incorrectly.
The CORRECT obj value (an example) is as follows:
"obj": "\"kafka.server\":type=\"BrokerTopicMetrics\",name \"AllTopicsLogBytesAppendedPerSec\"",
"attr": [ "Count" ]
Note that the "obj" value requires additional quotes. This seems unusual and different than the normal pattern I have seen (no quotes) for other JMX obj values.
JMXTRANS provided valid output after putting the correct (quoted) values in the obj string...

Answer 2

As I could've found out in ./bin/jmxtrans.sh, by default the stdout/log file is /dev/null.
LOG_FILE=${LOG_FILE:-"/dev/null"}
That's why it's important to set the env var to something you can use to see the output:
LOG_FILE=log.txt ./bin/jmxtrans.sh start kafka.json
I'm using the following kafka.json configuration file:
{
"servers" : [ {
"port" : "10101",
"host" : "localhost",
"queries" : [ {
"outputWriters" : [ {
"#class" : "com.googlecode.jmxtrans.model.output.StdOutWriter",
"settings" : {
}
} ],
"obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=testowo",
"attr" : [ "Count" ]
} ],
"numQueryThreads" : 2
} ]
}
When you start jmxtrans it will query the broker with JMX on localhost:10101 about Count attribute for testowo topic. It will print the result out to the file LOG_FILE every 60 secs (you can change it using SECONDS_BETWEEN_RUNS env var), e.g.
LOG_FILE=log.txt SECONDS_BETWEEN_RUNS=5 ./bin/jmxtrans.sh start kafka.json
You may want to use other writers of jmxtrans so the output is not intermingled, e.g.
{
"servers" : [ {
"port" : "10101",
"host" : "localhost",
"queries" : [ {
"outputWriters" : [ {
"#class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",
"settings" : {
"outputFile" : "testowo-counts.txt",
"maxLogFileSize" : "10MB",
"maxLogBackupFiles" : 200,
"delimiter" : "\t",
"debug" : true
}
} ],
"obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=testowo",
"attr" : [ "Count" ]
} ],
"numQueryThreads" : 2
} ]
}
And the last but not least, to set the JMX port to a known value use JMX_PORT env var when starting a Kafka broker using ./bin/kafka-server-start.sh, i.e.
JMX_PORT=10101 ./bin/kafka-server-start.sh config/server.properties