jmxtrans and kafka not working - jmx

Question

I am trying to get the Kafka metrics (version 0.8.2) to my grafana server. Unfortunately I can only get java.lang metrics, but no kafka metrics. Connecting with jmxtrans and jconsoel is now problem and I can see the MBeans of Kafka.
Configuration of jmxtrans:
{
"servers" : [ {
"port" : "9393",
"host" : "localhost",
"queries" : [ {
"obj" : "kafka.network:type=RequestMetrics,name=LocalTimeMs,request=ConsumerMetadata",
"attr" : [ "Count" ],
"outputWriters" : [ {
"#class" : "com.googlecode.jmxtrans.model.output.GraphiteWriter",
"settings" : {
"port" : 2003,
"host" : "localhost"
}
} ]
}, {
"obj" : "kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=FetchConsumer",
"attr" : [ "Count" ],
"outputWriters" : [ {
"#class" : "com.googlecode.jmxtrans.model.output.GraphiteWriter",
"settings" : {
"port" : 2003,
"host" : "localhost"
}
} ]
}, {
"obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=OffsetFetch",
"attr" : [ "Count" ],
"outputWriters" : [ {
"#class" : "com.googlecode.jmxtrans.model.output.GraphiteWriter",
"settings" : {
"port" : 2003,
"host" : "localhost"
}
} ]
}, {
"obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Offsets",
"attr" : [ "Count" ],
"outputWriters" : [ {
"#class" : "com.googlecode.jmxtrans.model.output.GraphiteWriter",
"settings" : {
"port" : 2003,
"host" : "localhost"
}
} ]
}
],
"numQueryThreads" : 2
} ]
}
Log of jmxtrans:
15:52:11.564 [ServerScheduler_Worker-10] DEBUG org.quartz.core.JobRunShell - Calling execute on job ServerJob.localhost:9393-1443714611563-0428747962
15:52:11.564 [ServerScheduler_Worker-10] DEBUG c.googlecode.jmxtrans.jobs.ServerJob - +++++ Started server job: Server [host=localhost, port=9393, url=null, cronExpression=null, numQueryThreads=2]
15:52:11.569 [ServerScheduler_Worker-10] DEBUG com.googlecode.jmxtrans.jmx.JmxUtils - ----- Creating 4 query threads
15:52:11.638 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Executing queryName [kafka.network:name=RequestQueueTimeMs,request=FetchConsumer,type=RequestMetrics] from query [Query [obj=kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=FetchConsumer, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]]
15:52:11.638 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Finished running outputWriters for query: Query [obj=kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=FetchConsumer, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]
15:52:11.640 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Executing queryName [kafka.network:name=TotalTimeMs,request=OffsetFetch,type=RequestMetrics] from query [Query [obj=kafka.network:type=RequestMetrics,name=TotalTimeMs,request=OffsetFetch, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]]
15:52:11.640 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Finished running outputWriters for query: Query [obj=kafka.network:type=RequestMetrics,name=TotalTimeMs,request=OffsetFetch, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]
15:52:11.641 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Executing queryName [kafka.network:name=TotalTimeMs,request=Offsets,type=RequestMetrics] from query [Query [obj=kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Offsets, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]]
15:52:11.641 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Finished running outputWriters for query: Query [obj=kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Offsets, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]
15:52:11.642 [pool-60-thread-1] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Executing queryName [kafka.network:name=LocalTimeMs,request=ConsumerMetadata,type=RequestMetrics] from query [Query [obj=kafka.network:type=RequestMetrics,name=LocalTimeMs,request=ConsumerMetadata, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]]
15:52:11.642 [pool-60-thread-1] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Finished running outputWriters for query: Query [obj=kafka.network:type=RequestMetrics,name=LocalTimeMs,request=ConsumerMetadata, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]
15:52:11.643 [ServerScheduler_Worker-10] DEBUG c.googlecode.jmxtrans.jobs.ServerJob - +++++ Finished server job: Server [host=localhost, port=9393, url=null, cronExpression=null, numQueryThreads=2]
For me it looks like that jmxtrans can connect without problem and tries to get the data, but does not get any data at all from the kafka metrics.
JMX Options of Kafka:
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9393 -Djava.rmi.server.hostname=example.com -Dcom.sun.management.jmxremote.rmi.port=9011
Any idea why the Kafka metrics are not available?
Thanks.

Answer 1

Some versions of Kafka had quoted JMX elements (you could see the attributes having quotes in the jconsole). This has/is changing for future versions of Kafka. Check out the this post (How to monitor Kafka broker using jmxtrans?) for details, but if you use quoted strings it should work (see example below:)
"obj": "\"kafka.server\":type=\"BrokerTopicMetrics\",name \"AllTopicsLogBytesAppendedPerSec\"", "attr": [ "Count" ]
Note that the "obj" value requires additional quotes. When I tried this, JMXTRANS provided valid output after putting the correct (quoted) values in the obj string...