JMX Trans gives "Port already in use" but the port is free - jmx

Question

I'm trying to use jmxtrans to send metrics of Kafka to Influxdb. I've downloded the jar file and configured necessary files for my case. When I try to start the jmxtrans as JAR_FILE=jmxtrans-259-all.jar jmxtrans.sh jvm.json it gives the Port already in use: 5555 error (5555 is the jmx port). My kafka systemd file (prometheus doesn't use jmx port):
[Unit]
Description=Apache Kafka Server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=kafka-zookeeper.service
After=network.target remote-fs.target kafka-zookeeper.service
[Service]
Type=simple
User=kafka
Group=confluent
WorkingDirectory=/etc/systemd/system/
ExecStart=/bin/bash /etc/systemd/system/kafka
Restart=always
Environment="KAFKA_OPTS=-javaagent:/kafka/vty/jmx_prometheus_javaagent-0.3.1.jar=7080:/kafka/vty/kafka-0-8-2.yml"
#Environment="JMX_PORT=5555"
Environment="KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=isstdkafld01 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=5555"
[Install]
WantedBy=multi-user.target
I am pretty sure there is no process that uses the 5555 port, except kafka (PID 17397 is kafka). netstat -nltp also confirms that:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:7081 0.0.0.0:* LISTEN 124801/java
tcp 0 0 0.0.0.0:2191 0.0.0.0:* LISTEN 124801/java
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1184/rpcbind
tcp 0 0 0.0.0.0:9072 0.0.0.0:* LISTEN 17397/java
tcp 0 0 0.0.0.0:20048 0.0.0.0:* LISTEN 1637/rpc.mountd
tcp 0 0 0.0.0.0:5555 0.0.0.0:* LISTEN 17397/java
tcp 0 0 0.0.0.0:36629 0.0.0.0:* LISTEN 17397/java
tcp 0 0 0.0.0.0:33429 0.0.0.0:* LISTEN 17397/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 10716/sshd
tcp 0 0 0.0.0.0:43675 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:45759 0.0.0.0:* LISTEN 1649/rpc.statd
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:34183 0.0.0.0:* LISTEN 124801/java
tcp 0 0 0.0.0.0:7080 0.0.0.0:* LISTEN 17397/java
tcp6 0 0 :::9519 :::* LISTEN 94054/nsrexecd
tcp6 0 0 :::111 :::* LISTEN 1184/rpcbind
tcp6 0 0 :::20048 :::* LISTEN 1637/rpc.mountd
tcp6 0 0 :::8370 :::* LISTEN 94054/nsrexecd
tcp6 0 0 :::8146 :::* LISTEN 94054/nsrexecd
tcp6 0 0 :::4118 :::* LISTEN 28915/ds_agent
tcp6 0 0 :::3000 :::* LISTEN 20644/grafana-serve
tcp6 0 0 :::7937 :::* LISTEN 94054/nsrexecd
tcp6 0 0 :::41537 :::* LISTEN -
tcp6 0 0 :::2049 :::* LISTEN -
tcp6 0 0 :::9090 :::* LISTEN 123916/prometheus
tcp6 0 0 :::7938 :::* LISTEN 94054/nsrexecd
tcp6 0 0 :::33443 :::* LISTEN 1649/rpc.statd
What could be the reason? Mind that when I try to use another port which is not the jmx port, it doesn't give this error and goes for a timeout. The files I use are given below.
jvm.json (8086 is the port for influxdb)
{
"servers" : [ {
"port" : "9072",
"host" : "xxxxxxxxxx",
"alias": "kafka-1",
"queries" : [ {
"outputWriters" : [
{
"#class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://xxxxxxxxxx:8086/",
"database" : "kafka",
"username" : "admin",
"password" : "admin"
} ],
"obj" : "java.lang:type=Memory",
"attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
"resultAlias" : "jvmMemory"
}
],
"numQueryThreads" : 1
} ]
}
jmxtrans.sh
#!/bin/bash
#
# The MIT License
# Copyright © 2010 JmxTrans team
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#
if (( $# != 1 )); then
echo "Usage: jmxtrans.sh <json file or dir>"
exit 1
fi
if [[ -f "${1}" ]]; then
JSON_CONFIG_OPT="-f ${1}"
elif [[ -d "${1}" ]]; then
JSON_CONFIG_OPT="-j ${1}"
else
echo "${1} is not a file nor a directory"
exit 1
fi
JAVA_HOME=${JAVA_HOME:-"/usr"}
LOG_DIR=${LOG_DIR:-"."}
if [[ "${JAR_FILE}" == "" ]]; then
JAR_FILE="$( cd "$( dirname "${BASH_SOURCE[0]}" )/../lib" >/dev/null && pwd )/jmxtrans-all.jar"
fi
ADDITIONAL_JARS=${ADDITIONAL_JARS:-""}
SECONDS_BETWEEN_RUNS=${SECONDS_BETWEEN_RUNS:-"60"}
JAVA=${JAVA:-"${JAVA_HOME}/bin/java"}
CHECK_JAVA=${CHECK_JAVA:-"true"}
JAVA_OPTS=${JAVA_OPTS:-"-Djava.awt.headless=true -Djava.net.preferIPv4Stack=true"}
HEAP_SIZE=${HEAP_SIZE:-"512"}
PERM_SIZE=${PERM_SIZE:-"384"}
MAX_PERM_SIZE=${MAX_PERM_SIZE:-"384"}
JMX_PORT=${JMX_PORT:-"5555"}
LOG_LEVEL=${LOG_LEVEL:-"info"}
CONTINUE_ON_ERROR=${CONTINUE_ON_ERROR:-"false"}
JMXTRANS_OPTS="${JMXTRANS_OPTS} -Djmxtrans.log.level=${LOG_LEVEL} -Djmxtrans.log.dir=${LOG_DIR}"
MONITOR_OPTS=${MONITOR_OPTS:-"-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=${JMX_PORT}"}
GC_OPTS=${GC_OPTS:-"-Xms${HEAP_SIZE}m -Xmx${HEAP_SIZE}m"}
SSL_OPTS=${SSL_OPTS:-""}
if [[ "${SSL_TRUSTSTORE}" != "" ]]; then
SSL_OPTS="${SSL_OPTS} -Djavax.net.ssl.trustStore=${SSL_TRUSTSTORE}"
if [[ "${SSL_TRUSTSTORE_PASSWORD}" != "" ]]; then
SSL_OPTS="${SSL_OPTS} -Djavax.net.ssl.trustStorePassword=${SSL_TRUSTSTORE_PASSWORD}"
fi
fi
if [[ "${ADDITIONAL_JARS}" == "" ]]; then
ADDITIONAL_JARS_OPTS=""
else
ADDITIONAL_JARS_OPTS="-a ${ADDITIONAL_JARS}"
fi
if [[ "${CHECK_JAVA}" == "true" ]]; then
JAVA_VERSION=`${JAVA} -version 2>&1`
if [[ $? != 0 ]]; then
echo "Cannot execute ${JAVA}!"
exit 1
fi
fi
if [[ ! -f ${JAR_FILE} ]]; then
echo "File not found - ${JAR_FILE}"
exit 1
fi
EXEC=${EXEC:-"-jar ${JAR_FILE} -e ${JSON_CONFIG_OPT} -s ${SECONDS_BETWEEN_RUNS} -c ${CONTINUE_ON_ERROR} ${ADDITIONAL_JARS_OPTS} ${ADDITIONAL_OPTS}"}
${JAVA} -server ${JAVA_OPTS} ${JMXTRANS_OPTS} ${GC_OPTS} ${MONITOR_OPTS} ${SSL_OPTS} ${EXEC}

Answer 1

Address already use issue:
Open your terminal:
ps aux || grep
list all running services and find the id you want to kill.
sudo kill -9 <id you want to kill>

Answer 2

Leaving the JMX_PORT=${JMX_PORT:-"5555"} line in jmxtrans.sh as its default value, 2101 solved the problem.