Why does kafka-run-class command to fetch JMX metrics fails with java.net.BindException

Description

When running below command to fetch JMX metrics on connect worker machine 

kafka-run-class kafka.tools.JmxTool --object-name kafka.connect:type=connector-metrics,connector=* --jmx-url service:jmx:rmi:///jndi/rmi://<host>:<port>/jmxrmi --reporting-interval 1000

It fails with below error

jdk.internal.agent.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: <port number>; nested exception is:
    java.net.BindException: Address already in use
    at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:843)
    at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:480)
    at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447)
    at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)
Caused by: java.rmi.server.ExportException: Port already in use: <port number>; nested exception is:
    java.net.BindException: Address already in use
    at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:346)
    at java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243)
    at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:415)
    at java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
    at java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:235)
    at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$PermanentExporter.exportObject(ConnectorBootstrap.java:205)
    at java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:153)
    at java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:138)
    at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:453)
    at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:839)
    ... 3 more
Caused by: java.net.BindException: Address already in use
    at java.base/sun.nio.ch.Net.bind0(Native Method)
    at java.base/sun.nio.ch.Net.bind(Net.java:555)
    at java.base/sun.nio.ch.Net.bind(Net.java:544)
    at java.base/sun.nio.ch.NioSocketImpl.bind(NioSocketImpl.java:648)
    at java.base/java.net.ServerSocket.bind(ServerSocket.java:393)
    at java.base/java.net.ServerSocket.<init>(ServerSocket.java:279)
    at java.base/java.net.ServerSocket.<init>(ServerSocket.java:172)
    at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createServerSocket(TCPDirectSocketFactory.java:45)
    at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:673)
    at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
    ... 12 more

Applies To

Confluent Platform

Cause

The actual cause of Address already in use error is because kafka-run-class command tries to listen on a port which is occupied by some other process for JMX. 

kafka-run-class looks for KAFA_JMX_OPTS environment variable to find the port on which it has to listen for below two properties 

-Dcom.sun.management.jmxremote.rmi.port=<port> -Dcom.sun.management.jmxremote.port=<port> 

Usually in case when it is run from the connect worker machine, the connect worker process is listening on the same port, and if the shell where kafka-run-class command ran, KAFA_JMX_OPTS environment variable is set (Something pointing as below to the same port) 

KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false  -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=<port> -Dcom.sun.management.jmxremote.port=<port> 

This is expected to fail as port is already occupied by connect worker process. 

Resolution

To resolve the issue you can follow either of the approach listed below

1. Point below two properties under KAFKA_JMX_OPTS environment variable to a unique port to which no other process is listening

-Dcom.sun.management.jmxremote.rmi.port=<port> -Dcom.sun.management.jmxremote.port=<port> 

2. You can also unset the KAFA_JMX_OPTS environment variable from the current shell as 

unset KAFA_JMX_OPTS

And then run the kafka-run-class command.