I have used JMX exporter to monitor Java application deployed based on jetty .
I have downloaded the jmx_prometheus_javaagent-0.1.0.jar file
Deployed the java application with jmx_exporter command
nohup java -javaagent:./jmx_prometheus_javaagent-0.1.0.jar=7101:config.yaml -Dorg.eclipse.jetty.server.Request.maxFormContentSize=10000000 -Xms256m -Xmx256m -Djava.io.tmpdir=epoch_temp_dir -jar jetty-runner-9.0.7.v20131107.jar --log yyyy_mm_dd-java-application-1-request.log --out yyyy_mm_dd-java-application-1-output.log --port 8091 --path /java-application-1 java-app1.war >> java-application-1.log 2>&1 &
config.yaml
#cat config.yaml
---
startDelaySeconds: 0
jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:7101/jmxrmi
ssl: false
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
- pattern: ".*"
Prometheus shows connection timed out in status page
output log : of the java application deployed
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector collect
SEVERE: JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error du
ring JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:106)
at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:415)
at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:180)
at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:213)
at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:134)
at io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:59)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
... 17 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
Opened the 7101 ports in the client server and grant access to Prometheus server.
Prometheus Server RAM usage:
# free -h
total used free shared buff/cache available
Mem: 4.8G 1.9G 118M 256M 2.8G 2.3G
Swap: 0B 0B 0B
Client server RAM :
# free -h
total used free shared buff/cache available
Mem: 9.8G 3.7G 435M 16M 5.6G 5.7G
Swap: 0B 0B 0B
Curl localhost:7101 in the client server is not responding
Remove the jmxUrl from the config, that's not required for agent use.
Related
I am running keycloak v8.0.0 using docker compose file. I am persisting the data into a MySQL database which I am running locally.
version: '3'
services:
keycloak:
image: jboss/keycloak:8.0.0
ports:
- "9999:8080"
environment:
KEYCLOAK_USER: admin
KEYCLOAK_PASSWORD: <ADMIN_PASSWORD>
DB_VENDOR: mysql
DB_ADDR: <HOST>
DB_PORT: <PORT>
DB_USER: <USER>
DB_PASSWORD: <PASSWORD>
JDBC_PARAMS: "useSSL=false"
volumes:
- "./realms:/tmp"
I tried exporting a realm from the running docker container by using following command.
docker exec -it kc /opt/jboss/keycloak/bin/standalone.sh \
-Djboss.socket.binding.port-offset=100 -Dkeycloak.migration.action=export \
-Dkeycloak.migration.provider=singleFile \
-Dkeycloak.migration.realmName=my_realm\
-Dkeycloak.migration.usersExportStrategy=REALM_FILE \
-Dkeycloak.migration.file=/tmp/my_realm-realm.json
Below is the stack trace of the error i am getting when I run the above command. When I am running the above command keycloak is not able to connect to database server i.e. MySQL in this case. How to specify database connection configuration in this scenario?
16:30:04,000 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 65) MSC000001: Failed to start service jboss.deployment.unit."keycloak-server.war".undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.unit."keycloak-server.war".undertow-deployment: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()
at org.wildfly.extension.undertow#18.0.0.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.jboss.threads#2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads#2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at org.jboss.threads#2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads#2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.base/java.lang.Thread.run(Thread.java:834)
at org.jboss.threads#2.3.3.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()
at org.jboss.resteasy.resteasy-jaxrs#3.9.0.Final//org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:164)
at org.jboss.resteasy.resteasy-jaxrs#3.9.0.Final//org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2784)
at org.jboss.resteasy.resteasy-jaxrs#3.9.0.Final//org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:364)
at org.jboss.resteasy.resteasy-jaxrs#3.9.0.Final//org.jboss.resteasy.spi.ResteasyDeployment.startInternal(ResteasyDeployment.java:277)
at org.jboss.resteasy.resteasy-jaxrs#3.9.0.Final//org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:89)
at org.jboss.resteasy.resteasy-jaxrs#3.9.0.Final//org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:119)
at org.jboss.resteasy.resteasy-jaxrs#3.9.0.Final//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)
at io.undertow.servlet#2.0.26.Final//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
at org.wildfly.extension.undertow#18.0.0.Final//org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
at io.undertow.servlet#2.0.26.Final//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
at io.undertow.servlet#2.0.26.Final//io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:305)
at io.undertow.servlet#2.0.26.Final//io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:145)
at io.undertow.servlet#2.0.26.Final//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:585)
at io.undertow.servlet#2.0.26.Final//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:556)
at io.undertow.servlet#2.0.26.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet#2.0.26.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow#18.0.0.Final//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow#18.0.0.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
at org.wildfly.extension.undertow#18.0.0.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
at org.wildfly.extension.undertow#18.0.0.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
at org.wildfly.extension.undertow#18.0.0.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
at io.undertow.servlet#2.0.26.Final//io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:598)
at org.wildfly.extension.undertow#18.0.0.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:97)
at org.wildfly.extension.undertow#18.0.0.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
... 8 more
Caused by: java.lang.RuntimeException: Failed to connect to database
at org.keycloak.keycloak-model-jpa#8.0.0//org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.getConnection(DefaultJpaConnectionProviderFactory.java:372)
at org.keycloak.keycloak-model-jpa#8.0.0//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.lazyInit(LiquibaseDBLockProvider.java:65)
at org.keycloak.keycloak-model-jpa#8.0.0//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.lambda$waitForLock$2(LiquibaseDBLockProvider.java:96)
at org.keycloak.keycloak-server-spi-private#8.0.0//org.keycloak.models.utils.KeycloakModelUtils.suspendJtaTransaction(KeycloakModelUtils.java:682)
at org.keycloak.keycloak-model-jpa#8.0.0//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.waitForLock(LiquibaseDBLockProvider.java:94)
at org.keycloak.keycloak-services#8.0.0//org.keycloak.services.resources.KeycloakApplication$1.run(KeycloakApplication.java:178)
at org.keycloak.keycloak-server-spi-private#8.0.0//org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:227)
at org.keycloak.keycloak-services#8.0.0//org.keycloak.services.resources.KeycloakApplication.startup(KeycloakApplication.java:171)
at org.keycloak.keycloak-services#8.0.0//org.keycloak.services.resources.KeycloakApplication.init(KeycloakApplication.java:162)
at org.keycloak.keycloak-services#8.0.0//org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:143)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.jboss.resteasy.resteasy-jaxrs#3.9.0.Final//org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:152)
... 31 more
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/KeycloakDS
at org.jboss.ironjacamar.jdbcadapters#1.4.17.Final//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:159)
at org.jboss.as.connector#18.0.0.Final//org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
at org.keycloak.keycloak-model-jpa#8.0.0//org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.getConnection(DefaultJpaConnectionProviderFactory.java:366)
... 45 more
Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/KeycloakDS
at org.jboss.ironjacamar.impl#1.4.17.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:690)
at org.jboss.ironjacamar.impl#1.4.17.Final//org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:440)
at org.jboss.ironjacamar.impl#1.4.17.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
at org.jboss.ironjacamar.jdbcadapters#1.4.17.Final//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151)
... 47 more
Caused by: javax.resource.ResourceException: IJ031084: Unable to create connection
at org.jboss.ironjacamar.jdbcadapters#1.4.17.Final//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:345)
at org.jboss.ironjacamar.jdbcadapters#1.4.17.Final//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:352)
at org.jboss.ironjacamar.jdbcadapters#1.4.17.Final//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:287)
at org.jboss.ironjacamar.impl#1.4.17.Final//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.createConnectionEventListener(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1328)
at org.jboss.ironjacamar.impl#1.4.17.Final//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:499)
at org.jboss.ironjacamar.impl#1.4.17.Final//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:632)
at org.jboss.ironjacamar.impl#1.4.17.Final//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:604)
at org.jboss.ironjacamar.impl#1.4.17.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
... 50 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 97 milliseconds ago. The last packet sent successfully to the server was 91 milliseconds ago.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:201)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.MysqlIO.negotiateSSLConnection(MysqlIO.java:4912)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1663)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1224)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2190)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at org.jboss.ironjacamar.jdbcadapters#1.4.17.Final//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:321)
... 57 more
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at java.base/sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:169)
at java.base/sun.security.ssl.ClientHandshakeContext.<init>(ClientHandshakeContext.java:98)
at java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:216)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:186)
... 73 more
The first solution did not work for me, so I had to investigate further. In my case, the reason was, that the DB_PORT environment variable was not added to the connection string, so keycloak tried to connect to the postgres DB under the dafault port 5432, as seen in this log:
11:29:10,185 ERROR [stderr] (ServerService Thread Pool -- 68) FINE: Connecting with URL: jdbc:postgresql://156149bb-0045-4086-b11a-1b55458b2a5f.blrv234f085n1l00bhrg.private.databases.appdomain.cloud/ibmclouddb?connectTimeout=30&ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory&loggerLevel=DEBUG
[...]
11:29:10,194 ERROR [stderr] (ServerService Thread Pool -- 68) FINE: Trying to establish a protocol version 3 connection to 156149bb-0045-4086-b11a-1b55458b2a5f.blrv234f085n1l00bhrg.private.databases.appdomain.cloud:5432
So I added the Port to the DB_ADDR variable and everything worked fine.
I got the same error today,and found is the JDBC_PARAMS format error;
so you can exec in the container,
docker exec -it kc bash
and modify the JDBC_PARAMS
# Append '?' in the beggining of the string if JDBC_PARAMS value isn't empty
export JDBC_PARAMS=$(echo ${JDBC_PARAMS} | sed '/^$/! s/^/?/')
Or redefine JDBC_PARAMS env
export JDBC_PARAMS='?connectTimeout=30&useSSL=false&allowPublicKeyRetrieval=true&characterEncoding=UTF-8'
the key is prefix ? char
then exec
/opt/jboss/keycloak/bin/standalone.sh \
-Djboss.socket.binding.port-offset=100 -Dkeycloak.migration.action=export \
-Dkeycloak.migration.provider=singleFile \
-Dkeycloak.migration.realmName=my_realm\
-Dkeycloak.migration.usersExportStrategy=REALM_FILE \
-Dkeycloak.migration.file=/tmp/my_realm-realm.json
everything is OK!
I want to use the schema registry docker (image owned by confluent) with my open-source Kafka I installed locally on my PC.
I am using the following command to run the image :
docker run -p 8081:8081 \
-e SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS=PLAINTEXT://127.0.0.1:9092 \
-e SCHEMA_REGISTRY_LISTENERS=http://0.0.0.0:8081 \
-e SCHEMA_REGISTRY_DEBUG=true confluentinc/cp-schema-registry:latest
but I am getting the following connection errors:
[kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.
[kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.
[main] ERROR io.confluent.admin.utils.ClusterStatus - Error while getting broker list.
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:149)
at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[main] INFO io.confluent.admin.utils.ClusterStatus - Expected 1 brokers but found only 0. Trying to query Kafka for metadata again ...
I have Kafka installed on my localhost.
Any idea to solve this, please?
I have Kafka installed on my localhost
As commented, localhost is unclear when you're actually using multiple machines (one physical and at least one virtual)
You need to use host.docker.internal:9092
https://docs.docker.com/docker-for-windows/networking/ (removed because host is not windows)
On a Linux host, you need to use host networking mode
https://docs.docker.com/network/host/
Although, realistically, running Kafka in a container would be simpler for connecting the two
I am super new to docker and trying to configure single node hadoop using docker in ubuntu server. Here is what I have already done.
$ docker pull sequenceiq/hadoop-docker:2.7.1
......
$ docker run -it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash
Starting sshd: [ OK ]
18/06/27 12:59:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [cb46e163e0be]
cb46e163e0be: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-cb46e163e0be.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-cb46e163e0be.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-cb46e163e0be.out
18/06/27 12:59:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn--resourcemanager-cb46e163e0be.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-cb46e163e0be.out
bash-4.1# jps
532 ResourceManager
204 DataNode
118 NameNode
371 SecondaryNameNode
918 Jps
620 NodeManager
jps shows that resource manager is running. Now I tried to test the hadoop
bash-4.1# cd $HADOOP_PREFIX
bash-4.1# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'
18/06/27 13:02:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/06/27 13:02:25 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/06/27 13:02:27 INFO input.FileInputFormat: Total input paths to process : 31
18/06/27 13:02:27 INFO mapreduce.JobSubmitter: number of splits:31
18/06/27 13:02:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1530118774059_0001
18/06/27 13:02:28 INFO impl.YarnClientImpl: Submitted application application_1530118774059_0001
18/06/27 13:02:28 INFO mapreduce.Job: The url to track the job: http://cb46e163e0be:8088/proxy/application_1530118774059_0001/
18/06/27 13:02:28 INFO mapreduce.Job: Running job: job_1530118774059_0001
18/06/27 13:02:44 INFO mapreduce.Job: Job job_1530118774059_0001 running in uber mode : false
18/06/27 13:02:44 INFO mapreduce.Job: map 0% reduce 0%
18/06/27 13:05:56 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
18/06/27 13:05:57 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
18/06/27 13:05:58 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
18/06/27 13:05:59 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
18/06/27 13:06:00 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
18/06/27 13:06:01 INFO ipc.Client: Retrying connect to server: cb46e163e0be/172.17.0.2:42698. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
^C
bash-4.1# jps
532 ResourceManager
204 DataNode
1671 Jps
371 SecondaryNameNode
bash-4.1#
Now I don't understand 2 things here.
This is kind of official image of hadoop so why it is not running correctly? Did I make any mistake? IF yes then what ?
When I did jps before running the example code you can see node manager and name node were returned by jps. But after running the example and quitting the example, these couple of objects were not returned by jps. Why is it so?
Please help. Thanks
I correctly start server:
~/Downloads/neo4j-community-3.2.1/bin $ ./neo4j start Active database:
graph.db Directories in use: home:
/home/user/Downloads/neo4j-community-3.2.1 config:
/home/user/Downloads/neo4j-community-3.2.1/conf logs:
/home/user/Downloads/neo4j-community-3.2.1/logs plugins:
/home/user/Downloads/neo4j-community-3.2.1/plugins import:
/home/user/Downloads/neo4j-community-3.2.1/import data:
/home/user/Downloads/neo4j-community-3.2.1/data certificates:
/home/user/Downloads/neo4j-community-3.2.1/certificates run:
/home/user/Downloads/neo4j-community-3.2.1/run Starting Neo4j.
WARNING: Max 1024 open files allowed, minimum of 40000 recommended.
See the Neo4j manual. Started neo4j (pid 29246). It is available at
http://localhost:7474/ There may be a short delay until the server is
ready. See /home/user/Downloads/neo4j-community-3.2.1/logs/neo4j.log
for current status.
then when I try to launch the neo4j-shell:
~/Downloads/neo4j-community-3.2.1/bin $ ./neo4j-shell -v ERROR (-v for expanded information): Connection
refused java.rmi.ConnectException: Connection refused to host:
localhost; nested exception is: java.net.ConnectException:
Connection refused at
sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) at
sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at
sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) at
sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) at
java.rmi.Naming.lookup(Naming.java:101) at
org.neo4j.shell.impl.RmiLocation.getBoundObject(RmiLocation.java:191)
at
org.neo4j.shell.impl.RemoteClient.findRemoteServer(RemoteClient.java:72)
at org.neo4j.shell.impl.RemoteClient.(RemoteClient.java:65) at
org.neo4j.shell.impl.RemoteClient.(RemoteClient.java:46) at
org.neo4j.shell.ShellLobby.newClient(ShellLobby.java:204) at
org.neo4j.shell.StartClient.startRemote(StartClient.java:358) at
org.neo4j.shell.StartClient.start(StartClient.java:229) at
org.neo4j.shell.StartClient.main(StartClient.java:147) Caused by:
java.net.ConnectException: Connection refused at
java.net.PlainSocketImpl.socketConnect(Native Method) at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
java.net.Socket.connect(Socket.java:589) at
java.net.Socket.connect(Socket.java:538) at
java.net.Socket.(Socket.java:434) at
java.net.Socket.(Socket.java:211) at
sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at
sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 13 more
-host Domain name or IP of host to connect to (default:
localhost) -port Port of host to connect to (default: 1337)
-name RMI name, i.e. rmi://:/ (default: shell) -pid Process ID to connect to -c Command line to execute. After executing it the shell exits -file File
containing commands to execute, or '-' to read from stdin. After
executing it the shell exits -readonly Connect in readonly mode
(only for connecting with -path) -path Points to a neo4j db path
so that a local server can be started there -config Points to a
config file when starting a local server
Example arguments for remote: -port 1337 -host 192.168.1.234 -port
1337 -name shell -host localhost -readonly ...or no arguments for
default values Example arguments for local: -path /path/to/db -path
/path/to/db -config /path/to/neo4j.config -path /path/to/db -readonly
the server is in its default initial configuration, the only thing I changed is the graph username and password.
neo4j-shell is deprecated since version 3.1, you should use cypher-shell instead.
But you can enable it by adding this configuration : dbms.shell.enabled=true
Cheers.
So I am trying to connect my jnlp slave agent (via java web start) to my master jenkins machine.
I have port 49187 fixed for tcp connection of jnlp slave agents and its open in slave machines. As I try and connect, it goes past the handshake but then fails with the following error:
Aug 19, 2014 3:51:10 PM com.youdevise.hudson.slavestatus.SlaveListener call
INFO: Slave-status listener starting
Aug 19, 2014 3:51:10 PM com.youdevise.hudson.slavestatus.SlaveListener$1 run
SEVERE: Could not listen on port
java.net.BindException: Address already in use: JVM_Bind
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(Unknown Source)
at java.net.ServerSocket.bind(Unknown Source)
at java.net.ServerSocket.<init>(Unknown Source)
at java.net.ServerSocket.<init>(Unknown Source)
at com.youdevise.hudson.slavestatus.SocketHTTPListener.waitForConnection
(SlaveListener.java:129)
at com.youdevise.hudson.slavestatus.SlaveListener$1.run(SlaveListener.ja
va:63)
at com.youdevise.hudson.slavestatus.Daemon.go(Daemon.java:16)
at com.youdevise.hudson.slavestatus.SlaveListener.call(SlaveListener.jav
a:83)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecut
orService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source
)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:58)
at java.lang.Thread.run(Unknown Source)
I have inbound , outbound opne but no idea why I am facing this error..Any ideas?
Heres the output for suggested answer:
[root#ip-10-192-35-89 ~]# netstat -ntpl | grep 49187
tcp 0 0 :::49187 :::* LISTEN 1054/java
[root#ip-10-192-35-89 ~]# ps -ef|grep 1054
jenkins 1054 1 2 Aug14 ? 02:56:02 /etc/alternatives/java -Djava.awt.headless=true -Xmx2048m -XX:MaxPermSize=512m -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --httpPort=7777 --ajp13Port=8009 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
root 2483 2463 0 19:18 pts/0 00:00:00 grep 1054
From the following lines, it seems some process is already using the port 49187:
SEVERE: Could not listen on port
java.net.BindException: Address already in use: JVM_Bind
Just run the following command to fetch the process id:
netstat -ntpl | grep 49187
From the output, fetch the process-id and then run the following command to see which process is using the port 49187. You can then kill that process and try connecting to JNLP slave agent.
ps -ef | grep PID_from_above_output
For example,
[root#jenkins gc]# netstat -ntpl | grep 1569
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1569/sendmail
[root#jenkins gc]#
[root#jenkins gc]# ps -ef | grep 1569
root 1569 1 0 Jun26 ? 00:01:13 sendmail: accepting connections
root 8083 8059 0 21:35 pts/0 00:00:00 grep 1569
As you can see, sendmail program was using port 1569 to listen to requests.
When i got into the issues where it says that
JNLP Port is already in use. I tried grepping and doing a netstat on jenkins master for our JNLP port, i never got any process running on that particular port. Seems jenkins is storing the JNLP port information somewhere and not sure of it.
What i did was to change the JNLP port to some other port, saved the configuration, tried running the slave command, then updated the JNLP port with the same old one again and restarted the jnlp slave again. It works with out any issues now. Apparently the problem occurred when we restarted our jenkins master.