I have the following setup:
Keycloak running in docker, public interface mapped to 127.0.0.1:8180, internal keycloak-n:8080
Quarkus running in docker, public interface mapped to 127.0.0.1:8080
Both run in the same docker network and can communicate
An external AutzClient (not in docker) that uses the token communicate with quarkus
Everything works if client and quarkus are outside of Docker and communicate with keycloak via the same interface. As soon as quarkus is in docker, I can't get it to work.
I've tried many changes so far. On keycloak I set the frontendUrl with /subsystem=keycloak-server/spi=hostname/provider=default:write-attribute(name=properties.frontendUrl="http://127.0.0.1:8180/auth"
My current quarkus config (oidc part) looks like:
# OIDC Configuration
quarkus.oidc.auth-server-url=http://keycloak-n:8080/auth/realms/quarkus
quarkus.oidc.client-id=backend-service
quarkus.oidc.credentials.secret=85174256-b231-4385-9fa9-257dd0d27bf0
quarkus.oidc.token.lifespan-grace=20
quarkus.oidc.introspection-path=.well-known/openid-configuration
quarkus.oidc.jwks-path=.well-known/jwks.json
quarkus.oidc.token.issuer=http://127.0.0.1:8180/auth/realms/quarkus
# Enable Policy Enforcement
quarkus.keycloak.policy-enforcer.enable=true
If I remove the token issuer, I get from vertx a issuer validation failed. With the current configuration the initial auth works, but than I get a Connection refused (Connection refused) from PolicyEnforcer, because it tries to communicate with 127.0.0.1. Stacktrace is:
2020-08-03 05:43:27,933 DEBUG [org.apa.htt.imp.con.tsc.ConnPoolByRoute] (executor-thread-1) Releasing connection [{}->http://keycloak-n:8080][null]
2020-08-03 05:43:27,933 DEBUG [org.apa.htt.imp.con.tsc.ConnPoolByRoute] (executor-thread-1) Pooling connection [{}->http://keycloak-n:8080][null]; keep alive indefinitely
2020-08-03 05:43:27,933 DEBUG [org.apa.htt.imp.con.tsc.ConnPoolByRoute] (executor-thread-1) Notifying no-one, there are no waiting threads
2020-08-03 05:43:27,944 DEBUG [org.apa.htt.imp.con.tsc.ThreadSafeClientConnManager] (executor-thread-1) Get connection: {}->http://127.0.0.1:8180, timeout = 0
2020-08-03 05:43:27,944 DEBUG [org.apa.htt.imp.con.tsc.ConnPoolByRoute] (executor-thread-1) [{}->http://127.0.0.1:8180] total kept alive: 1, total issued: 0, total allocated: 1 out of 20
2020-08-03 05:43:27,944 DEBUG [org.apa.htt.imp.con.tsc.ConnPoolByRoute] (executor-thread-1) No free connections [{}->http://127.0.0.1:8180][null]
2020-08-03 05:43:27,944 DEBUG [org.apa.htt.imp.con.tsc.ConnPoolByRoute] (executor-thread-1) Available capacity: 20 out of 20 [{}->http://127.0.0.1:8180][null]
2020-08-03 05:43:27,944 DEBUG [org.apa.htt.imp.con.tsc.ConnPoolByRoute] (executor-thread-1) Creating new connection [{}->http://127.0.0.1:8180]
2020-08-03 05:43:27,944 DEBUG [org.apa.htt.imp.con.DefaultClientConnectionOperator] (executor-thread-1) Connecting to 127.0.0.1:8180
2020-08-03 05:43:27,945 DEBUG [org.apa.htt.imp.con.DefaultClientConnection] (executor-thread-1) Connection org.apache.http.impl.conn.DefaultClientConnection#6ba49b73 closed
2020-08-03 05:43:27,946 DEBUG [org.apa.htt.imp.con.DefaultClientConnection] (executor-thread-1) Connection org.apache.http.impl.conn.DefaultClientConnection#6ba49b73 shut down
2020-08-03 05:43:27,946 DEBUG [org.apa.htt.imp.con.tsc.ThreadSafeClientConnManager] (executor-thread-1) Released connection is not reusable.
2020-08-03 05:43:27,946 DEBUG [org.apa.htt.imp.con.tsc.ConnPoolByRoute] (executor-thread-1) Releasing connection [{}->http://127.0.0.1:8180][null]
2020-08-03 05:43:27,946 DEBUG [org.apa.htt.imp.con.DefaultClientConnection] (executor-thread-1) Connection org.apache.http.impl.conn.DefaultClientConnection#6ba49b73 closed
2020-08-03 05:43:27,946 DEBUG [org.apa.htt.imp.con.tsc.ConnPoolByRoute] (executor-thread-1) Notifying no-one, there are no waiting threads
2020-08-03 05:43:27,947 ERROR [org.key.ada.aut.PolicyEnforcer] (executor-thread-1) Could not lazy load resource with path [/hello/find/1] from server: java.lang.RuntimeException: Could not find resource
at org.keycloak.authorization.client.util.Throwables.retryAndWrapExceptionIfNecessary(Throwables.java:91)
at org.keycloak.authorization.client.resource.ProtectedResource.find(ProtectedResource.java:232)
at org.keycloak.authorization.client.resource.ProtectedResource.findByMatchingUri(ProtectedResource.java:291)
at org.keycloak.adapters.authorization.PolicyEnforcer$PathConfigMatcher.matches(PolicyEnforcer.java:268)
at org.keycloak.adapters.authorization.AbstractPolicyEnforcer.getPathConfig(AbstractPolicyEnforcer.java:351)
at org.keycloak.adapters.authorization.AbstractPolicyEnforcer.authorize(AbstractPolicyEnforcer.java:72)
at io.quarkus.keycloak.pep.runtime.KeycloakPolicyEnforcerAuthorizer.apply(KeycloakPolicyEnforcerAuthorizer.java:45)
at io.quarkus.keycloak.pep.runtime.KeycloakPolicyEnforcerAuthorizer.apply(KeycloakPolicyEnforcerAuthorizer.java:29)
at io.quarkus.vertx.http.runtime.security.HttpAuthorizer$1$1$1.run(HttpAuthorizer.java:68)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at java.base/java.lang.Thread.run(Thread.java:834)
at org.jboss.threads.JBossThread.run(JBossThread.java:479)
Caused by: java.lang.RuntimeException: Error executing http method [GET]. Response : null
at org.keycloak.authorization.client.util.HttpMethod.execute(HttpMethod.java:106)
at org.keycloak.authorization.client.util.HttpMethodResponse$3.execute(HttpMethodResponse.java:68)
at org.keycloak.authorization.client.resource.ProtectedResource$5.call(ProtectedResource.java:226)
at org.keycloak.authorization.client.resource.ProtectedResource$5.call(ProtectedResource.java:222)
at org.keycloak.authorization.client.resource.ProtectedResource.find(ProtectedResource.java:230)
... 15 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:609)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:134)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:605)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:440)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.keycloak.authorization.client.util.HttpMethod.execute(HttpMethod.java:84)
... 19 more
2020-08-03 05:43:27,951 DEBUG [org.key.ada.aut.AbstractPolicyEnforcer] (executor-thread-1) Checking permissions for path [http://127.0.0.1:8080/hello/find/1] with config [null].
2020-08-03 05:43:27,951 DEBUG [org.key.ada.aut.AbstractPolicyEnforcer] (executor-thread-1) Could not find a configuration for path [/hello/find/1]
Is there any real example on how to configure such a scenario? I already tried to set the frontendUrl to the internal address, that actually works for the runtime, but the web frontend is no longer accessible.
UPDATE:
From front end code (abbreviated):
java.io.InputStream stream = Thread.currentThread().getContextClassLoader()
.getResourceAsStream("META-INF/keycloak.json");
auth=AuthzClient.create(stream);
response = auth.obtainAccessToken(user, password);
final String accessToken = response.getToken();
...
requestContext.getHeaders().add(HttpHeaders.AUTHORIZATION, AUTH_HEADER_PREFIX + accessToken);
...
and config in keycloak.json is
{
"realm": "quarkus",
"auth-server-url": "http://localhost:8180/auth/",
"ssl-required": "external",
"resource": "backend-service",
"verify-token-audience": true,
"credentials": {
"secret": "85174256-b231-4385-9fa9-257dd0d27bf0"
},
"confidential-port": 0,
"policy-enforcer": {}
}
Many thanks
So the following setup works for me:
frontendUrl: external-docker-ip --> NOT localhost!
set in jboss cli by e.g.:
/subsystem=keycloak-server/spi=hostname/provider=default:write-attribute(name=properties.frontendUrl,value="http://172.20.48.1:8180/auth")
##quarkus config
quarkus.oidc.auth-server-url=http://internal_keycloak_docker_IP:8080/auth/realms/quarkus
quarkus.oidc.token.issuer=http://external-docker-ip:8180/auth/realms/quarkus
##client json file
"auth-server-url": "http://external-docker-ip:8180/auth/"
Related
I'm having trouble with stopping infinite retrying when rabbitmq server is down. I tried using this code snippet
#Bean(name = "rabbitListenerContainerFactory")
public SimpleRabbitListenerContainerFactory simpleRabbitListenerContainerFactory(
SimpleRabbitListenerContainerFactoryConfigurer configurer,
ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
BackOff recoveryBackOff = new FixedBackOff(5000, 1);
factory.setRecoveryBackOff(recoveryBackOff);
return factory;
}
But I am still getting endless loop of retrying
17:49:47,417 DEBUG o.s.a.r.l.BlockingQueueConsumer [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#13-1] Starting consumer Consumer#4f35699: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2020-08-31 17:49:49,431 WARN o.s.a.r.l.SimpleMessageListenerContainer [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#34-2] stopping container - restart recovery attempts exhausted
2020-08-31 17:49:49,431 DEBUG o.s.a.r.l.SimpleMessageListenerContainer [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#34-2] Shutting down Rabbit listener container
2020-08-31 17:49:49,431 INFO o.s.a.r.c.CachingConnectionFactory [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#13-1] Attempting to connect to: [localhost:5672]
2020-08-31 17:49:49,431 INFO o.s.a.r.l.SimpleMessageListenerContainer [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#34-2] Waiting for workers to finish.
2020-08-31 17:49:49,431 INFO o.s.a.r.l.SimpleMessageListenerContainer [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#34-2] Successfully waited for workers to finish.
2020-08-31 17:49:49,431 DEBUG o.s.a.r.l.SimpleMessageListenerContainer [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#34-2] Cancelling Consumer#e56de36: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2020-08-31 17:49:49,431 DEBUG o.s.a.r.l.BlockingQueueConsumer [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#34-2] Closing Rabbit Channel: null
2020-08-31 17:49:51,434 DEBUG o.s.a.r.l.SimpleMessageListenerContainer [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#13-1] Recovering consumer in 5000 ms.
2020-08-31 17:49:51,434 DEBUG o.s.a.r.l.SimpleMessageListenerContainer [main] Starting Rabbit listener container.
2020-08-31 17:49:51,434 INFO o.s.a.r.c.CachingConnectionFactory [main] Attempting to connect to: [localhost:5672]
2020-08-31 17:49:53,485 INFO o.s.a.r.l.SimpleMessageListenerContainer [main] Broker not available; cannot force queue declarations during start: java.net.ConnectException: Connection refused: connect
2020-08-31 17:49:53,485 INFO o.s.a.r.c.CachingConnectionFactory [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#35-1] Attempting to connect to: [localhost:5672]
2020-08-31 17:49:55,518 ERROR o.s.a.r.l.SimpleMessageListenerContainer [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#35-1] Failed to check/redeclare auto-delete queue(s).
org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused: connect
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:62) ~[spring-rabbit-2.1.7.RELEASE.jar:2.1.7.RELEASE]
What I am trying to achieve, is that after one attempt of connecting, stop trying
Edit 1
I have my whole consumer side config here, I can't seem to find where another configurable container factory could be. The thing is, if I have fixed back off at for example 3000 ms, the message changes to Recovering consumer in 3000 ms.
#Configuration
#EnableRabbit
#AllArgsConstructor
public class RabbitConfig {
#Bean
public MessageConverter jsonMessageConverter() {
ObjectMapper jsonObjectMapper = new ObjectMapper();
jsonObjectMapper.registerModule(new JavaTimeModule());
return new Jackson2JsonMessageConverter(jsonObjectMapper);
}
#Bean
public RabbitErrorHandler rabbitExceptionHandler() {
return new RabbitErrorHandler();
}
#Bean(name = "rabbitListenerContainerFactory")
public SimpleRabbitListenerContainerFactory simpleRabbitListenerContainerFactory(
SimpleRabbitListenerContainerFactoryConfigurer configurer,
ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
BackOff recoveryBackOff = new FixedBackOff(5000, 1);
factory.setRecoveryBackOff(recoveryBackOff);
return factory;
}
}
I'm using version 2.1.7
I just copied your code and it worked as expected:
2020-08-31 11:58:08.997 INFO 94891 --- [ main] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: [localhost:5672]
2020-08-31 11:58:09.002 INFO 94891 --- [ main] o.s.a.r.l.SimpleMessageListenerContainer : Broker not available; cannot force queue declarations during start: java.net.ConnectException: Connection refused (Connection refused)
2020-08-31 11:58:09.006 INFO 94891 --- [ntContainer#0-1] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: [localhost:5672]
2020-08-31 11:58:09.014 INFO 94891 --- [ main] com.example.demo.So63673274Application : Started So63673274Application in 0.929 seconds (JVM running for 1.39)
2020-08-31 11:58:14.091 WARN 94891 --- [ntContainer#0-1] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2020-08-31 11:58:14.093 INFO 94891 --- [ntContainer#0-1] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#6f2cb653: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2020-08-31 11:58:14.094 INFO 94891 --- [ntContainer#0-2] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: [localhost:5672]
2020-08-31 11:58:14.095 WARN 94891 --- [ntContainer#0-2] o.s.a.r.l.SimpleMessageListenerContainer : stopping container - restart recovery attempts exhausted
2020-08-31 11:58:14.095 INFO 94891 --- [ntContainer#0-2] o.s.a.r.l.SimpleMessageListenerContainer : Waiting for workers to finish.
2020-08-31 11:58:14.096 INFO 94891 --- [ntContainer#0-2] o.s.a.r.l.SimpleMessageListenerContainer : Successfully waited for workers to finish.
I see you have
stopping container - restart recovery attempts exhausted
too.
Perhaps the listener that keeps trying is from a different container factory?
Recovering consumer in 5000 ms.
Nebula Graph fails on CentOS 6.5, the error message is as follows:
# storage log
Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
# meta log
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0415 22:32:38.944437 15532 AsyncServerSocket.cpp:762] failed to set SO_REUSEPORT on async server socket Protocol not available
E0415 22:32:38.945001 15510 ThriftServer.cpp:440] Got an exception while setting up the server: 92failed to bind to async server socket: [::]:0: Protocol not available
E0415 22:32:38.945057 15510 RaftexService.cpp:90] Setup the Raftex Service failed, error: 92failed to bind to async server socket: [::]:0: Protocol not available
E0415 22:32:38.949586 15463 NebulaStore.cpp:47] Start the raft service failed
E0415 22:32:38.949597 15463 MetaDaemon.cpp:88] Nebula store init failed
E0415 22:32:38.949796 15463 MetaDaemon.cpp:215] Init kv failed!
Nebula service status is as follows:
[root#redhat6 scripts]# ./nebula.service status all
[WARN] The maximum files allowed to open might be too few: 1024
[INFO] nebula-metad: Exited
[INFO] nebula-graphd: Exited
[INFO] nebula-storaged: Running as 15547, Listening on 44500
Reason for error: CentOS 6.5 system kernel version is 2.6.32, which is less than 3.9. However, SO_REUSEPORT only supports Linux 3.9 and above.
Upgrading the system to CentOS 7.5 can solve the problem by itself.
Windows 10 setup :
Thingsboard server running as local service on windows
Thingsboard.yml mqtt parameters
MQTT server parameters
mqtt:
bind_address: "${MQTT_BIND_ADDRESS:0.0.0.0}"
bind_port: "${MQTT_BIND_PORT:1883}"
adaptor: "${MQTT_ADAPTOR_NAME:JsonMqttAdaptor}"
timeout: "${MQTT_TIMEOUT:10000}"
Thingsboard gateway service running as local service on windows
> tb-gateway.yml mqtt parameters
> mqtt:
> enabled: true
> configuration: mqtt-config.json
configuration file of mqtt is set by default,below.
mqtt-config.json mqtt parameters
"brokers": [
{
"host": "localhost",
"port": 1883,
"ssl": false,
"retryInterval": 3000,
"credentials": {
"type": "anonymous"
},
These are the only two services running on my laptop, I published a mqtt message as follows, per docs :
mosquitto_pub -h localhost -p 1883 -u "XXXXXXXX" -t "sensors" -m '{"serialNumber":"TB-GW-SN-001","model":"TB-GW-T1000","temperature":35.2}'
I see errors in both the logs.
thingsboard.log
2018-01-10 20:14:56,174 [nioEventLoopGroup-6-11] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt815] Processing connect msg for client: efd91958-ba8f-480a-9a56-ad9d5588c8c7!
2018-01-10 20:14:56,177 [nioEventLoopGroup-6-12] INFO o.t.s.t.mqtt.MqttTransportHandler - [127.0.0.1:51192] Invalid message received
2018-01-10 20:14:59,183 [nioEventLoopGroup-6-1] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt817] Processing connect msg for client: efd91958-ba8f-480a-9a56-ad9d5588c8c7!
2018-01-10 20:14:59,188 [nioEventLoopGroup-6-2] INFO o.t.s.t.mqtt.MqttTransportHandler - [127.0.0.1:51194] Invalid message received
2018-01-10 20:15:02,193 [nioEventLoopGroup-6-3] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt819] Processing connect msg for client: efd91958-ba8f-480a-9a56-ad9d5588c8c7!
2018-01-10 20:15:02,197 [nioEventLoopGroup-6-4] INFO o.t.s.t.mqtt.MqttTransportHandler - [127.0.0.1:51196] Invalid message received
error in thingsboard gateway is strange, rather.
tb-gateway.log
2018-01-10 20:14:59,191 [main] WARN o.t.g.e.m.client.MqttBrokerMonitor - [localhost:1883] MQTT broker connection failed!
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:164)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readByte(Unknown Source)
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:92)
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:116)
... 1 common frames omitted
2018-01-10 20:15:02,198 [main] WARN o.t.g.e.m.client.MqttBrokerMonitor - [localhost:1883] MQTT broker connection failed!
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:164)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readByte(Unknown Source)
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:92)
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:116)
... 1 common frames omitted
What am I doing wrong? I have mosquitto installed locally, hence used the mosquitto_pub tool to publish the messages.
Any clues, folks?
All is well.
As https://stackoverflow.com/users/3203737/andrew pointed out, I wasnt running mosquitto and the ports were the same. I cleaned up the mess and the devices were registered perfectly using TB-gateway, in the dashboard.
I am trying to connect opscenter to DSE cluster, I tried and verified the same configurations with in the other environments (DEV, STAGING) but while I am trying to do the same on Prod, I am getting these error on agent.log file
INFO [async-dispatch-64] 2017-12-14 18:29:24,728 Starting system.
INFO [async-dispatch-64] 2017-12-14 18:29:24,730 Starting
JMXComponent
WARN [async-dispatch-64] 2017-12-14 18:29:24,732 Exception while
processing JMX data: java.lang.SecurityException: Authentication
failed! Credentials required
ERROR [async-dispatch-64] 2017-12-14 18:29:24,732 Security failure
connecting to JMX: Authentication failed! Credentials required
INFO [async-dispatch-64] 2017-12-14 18:29:24,733 Starting
StompComponent
INFO [async-dispatch-64] 2017-12-14 18:29:24,733 SSL communication is
disabled
INFO [async-dispatch-64] 2017-12-14 18:29:24,733 Creating stomp
connection to x.y.z.x:61620
ERROR [async-dispatch-64] 2017-12-14 18:29:24,736 Dec 14, 2017
6:29:24 PM org.jgroups.client.StompConnection connect
INFO: Connected to x.y.z.x:1234
WARN [async-dispatch-64] 2017-12-14 18:29:29,738 Attempted to ping
opscenterd on stomp but did not receive a reply in time, will retry
again later.
ERROR [StompConnection receiver] 2017-12-14 18:29:29,740 Dec 14, 2017
6:29:29 PM org.jgroups.client.StompConnection run
SEVERE: JGRP000112: Connection closed unexpectedly:
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:224)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.jgroups.util.Util.readLine(Util.java:2825)
at org.jgroups.protocols.STOMP.readFrame(STOMP.java:240)
at org.jgroups.client.StompConnection.run(StompConnection.java:274)
at java.lang.Thread.run(Thread.java:745)
INFO [async-dispatch-64] 2017-12-14 18:29:29,742 Starting
JMXComponent
WARN [async-dispatch-64] 2017-12-14 18:29:29,744 Exception while
processing JMX data: java.lang.SecurityException: Authentication
failed! Credentials required
ERROR [async-dispatch-64] 2017-12-14 18:29:29,744 Security failure
connecting to JMX: Authentication failed! Credentials required
INFO [async-dispatch-64] 2017-12-14 18:29:29,744 Starting
JMXComponent
WARN [async-dispatch-64] 2017-12-14 18:29:29,746 Exception while
processing JMX data: java.lang.SecurityException: Authentication
failed! Credentials required
ERROR [async-dispatch-64] 2017-12-14 18:29:29,746 Security failure
connecting to JMX: Authentication failed! Credentials required
Also the opscenterd.log show no errors for the cluster.
The configuration that i used are below.
I have used address.yaml
stomp_interface: x.x.x.x
use_ssl: 0
metrics_enabled: 1
hosts: [x.x.x.x, x.x.x.x]
storage_keyspace: opscenter_abc
cluster_name.conf
[jmx]
port = 7199
password = abc
username = abc
[cassandra]
seed_hosts = x.x.x.x , x.x.x.x , x.x.x.x
api_port = 9160
cql_port = 9042
password = abc
username = fabe
[storage_cassandra]
seed_hosts = x.x.x.x, x.x.x.x
api_port = 9160
cql_port = 9042
keyspace = opscenter_abc
[cassandra_metrics]
5min_ttl = 2419200
[cluster_display_options]
display_name = badkfj
I have been using flume for a while now, I have got agent and collector running on same machine.
Configuration
agent: exec("/usr/bin/tail -n +0 -F /path/to/file") | agentE2ESink("hostname", 35855)
collector: collectorSource(35855) | collector(10000) { collectorSink("/hdfs/path/to/sink","name") }
Facing issues in the agent node:
2012-06-04 19:13:33,625 [naive file wal consumer-27] INFO debug.InsistentOpenDecorator: open attempt 0 failed, backoff (1000ms): Failed to open thrift event sink to hostname:35855 : java.net.ConnectException: Connection refused
2012-06-04 19:13:34,625 [logicalNode hostname-19] ERROR connector.DirectDriver: Expected ACTIVE but timed out in state OPENING
2012-06-04 19:13:34,632 [naive file wal consumer-27] INFO debug.InsistentOpenDecorator: open attempt 1 failed, backoff (2000ms): Failed to open thrift event sink to hostname:35855 : java.net.ConnectException: Connection refused
2012-06-04 19:13:36,635 [naive file wal consumer-27] INFO debug.InsistentOpenDecorator: open attempt 2 failed, backoff (4000ms): Failed to open thrift event sink to hostname:35855 : java.net.ConnectException: Connection refused
and then empty ACKs will be sent continuously
2012-06-04 19:19:56,960 [Roll-TriggerThread-0] INFO endtoend.AckListener$Empty: Empty Ack Listener began 20120604-191956958+0530.881565921235084.00000026
2012-06-04 19:20:07,043 [Roll-TriggerThread-0] INFO hdfs.SeqfileEventSink: closed /tmp/flume-user1/agent/hostname/writing/20120604-191956958+0530.881565921235084.00000026
I dont understand why the connection is refused. Are there any system level changes that needs to be done ?
Note: the collector is listening to the port but agent is unable to send data through the 35855 port.
Can anyone help me with this problem.
Thanks
If you are running both the agent and the collector on the same box, you should be using localhost as the address.
agentE2ESink("localhost", 35855)