NATS Server error:
node2-c1_1 | [1] 2021/12/13 15:35:15.464884 [INF] Creating MQTT streams/consumers with replicas 1 for account "ADKPM577BVYF7QXUNN6YOFEO4CJUQOPTD3WDEIFNXLW264566UABY3QG"
node2-c1_1 | [1] 2021/12/13 15:35:19.470607 [WRN] 172.18.0.1:60930 - mid:36 - "mqttClient-MQTT_3_1_1-86fbcb3e-3d62-4bef-9ac3-684f54945852" - Readloop processing time: 4.0328908s
node2-c1_1 | [1] 2021/12/13 15:35:19.470719 [ERR] 172.18.0.1:60930 - mid:36 - "mqttClient-MQTT_3_1_1-86fbcb3e-3d62-4bef-9ac3-684f54945852" - unable to connect: create messages stream for account "ADKPM577BVYF7QXUNN6YOFEO4CJUQOPTD3WDEIFNXLW264566UABY3QG": timeout for request type "SC" on "$JS.API.STREAM.CREATE.$MQTT_msgs" (reply="$MQTT.JSA.n50ntl9W.SC.FocGVk4ZKwLQbRYWqJEJUk")
node2-c1_1 | [1] 2021/12/13 15:35:19.470741 [DBG] 172.18.0.1:60930 - mid:36 - "mqttClient-MQTT_3_1_1-86fbcb3e-3d62-4bef-9ac3-684f54945852" - Client connection closed: Protocol Violation
Client Error:
CLIENT mqttClient-MQTT_3_1_1-86fbcb3e-3d62-4bef-9ac3-684f54945852: sending CONNECT with Mqtt3Connect: MqttConnect{keepAlive=60, cleanSession=true, simpleAuth=MqttSimpleAuth{username and password}}
PUBLISH: com.hivemq.client.mqtt.exceptions.ConnectionClosedException: Server closed connection without DISCONNECT.
at com.hivemq.client.internal.mqtt.MqttBlockingClient.connect(MqttBlockingClient.java:91)
at com.hivemq.client.internal.mqtt.mqtt3.Mqtt3BlockingClientView.connect(Mqtt3BlockingClientView.java:69)
at com.hivemq.cli.mqtt.MqttClientExecutor.mqtt3Connect(MqttClientExecutor.java:104)
at com.hivemq.cli.mqtt.AbstractMqttClientExecutor.connectMqtt3Client(AbstractMqttClientExecutor.java:266)
at com.hivemq.cli.mqtt.AbstractMqttClientExecutor.connect(AbstractMqttClientExecutor.java:203)
at com.hivemq.cli.mqtt.MqttClientExecutor.connect(MqttClientExecutor.java:67)
at com.hivemq.cli.mqtt.AbstractMqttClientExecutor.getMqttClientFromCacheOrConnect(AbstractMqttClientExecutor.java:434)
at com.hivemq.cli.mqtt.AbstractMqttClientExecutor.subscribe(AbstractMqttClientExecutor.java:84)
at com.hivemq.cli.mqtt.MqttClientExecutor.subscribe(MqttClientExecutor.java:67)
at com.hivemq.cli.commands.cli.SubscribeCommand.run(SubscribeCommand.java:114)
at picocli.CommandLine.executeUserObject(CommandLine.java:1729)
at picocli.CommandLine.access$900(CommandLine.java:145)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2101)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2068)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1935)
at picocli.CommandLine.execute(CommandLine.java:1864)
at com.hivemq.cli.MqttCLIMain.main(MqttCLIMain.java:73)
PUBLISH: Server closed connection without DISCONNECT.
Related
We are using testcontainers to instantiate Kafka containers for testing an app that creates a kstream StateStore and kafka consumer to process some messages. The test goes fine and we could both produce and consume messages. But these tests continuously print annoying WARN messages -
(kafka-admin-client-thread)([]) WARN - NetworkClient - [AdminClient
clientId=test_clientId] Connection to node 1
(localhost/127.0.0.1:55304) could not be established. Broker may not
be available.
It seems the AdminClient is scheduled to poll every second and this creates substantial log files.
On enabling DEBUG logs, I observe the error
[2022-09-16T22:48:11,747Z](test_x-GlobalStreamThread)([]) DEBUG - NetworkClient - [Consumer clientId=test_x-global-consumer, groupId=null] Node 1 disconnected.
[2022-09-16T22:48:11,747Z](test_x-GlobalStreamThread)([]) WARN - NetworkClient - [Consumer clientId=test_x-global-consumer, groupId=null] Connection to node 1 (localhost/127.0.0.1:55304) could not be established. Broker may not be available.
[2022-09-16T22:48:11,747Z](test_x-GlobalStreamThread)([]) DEBUG - ConsumerNetworkClient - [Consumer clientId=test_x-global-consumer, groupId=null] Cancelled request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=test_x-global-consumer, correlationId=11) due to node 1 being disconnected
[2022-09-16T22:48:11,747Z](test_x-GlobalStreamThread)([]) INFO - FetchSessionHandler - [Consumer clientId=test_x-global-consumer, groupId=null] Error sending fetch request (sessionId=1421844635, epoch=INITIAL) to node 1:
org.apache.kafka.common.errors.DisconnectException: null
[2022-09-16T22:48:11,792Z](kafka-producer-network-thread | producer-1)([]) DEBUG - NetworkClient - [Producer clientId=producer-1] Initialize connection to node localhost:55304 (id: 1 rack: null) for sending metadata request
[2022-09-16T22:48:11,792Z](kafka-admin-client-thread | adminclient-1)([]) DEBUG - NetworkClient - [AdminClient clientId=adminclient-1] Initiating connection to node localhost:55304 (id: 1 rack: null) using address localhost/127.0.0.1
[2022-09-16T22:48:11,792Z](kafka-producer-network-thread | producer-1)([]) DEBUG - NetworkClient - [Producer clientId=producer-1] Initiating connection to node localhost:55304 (id: 1 rack: null) using address localhost/127.0.0.1
[2022-09-16T22:48:11,793Z](kafka-producer-network-thread | producer-1)([]) DEBUG - Selector - [Producer clientId=producer-1] Connection with localhost/127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:219)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:530)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:325)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240)
at java.base/java.lang.Thread.run(Thread.java:833)
[2022-09-16T22:48:11,793Z](kafka-admin-client-thread | adminclient-1)([]) DEBUG - Selector - [AdminClient clientId=adminclient-1] Connection with localhost/127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:219)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:530)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1293)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
at java.base/java.lang.Thread.run(Thread.java:833)
[2022-09-16T22:48:11,793Z](kafka-producer-network-thread | producer-1)([]) DEBUG - NetworkClient - [Producer clientId=producer-1] Node 1 disconnected.
[2022-09-16T22:48:11,793Z](kafka-admin-client-thread | adminclient-1)([]) DEBUG - NetworkClient - [AdminClient clientId=adminclient-1] Node 1 disconnected.
Similar logs are also coming from GlobalStreamThread
[2022-09-16T22:48:11,746Z](test_x-GlobalStreamThread)([]) DEBUG - NetworkClient - [Consumer clientId=test_x-global-consumer, groupId=null] Initiating connection to node localhost:55304 (id: 1 rack: null) using address localhost/127.0.0.1
[2022-09-16T22:48:11,747Z](test_x-GlobalStreamThread)([]) DEBUG - Selector - [Consumer clientId=test_x-global-consumer, groupId=null] Connection with localhost/127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:219)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:530)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1296)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
at org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:237)
at org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:284)
Docker port mapping looks like
0.0.0.0:55306->2181/tcp, 0.0.0.0:55305->9092/tcp, 0.0.0.0:55304->9093/tcp
Testcontainer is started as
KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:6.2.1"))
.withNetwork(Network.SHARED)
Testcontainer maps KAFKA_LISTENERS here
https://github.com/testcontainers/testcontainers-java/blob/master/modules/kafka/src/main/java/org/testcontainers/containers/KafkaContainer.java#L49
Could you help me understand what is happening here and how to provide the correct port mappings via testcontainers for these threads to work fine?
I'm running Fabric v2.3 test network, and I'm trying to take a snapshot of the ledger.
When I run:
# peer snapshot submitrequest -c mychannel -b 0 --peerAddress peer0.org1.example.com:7051 --tlsRootCertFile /opt/gopath/src/github.com/hyperledger/fabric/peer/o
rganizations/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt
I obtain:
Error: failed to retrieve snapshot client: snapshot client failed to connect to peer0.org1.example.com:7051: failed to create new connection: context deadline exceeded
The cli container can connect to peer1. Inspecting the orderer logs, it seems that a connection to 172.19.0.1 (on variable ports) is not successful: https://gist.github.com/RafaelAPB/962477f9bab5f34076add1a3b9f63588
Part of it:
2021-10-04 00:31:54.519 UTC [common.deliver] Handle -> WARN 066 Error reading from 172.19.0.8:53714: rpc error: code = Canceled desc = context canceled
By inspecting the logs of peer0.org1, it really seems that the problem is the connection with the orderer (in particular the TLS handshake): https://gist.github.com/RafaelAPB/6adf0c6e702387e4e0edd8de0dcf0a76
Part of it:
2021-10-04 00:52:03.595 UTC [comm.grpc.server] 1 -> INFO e6d unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.19.0.1:52374 grpc.code=OK grpc.call_duration=4.028085ms
2021-10-04 00:52:04.293 UTC [core.comm] ServerHandshake -> ERRO e6e Server TLS handshake failed in 33.508µs with error tls: first record does not look like a TLS handshake server=PeerServer remoteaddress=172.19.0.8:58532
2021-10-04 00:52:04.419 UTC [gossip.privdata] StoreBlock -> INFO e6f Received block [505] from buffer channel=mychannel
No configurations were added to the configtx.yaml regarding the snapshotting feature.
Any ideas on how to run the snapshot command successfully?
Windows 10 setup :
Thingsboard server running as local service on windows
Thingsboard.yml mqtt parameters
MQTT server parameters
mqtt:
bind_address: "${MQTT_BIND_ADDRESS:0.0.0.0}"
bind_port: "${MQTT_BIND_PORT:1883}"
adaptor: "${MQTT_ADAPTOR_NAME:JsonMqttAdaptor}"
timeout: "${MQTT_TIMEOUT:10000}"
Thingsboard gateway service running as local service on windows
> tb-gateway.yml mqtt parameters
> mqtt:
> enabled: true
> configuration: mqtt-config.json
configuration file of mqtt is set by default,below.
mqtt-config.json mqtt parameters
"brokers": [
{
"host": "localhost",
"port": 1883,
"ssl": false,
"retryInterval": 3000,
"credentials": {
"type": "anonymous"
},
These are the only two services running on my laptop, I published a mqtt message as follows, per docs :
mosquitto_pub -h localhost -p 1883 -u "XXXXXXXX" -t "sensors" -m '{"serialNumber":"TB-GW-SN-001","model":"TB-GW-T1000","temperature":35.2}'
I see errors in both the logs.
thingsboard.log
2018-01-10 20:14:56,174 [nioEventLoopGroup-6-11] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt815] Processing connect msg for client: efd91958-ba8f-480a-9a56-ad9d5588c8c7!
2018-01-10 20:14:56,177 [nioEventLoopGroup-6-12] INFO o.t.s.t.mqtt.MqttTransportHandler - [127.0.0.1:51192] Invalid message received
2018-01-10 20:14:59,183 [nioEventLoopGroup-6-1] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt817] Processing connect msg for client: efd91958-ba8f-480a-9a56-ad9d5588c8c7!
2018-01-10 20:14:59,188 [nioEventLoopGroup-6-2] INFO o.t.s.t.mqtt.MqttTransportHandler - [127.0.0.1:51194] Invalid message received
2018-01-10 20:15:02,193 [nioEventLoopGroup-6-3] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt819] Processing connect msg for client: efd91958-ba8f-480a-9a56-ad9d5588c8c7!
2018-01-10 20:15:02,197 [nioEventLoopGroup-6-4] INFO o.t.s.t.mqtt.MqttTransportHandler - [127.0.0.1:51196] Invalid message received
error in thingsboard gateway is strange, rather.
tb-gateway.log
2018-01-10 20:14:59,191 [main] WARN o.t.g.e.m.client.MqttBrokerMonitor - [localhost:1883] MQTT broker connection failed!
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:164)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readByte(Unknown Source)
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:92)
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:116)
... 1 common frames omitted
2018-01-10 20:15:02,198 [main] WARN o.t.g.e.m.client.MqttBrokerMonitor - [localhost:1883] MQTT broker connection failed!
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:164)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readByte(Unknown Source)
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:92)
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:116)
... 1 common frames omitted
What am I doing wrong? I have mosquitto installed locally, hence used the mosquitto_pub tool to publish the messages.
Any clues, folks?
All is well.
As https://stackoverflow.com/users/3203737/andrew pointed out, I wasnt running mosquitto and the ports were the same. I cleaned up the mess and the devices were registered perfectly using TB-gateway, in the dashboard.
I am using Datastax enterprise edition and in cluster one is Hadoop/Hive .I am trying to connect to hive with datastax hive odbc connector.I am getting error like :
Connector Version: V1.0.0.1007
Running connectivity tests...
Attempting connection
Failed to establish connection
SQLSTATE: HY000[DataStax][Hardy] (22) Error from ThriftHiveClient: connect() failed: errno = 10061
TESTS COMPLETED WITH ERROR
The error 10061 means connection refused
Seems like you have not started the hive service on your Analytics node therefore nothing is listening on TCP 10000
Please login into one of your DSE Analytics node and execute:
dse hive --service hiveserver
Then try again your test from your Windows system
Source: http://www.datastax.com/documentation/datastax_enterprise/4.0/datastax_enterprise/ana/anaHivStrtSvr.html
I have been using flume for a while now, I have got agent and collector running on same machine.
Configuration
agent: exec("/usr/bin/tail -n +0 -F /path/to/file") | agentE2ESink("hostname", 35855)
collector: collectorSource(35855) | collector(10000) { collectorSink("/hdfs/path/to/sink","name") }
Facing issues in the agent node:
2012-06-04 19:13:33,625 [naive file wal consumer-27] INFO debug.InsistentOpenDecorator: open attempt 0 failed, backoff (1000ms): Failed to open thrift event sink to hostname:35855 : java.net.ConnectException: Connection refused
2012-06-04 19:13:34,625 [logicalNode hostname-19] ERROR connector.DirectDriver: Expected ACTIVE but timed out in state OPENING
2012-06-04 19:13:34,632 [naive file wal consumer-27] INFO debug.InsistentOpenDecorator: open attempt 1 failed, backoff (2000ms): Failed to open thrift event sink to hostname:35855 : java.net.ConnectException: Connection refused
2012-06-04 19:13:36,635 [naive file wal consumer-27] INFO debug.InsistentOpenDecorator: open attempt 2 failed, backoff (4000ms): Failed to open thrift event sink to hostname:35855 : java.net.ConnectException: Connection refused
and then empty ACKs will be sent continuously
2012-06-04 19:19:56,960 [Roll-TriggerThread-0] INFO endtoend.AckListener$Empty: Empty Ack Listener began 20120604-191956958+0530.881565921235084.00000026
2012-06-04 19:20:07,043 [Roll-TriggerThread-0] INFO hdfs.SeqfileEventSink: closed /tmp/flume-user1/agent/hostname/writing/20120604-191956958+0530.881565921235084.00000026
I dont understand why the connection is refused. Are there any system level changes that needs to be done ?
Note: the collector is listening to the port but agent is unable to send data through the 35855 port.
Can anyone help me with this problem.
Thanks
If you are running both the agent and the collector on the same box, you should be using localhost as the address.
agentE2ESink("localhost", 35855)