MQTT5 User Properties with Mosquitto Bridge - mqtt

I am running a local Mosquitto (MQTT) Broker that connects to a remote Mosquitto Broker using the build in MQTT Bridge functionality of Mosquitto. My mosquitto.conf looks like this:
# =================================================================
# Listeners
# =================================================================
listener 1883
# =================================================================
# Security
# =================================================================
allow_anonymous true
# =================================================================
# Bridges
# =================================================================
connection myConnectionName
address <<Remote Broker IP>>:1883
remote_username <<Remote Broker Username>>
remote_password <<Remote Broker Password>>
topic mytopic/# out 1 "" B2/
bridge_protocol_version mqttv50
cleansession false
bridge_attempt_unsubscribe true
upgrade_outgoing_qos true
max_queued_messages 5000
For testing I run a MqttPublisher using a C# console application which uses the MQTTnet library (version 3) and a MqttSubsbriber (also C# console application with MqttNet).
Now I want the Publisher to publish MQTT messages with User Properties (introduced by MQTT 5).
I build the message like this:
using MQTTnet;
using MQTTnet.Client;
using MQTTnet.Client.Options;
class Program
{
static void Main()
{
// Create a new MQTT client instance
var factory = new MqttFactory();
var mqttClient = factory.CreateMqttClient();
// Setup the options for the MQTT client
var options = new MqttClientOptionsBuilder()
.WithClientId("MqttPublisher")
.WithTcpServer("localhost", 1883)
.WithProtocolVersion(MQTTnet.Formatter.MqttProtocolVersion.V500)
.WithCleanSession()
.Build();
mqttClient.ConnectAsync(options).Wait();
var i = 0;
while (true)
{
Console.WriteLine("Client connected: " + mqttClient.IsConnected);
var message = new MqttApplicationMessageBuilder()
.WithTopic("mytopic/test")
.WithUserProperty("UPTest","Hi UP")
.WithPayload("Hello World: " + i)
.Build();
mqttClient.PublishAsync(message).Wait();
Console.WriteLine("Published message with payload: " + System.Text.Encoding.UTF8.GetString(message.Payload));
i++;
System.Threading.Thread.Sleep(1000);
}
mqttClient.DisconnectAsync().Wait();
}
}
With the subscriber (also with WithProtocolVersion(MQTTnet.Formatter.MqttProtocolVersion.V500) if I subscribe to the topic I get all the messages and I can read the MQTTnet.MqttApplicationMessage like shown in the following screenshot:
The messages are also published to the remote MQTT Broker due to the MQTT Bride configured. However, if I subscribe to the remote Broker with my MqttSubscriber, I am not getting the User Properties anymore:
Is there any way to configure the Mosquitto Bridge that also the user properties are send? I cant find a way and any help and comments are appreciated.
Thanks
Joshua

Using mosqutto 2.0.15 I have verified that MQTTv5 message properties are passed over the bridge.
Broker one.conf
listener 1883
allow_anonymous true
Broker two.conf
listener 1884
allow_anonymous true
connection one
address 127.0.0.1:1883
topic foo/# both 0
bridge_protocol_version mqttv50
Subscriber connected to Broker two
$ mosquitto_sub -t 'foo/#' -V mqttv5 -p 1884 -F "%t %P %p"
foo/bar foo:bar ben
Publisher connected to Broker one
$ mosquitto_pub -D PUBLISH user-property foo bar -t 'foo/bar' -m ben -p 1883
As you can see the the %P in the output format for the subscriber is outputting the user property foo with a value of bar when subscribed over the bridge.

Related

Python KafkaConsumer not connecting

Setup:
I have 3 docker containers
1) For Kafka
2) For Zookeeper
3) For JupyterLab
I setup networking between these containers and I see that kafka producer is able to run and produce the data.
KafkaProducer.ipynb
KAFKA_BROKER = ['172.20.0.2:9093']
from kafka import KafkaProducer
from kafka.errors import KafkaError
producer = KafkaProducer(bootstrap_servers=KAFKA_BROKER)
for _ in range(100):
print("sending")
producer.send('my-topic', key=b'foo', value=b'bar')
print("success")
Here the send() sends message 100 times.
KafkaConsumer.ipynb
KAFKA_BROKER = ['172.20.0.2:9093']
from kafka import KafkaConsumer
consumer = KafkaConsumer('my-topic',group_id='my-group',bootstrap_servers=KAFKA_BROKER)
print("Comm success")
for message in consumer:
# message value and key are raw bytes -- decode if necessary!
# e.g., for unicode: `message.value.decode('utf-8')`
print ("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,
message.offset, message.key,
message.value))
In the above consumer code the line print("Comm success") never gets gets executed. Based on producer code execution, the network is open and jupyter is able to talk to kafka broker. But, client is not able to connect to the same broker for data consumption. How can I start debugging this?
By default auto.offset.reset value is latest, so set it to earliest with new group.id
consumer = KafkaConsumer('my-topic',group_id='new-group',auto_offset_reset = 'earliest',bootstrap_servers=KAFKA_BROKER)

Unable to connect to remote mqtt broker over ssl web-socket using Paho Javascript library

I am getting the error:
WebSocket connection to 'wss://iot.XXXX.GG:8883/mqtt' failed: Connection closed before receiving a handshake response
When trying to connect to a remote Mosquitto broker over SSL using Javascript Paho library on Windows 10.
What I have already tried is shown in the following listing:
<script type = "text/javascript" language = "javascript">
var mqtt;
var reconnectTimeout = 2000;
var host="iot.XXXX.GG" ;
var port=8883;
function onConnect() {
// Once a connection has been made, make a subscription and send a message.
console.log("Connected ");
message = new Paho.MQTT.Message("Hello World");
message.destinationName = "sensor1";
mqtt.send(message);
}
function MQTTconnect() {
console.log("connecting to "+ host +" "+ port);
mqtt = new Paho.MQTT.Client(host,port,"clientjs");
var options = {
useSSL:true,
timeout: 3,
userName:"abc",
password:"qweqwe",
onSuccess: onConnect
};
mqtt.connect(options);
};
</script>
Expected results should be a message saying 'Connected. Actual results are shown at the beginning of this post as the error I get.
By the way, my Mosquitto.conf file is:
allow_anonymous false
password_file /etc/mosquitto/passwd
listener 1883 localhost
protocol mqtt
listener 8883
certfile /etc/letsencrypt/live/iot.XXXX.GG/cert.pem
cafile /etc/letsencrypt/live/iot.XXXX.GG/chain.pem
keyfile /etc/letsencrypt/live/iot.XXXX.GG/privkey.pem
# WebSockets - insecure
listener 8083
protocol websockets
#http_dir /home/ΧΧΧΧ/domains/iot.XXXX.GG/public_html
#certfile /etc/letsencrypt/live/iot.XXXX.GG/cert.pem
#cafile /etc/letsencrypt/live/iot.XXXX.GG/chain.pem
#keyfile /etc/letsencrypt/live/iot.XXXX.GG/privkey.pem
The Paho MQTT client can only connect to a broker configured to run MQTT over WebSockets.
The mosquitto.conf file you have provided has 3 listeners defined.
The default native MQTT listener on port 1883 bound only to localhost
A native MQTT over SSL listener on port 8883 using the letsencrypt certificate
A MQTT over WebSockets listener on port 8083 with the certificates commented out.
If you want to connect from the web page using MQTT over WebSockets and SSL you need to uncomment the certificates from the 3rd listener and change the port you are connecting to in the page to 8083 (not 8883)

Error while connecting to port 1883

I have a Rpi-A connected to internet via 3G surf-stick and Rpi-B connected to internet via a WiFi hotspot. Rpi-A has a public ip address and also ports 1883 and 8883 are open. Both raspberry Pi's are on different networks. I am attempting to send binary data using MQTT from Rpi-B to Rpi-A.
update: I used the below code to test the MQTT connection. replacing XX.XX.XX.XX with public IP of raspberry Pi. Still I end up getting this error--->
error: [Errno 10060] A connection attempt failed because the connected
party did not properly respond after a period of time, or established
connection failed because connected host has failed to respond.
what might be the possible reason for this error ? Is there anything missing in my code
import paho.mqtt.client as mqtt
import time
def on_connect(client, userdata, flags, rc):
print("Connected with result code " + str(rc))
client = mqtt.Client()
client.on_connect = on_connect
client.connect("xx.xx.xxx.x", 1883, 60)
client.loop_start()
while True:
time.sleep(2)
client.publish('Due_0.72/cmd/in','hello')
print "publish.."

source data from syslog into flume

I tried to setup a flume agent to source data from syslog server.
basically, I have setup a syslog server on an server so-called (server1) to receive syslog events, then forward all messages to different server (server2) where the flume agent installed, then finally all data will be sink to kafka cluster.
Flume configuration as below.
# For each one of the sources, the type is defined
agent.sources.syslogSrc.type = syslogudp
agent.sources.syslogSrc.port = 9090
agent.sources.syslogSrc.host = server2
# The channel can be defined as follows.
agent.sources.syslogSrc.channels = memoryChannel
# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 100
# config for kafka sink
agent.sinks.kafkaSink.channel = memoryChannel
agent.sinks.kafkaSink.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.kafkaSink.kafka.topic = flume
agent.sinks.kafkaSink.kafka.bootstrap.servers = <kafka.broker.list>:9092
agent.sinks.kafkaSink.kafka.flumeBatchSize = 20
agent.sinks.kafkaSink.kafka.producer.acks = 1
agent.sinks.kafkaSink.kafka.producer.linger.ms = 1
agent.sinks.kafkaSink.kafka.producer.compression.type = snappy
But, somehow logsys is not getting injected into flume agent.
appricate for your advice.
I have setup a syslog server on an server so-called (server1)
The syslogudp Source must bind to server1 host
agent.sources.syslogSrc.host = server1
then forward all messages to different server (server2)
the different server refers to the Sink
agent.sinks.kafkaSink.kafka.bootstrap.servers = server2:9092
Flume agent is only a process that hosts these components (Source, Sink, Channel) to facilitate the flow of events.

Ganglia:No nodes were viewed in ganglia web (centOS7)

I installed ganglia server and client at the same machine. But no nodes can view in the web when it finished. No matter google or baidu,no resolution about this problem appeared.I need help.
So this is my gmetad.conf:
[root#tools etc]# egrep -v "^#|^$" gmetad.conf
data_source "trainor" localhost 127.0.0.1
setuid_username "apache"
rrd_rootdir "/var/lib/ganglia/rrds"
case_sensitive_hostnames 0
here is my gmond.conf:
[root#tools etc]# egrep -v "^#|^$" gmond.conf
globals {
user = apache
}
cluster{
name = "trainor"
owner = "apache"
latlong = "unspecified"
url = "unspecified"
}
udp_recv_channel {
port = 8649
}
tcp_accept_channel {
port = 8649
}
Do you have a udp_send_channel set? In my experience (3.1.7), gmond doesn't report a node's own stats over the TCP channel (xml reporting) unless it receives them over UDP (raw stats collection).
You can use "gstat" to connect to gmond to see what it's outputting, or netcat to the TCP port:
nc node1.domain.com 8649
I found these pages the most useful:
https://github.com/ganglia/monitor-core/wiki/Ganglia-Quick-Start
http://timstaley.co.uk/posts/ganglia-setup-explained/

Resources