My requirement is as follows:
There are multiple clients IoT devices. They send data to a server, they receive messages from server and change their behaviour. There are various front ends who want to monitor data from devices and send commands to devices.
I was reading about MQTT and understand it to have subscribers, publishers and a broker in between.
My question is, can I register my devices as publishers and subscribers to the same broker? Is this advisable? Thanks.
I do not see a problem with that.
To keep things separate, you may want to use different channels for transmitting data and control messages.
Yes, there should be no issue with MQTT publish and subscribe in the same client.
Here is a sample Java code for a MQTT Client with both Publish and Subscribe:
import org.eclipse.paho.client.mqttv3.*;
import org.eclipse.paho.client.mqttv3.persist.MqttDefaultFilePersistence;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class MQTTClient {
private static final Logger logger = LoggerFactory.getLogger(MQTTClient.class);
public static void main(String[] args) {
MqttClient mqttClient;
String tmpDir = System.getProperty("java.io.tmpdir");
String subscribeTopicName = "echo";
String publishTopicName = "thing";
String payload;
MqttDefaultFilePersistence dataStore = new MqttDefaultFilePersistence(tmpDir);
try {
mqttClient = new MqttClient("tcp://localhost:1883", "thing1", dataStore);
MqttConnectOptions mqttConnectOptions = new MqttConnectOptions();
mqttConnectOptions.setUserName("/:guest");
mqttConnectOptions.setPassword("guest".toCharArray());
mqttConnectOptions.setCleanSession(false);
mqttClient.connect(mqttConnectOptions);
logger.info("Connected to Broker");
mqttClient.subscribe(subscribeTopicName);
logger.info(mqttClient.getClientId() + " subscribed to topic: {}", subscribeTopicName);
mqttClient.setCallback(new MqttCallback() {
#Override
public void connectionLost(Throwable throwable) {
logger.info("Connection lost to MQTT Broker");
}
#Override
public void messageArrived(String topic, MqttMessage message) throws Exception {
logger.info("-------------------------------------------------");
logger.info("| Received ");
logger.info("| Topic: {}", topic);
logger.info("| Message: {}", new String(message.getPayload()));
logger.info("| QoS: {}", message.getQos());
logger.info("-------------------------------------------------");
}
#Override
public void deliveryComplete(IMqttDeliveryToken iMqttDeliveryToken) {
logger.info("Delivery Complete");
}
});
MqttMessage message = new MqttMessage();
for (int i = 1; i < 6; i++) {
payload = "Message " + i + " from Thing";
message.setPayload(payload
.getBytes());
logger.info("Set Payload: {}", payload);
logger.info(mqttClient.getClientId() + " published to topic: {}", publishTopicName);
//Qos 1
mqttClient.publish(publishTopicName, message);
}
} catch (MqttException me) {
logger.error("reason: {}", me.getReasonCode());
logger.error("msg: {}", me.getMessage());
logger.error("loc: {} ", me.getLocalizedMessage());
logger.error("cause: {}", me.getCause());
logger.error("excep: {}", me);
me.printStackTrace();
}
}
}
In the code above, please check mqttClient.subscribe for subscribe and mqttClient.publish for publish.
I have explained how this works end to end with RabbitMQ as MQTT broker in a blog and the sample working code that I have used is available on GitHub. Please check: http://softwaredevelopercentral.blogspot.com/2017/12/iot-internet-of-things-tutorial.html
There should be only one broker in the overall setup. For scalability reasons we may put multiple brokers which can work in unison and as a whole is always representation of a single broker. In the multiple broker set up too, the edge client will connect to a single broker only.
Make sure you keep one unique publishing topic and one unique subscribing topic per device for the scalability of the device, lower edge processing and easy understanding of humans.
Again there are always trade offs based on the usecase.
Cheers,
Ranjith
Related
I am using reactor in a project, and one of the features calls a blocking service, which connects to a device and gets an infinite stream of events.
I am trying to do a load test to see how many calls can I make to the blocking service.
I am generating around 1000 requests to the blocking service
Flux.just("ip1", "ip2", "ip3", "ip4")
.repeat(250)
The problem is that reactor is only processing the first 256 requests, after that it isn't making any more requests.
When I added the .log("preConnect") I can see that it is logging only one request(256) from the downstream subscriber.
I don't understand what I am doing wrong.
I am attaching simplified example which can reproduce the issue.
package test.reactor;
import org.junit.jupiter.api.Test;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Supplier;
import java.util.stream.Stream;
public class ReactorTest {
#Test
void testLoad() throws InterruptedException {
AtomicInteger id = new AtomicInteger(0);
Flux.just("ip1", "ip2", "ip3", "ip4")
.repeat(250) // will create a total of 1004 messages
.map(str -> str + " id: " + id.incrementAndGet())
.log("preConnect")
.flatMap(this::blocking)
.log()
.subscribeOn(Schedulers.parallel())
.subscribe();
new CountDownLatch(1).await();
}
private Flux<String> blocking(String ip) {
Mono<String> connectMono = Mono.fromCallable(this::connect)
.subscribeOn(Schedulers.boundedElastic())
.map(msg -> "Connected: "+ip + msg);
Flux<String> streamFlux = Mono.fromCallable(this::infiniteNetworkStream)
.subscribeOn(Schedulers.boundedElastic())
.flatMapMany(Flux::fromStream)
.map(msg -> ip + msg);
return connectMono.concatWith(streamFlux);
}
private Stream<String> infiniteNetworkStream() {
return Stream.generate(new Supplier<String>() {
#Override
public String get() {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
return "Hello";
}
});
}
private String connect() throws Exception{
Thread.sleep(100);
return "success";
}
}
Figured out the issue, it has to do with flatmap, the default concurrency for flatmap is 256. It will not request more items from the upstream publisher until the current subscriptions go below 256.
In my case since my flux is infinite, it wasn't processing any after 256.
The solution I found was to increase the concurrency
Flux.just("ip1", "ip2", "ip3", "ip4")
.repeat(250) // will create a total of 1004 messages
.map(str -> str + " id: " + id.incrementAndGet())
.log("preConnect")
.flatMap(this::blocking, 1000) // added 1000 here to increase concurrency
.log()
.subscribeOn(Schedulers.parallel())
.subscribe();
I'm making multi rooms chat with user authorization: users can have access only to some assigned rooms.
For every room I creating a topic with unique room id
How can I check permissions during the opening socket for reading?
On the server-side, for new inbound connection, I want to get room id from topic URL and check user access permissions for the room. But I didn't find how I can do it. I don't see the place, there it's possible.
AbstractSecurityWebSocketMessageBrokerConfigurer -- no way for dynamic check
#Configuration
class WebSocketSecurityConfig extends AbstractSecurityWebSocketMessageBrokerConfigurer {
#Override
protected void configureInbound(MessageSecurityMetadataSourceRegistry message) {
message.nullDestMatcher().permitAll()
.simpDestMatchers("/app/**").authenticated()
.anyMessage().hasRole("USER")
}
}
WebSocketMessageBrokerConfigurer -- can't get current url
#Configuration
class WebSocketSecurityConfig extends WebSocketMessageBrokerConfigurer {
#Override
public void configureClientInboundChannel(ChannelRegistration registration) {
registration.interceptors(new ChannelInterceptor() {
#Override
public Message<?> preSend(Message<?> message, MessageChannel channel) {
StompHeaderAccessor accessor = MessageHeaderAccessor.getAccessor(message, StompHeaderAccessor.class);
if (StompCommand.SUBSCRIBE.equals(accessor.getCommand())) {
...
}
return message
}
});
}
}
I know, how to check access during writing messages, but can't find, how to do it during opening a web socket for reading. What is the standard mechanism for this case?
Dependencies:
compile 'org.grails.plugins:grails-spring-websocket:2.5.0.RC1'
compile "org.springframework.security:spring-security-messaging"
compile "org.springframework.security:spring-security-config"
compile "org.springframework.security:spring-security-core:5.1.8.RELEASE"
compile "org.springframework:spring-messaging:5.1.6.RELEASE"
UPDATE
I can pass room id from the client as a header, but on the server in configureClientInboundChannel I can't be sure, that room id in header same with id in topic URL. I can use some hashes, generated on the server-side, but it looks too complex
var socket = new SockJS("${createLink(uri: '/stomp')}");
var client = webstomp.over(socket);
client.connect({room-id:"0"}, function() {
client.subscribe("/topic/room/1", function(message) {
console.log("/topic/room/1");
}, {roomId:"1"});
client.subscribe("/topic/room/2", function(message) {
console.log("/topic/room/2");
}, {roomId:"2"});
});
During debugging, I have checked headers of command with type StompCommand.CONNECT.
For StompCommand.SUBSCRIBE command current topic URL presented in simpDestination header
Final solution is:
#Configuration
class WebSocketSecurityConfig extends WebSocketMessageBrokerConfigurer {
#Override
public void configureClientInboundChannel(ChannelRegistration registration) {
registration.interceptors(new ChannelInterceptor() {
#Override
public Message<?> preSend(Message<?> message, MessageChannel channel) {
StompHeaderAccessor accessor = MessageHeaderAccessor.getAccessor(message, StompHeaderAccessor.class);
if (StompCommand.SUBSCRIBE.equals(accessor.getCommand())) {
def currentAuthentication = accessor.getHeader('simpUser') // from spring security
String destinationUrl = (String )accessor.getHeader('simpDestination')
// do check, and throw AuthenticationException
}
return message
}
});
}
}
I have a Broadcast receiver that checks WIFI_STATE_CHANGE to see if I have connected to a certain WiFi network. For example if I am coming home, I want a certain MQTT message to be sent. The problem I have is that it connects and sends the MQTT message, only when run the app the first time.
Process:
If I build the application and run it on the device and it recognised my home WiFi it sends the message.
I turn off Wifi from the device, and turn it back on again.
I get "Failure" which is a message when the MQTT connection to the server could not be established.
What I would need is that after I reconnect to the network, instead of "Failure" to get "Connected" but somehow it never happens...what could be wrong?
PS. I think it has to do with the fact that when WiFi is detected, the Broadcast Receiver runs the connection code, although Internet is not available at that point of time (obtaining IP etc.)
Here is the code of the Broadcast receiver:
package me.app.comehomedemo;
import ...
public class SynchronizeBroadcastReceiver extends BroadcastReceiver {
MqttAndroidClient client;
static String MQTTHOST = "myhost";
static String USERNAME = "myusername";
static String PASSWORD = "mypassword";
static String topicStr = "/topic/mac/control";
static String payload = "1";
#Override
public void onReceive(final Context context, Intent intent) {
NetworkInfo info = intent.getParcelableExtra(WifiManager.EXTRA_NETWORK_INFO);
if (info.isConnected()) {
WifiManager wifiManager = ( WifiManager ) context.getSystemService(Context.WIFI_SERVICE);
WifiInfo wifiInfo = wifiManager.getConnectionInfo();
int ip = wifiInfo.getIpAddress();
Toast.makeText(context, String.valueOf(ip), Toast.LENGTH_SHORT).show();
String ssid = wifiInfo.getSSID();
if (ssid.equals("\"mySSID\"")) {
String clientId = MqttClient.generateClientId();
client = new MqttAndroidClient(context.getApplicationContext(), MQTTHOST, clientId);
MqttConnectOptions options = new MqttConnectOptions();
options.setUserName(USERNAME);
options.setPassword(PASSWORD.toCharArray());
// options.setAutomaticReconnect(true);
try {
IMqttToken token = client.connect(options);
token.setActionCallback(new IMqttActionListener() {
#Override
public void onSuccess(IMqttToken asyncActionToken) {
// We are connected
Toast.makeText(context, "Connected", Toast.LENGTH_SHORT).show();
try {
client.publish(topicStr, payload.getBytes(), 0, false);
} catch (MqttException e) {
e.printStackTrace();
}
}
#Override
public void onFailure(IMqttToken asyncActionToken, Throwable exception) {
// Something went wrong e.g. connection timeout or firewall problems
Toast.makeText(context, "Failure", Toast.LENGTH_SHORT).show();
}
});
} catch (MqttException e) {
e.printStackTrace();
}
Uri notification = RingtoneManager.getDefaultUri(RingtoneManager.TYPE_NOTIFICATION);
MediaPlayer mp = MediaPlayer.create(context.getApplicationContext(), notification);
mp.start();
}
}
}
}
I have managed to solve it by waiting 2 seconds and then running the task. Used this solution and it worked. I had to wait for the Internet connection to get ready!
Since waiting 2 seconds has solved your problem, then it might be that the Wifi broadcast comes too early, before there is a connection established (like DHCP gives your phone IP and establishes the routes) for the MQTT connect and publish packets to be properly delivered.
But what happens if some other user needs to wait 10 and not 2 seconds?
My suggestion is to set the automatic reconnect option in MqttConnectOptions and then use the connection callback to publish the needed info to the broker and finally disconnect in publish callback:
private IMqttActionListener mConnectCallback = new IMqttActionListener() {
#Override
public void onSuccess(IMqttToken token) {
try {
client.publish(topicStr, new MqttMessage(payload.getBytes()), null, mPublishCallback);
} catch (Exception ex) {
ex.printStackTrace();
}
}
#Override
public void onFailure(IMqttToken token, Throwable ex) {
}
};
private IMqttActionListener mPublishCallback = new IMqttActionListener() {
#Override
public void onSuccess(IMqttToken token) {
// TODO disconnect
}
#Override
public void onFailure(IMqttToken token, Throwable ex) {
}
};
MqttAndroidClient client = new MqttAndroidClient(context, MQTTHOST, "my_id");
MqttConnectOptions options = new MqttConnectOptions();
options.setUserName(USERNAME);
options.setPassword(PASSWORD.toCharArray());
options.setAutomaticReconnect(true);
client.connect(options, null, mConnectCallback);
In a Spring AMQP project, I would like to get the number of messages in a certain queue (to make decisions based on that number of messages) in RabbitMQ in real time (I can't use the management plugin).
The basic configuration is this:
#Bean(name="managementServerHandler")
public ManagementServerHandler managementServerHandler(){
return new ManagementServerHandler();
}
#Bean
public MessageListenerAdapter broadcastManagementServerHandler() {
return new MessageListenerAdapter(managementServerHandler(), "handleMessage");
}
#Bean(name="broadcastManagementMessageListenerContainer")
public SimpleMessageListenerContainer broadcastManagementMessageListenerContainer()
{
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(_connectionFactory());
container.setQueueNames( REQUEST_MANAGEMENT_QUEUE );
container.setMessageListener(broadcastManagementServerHandler());
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
container.setAutoDeclare(true);
container.setAutoStartup(true);
container.setConcurrentConsumers(1);
container.setRabbitAdmin((RabbitAdmin)_amqpAdmin());
container.setPrefetchCount(50);
container.setDeclarationRetries(3);
container.setMissingQueuesFatal(true);
container.setFailedDeclarationRetryInterval(1000);
container.setRecoveryInterval(400);
return container;
}
Where the "ManagementServerHandler" is just:
public class ManagementServerHandler implements ServletContextAware, MessageListener
{
#Override
public void onMessage(Message msg)
{....}
}
I need the number of queued messages in the onMessage method, but I can't find the way to do it.
I asked this question, but I don't know how to get the AMQP channel:
RabbitMQ and queue data
Thanks!
Use RabbitAdmin.getQueueProperties(queue)
/**
* Returns 3 properties {#link #QUEUE_NAME}, {#link #QUEUE_MESSAGE_COUNT},
* {#link #QUEUE_CONSUMER_COUNT}, or null if the queue doesn't exist.
*/
#Override
public Properties getQueueProperties(final String queueName) {
I've been experiencing troubles with spring-boot consumer. I compared the work of two consumers.
First consumer:
import com.rabbitmq.client.*;
import java.io.IOException;
public class Recv {
private final static String QUEUE_NAME = "hello";
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
Consumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope,
AMQP.BasicProperties properties, byte[] body) throws IOException {
}
};
channel.basicConsume(QUEUE_NAME, true, consumer);
}
}
Second consumer:
#Controller
public class Consumer {
#RabbitListener(queues = "hello")
public void processMessage(Message message) {
}
}
There are no config files for spring-boot consumer installed, everything goes by default.
On my computer first one works 10 times faster. What might be the problem?
The default prefetch (basicQos) for Spring AMQP consumers is 1 which means only 1 message is outstanding at the consumer at any one time; configure the rabbitListenerContainerFactory #Bean to set the prefetchCount to something larger.
You will have to override the default boot-configured #Bean.