MQTT publish using URL HIVEMQ - mqtt

How can I publish to an MQTT topic using a URL.
i.e. "http://127.0.0.1/cmnd/power/on" will send "on" to "power" topic.
Ps: I am using HiveMQ

MQTT normally uses TCP as the underlaying protocol, (HTTP only in websocket context).
An Java-Example for connecting an mqtt client with the usage of the paho mqtt client lib:
import org.eclipse.paho.client.mqttv3.*;
import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence;
...
final MqttClient mqttClient = new MqttClient("tcp://localhost:1883",
MqttClient.generateClientId(),
new MemoryPersistence());
opt.setUserName("User");
...
mqttClient.connect(opt);
...
//subscribe to all topics
mqttClient.subscribe("#");
//publish your status ON with a QoS 1 message that is retained
mqttClient.publish("cmnd/power, ("on").getBytes(), 1, true);

First you need make mqtt Connection and once the connection is successful you could send any payload to desired topic.
This is how you need to initiate connection.
String clientId = MqttClient.generateClientId();
MqttConnectOptions options = new MqttConnectOptions();
options.setUserName("USERNAME");
options.setPassword("PASSWORD".toCharArray());
MqttAndroidClient client =
new MqttAndroidClient(this.getApplicationContext(), "tcp://broker.hivemq.com:1883",
clientId);
try {
IMqttToken token = client.connect(options);
token.setActionCallback(new IMqttActionListener() {
#Override
public void onSuccess(IMqttToken asyncActionToken) {
// We are connected
Log.d(TAG, "onSuccess");
}
#Override
public void onFailure(IMqttToken asyncActionToken, Throwable exception) {
// Something went wrong e.g. connection timeout or firewall problems
Log.d(TAG, "onFailure");
}
});
} catch (MqttException e) {
e.printStackTrace();
}
You can publish message to topic power
String topic = "power";
String payload = "ON";
byte[] encodedPayload = new byte[0];
try {
encodedPayload = payload.getBytes("UTF-8");
MqttMessage message = new MqttMessage(encodedPayload);
client.publish(topic, message);
} catch (UnsupportedEncodingException | MqttException e) {
e.printStackTrace();
}

Related

Cloud Dataflow - how does Dataflow do parallelism?

My question is, behind the scene, for element-wise Beam DoFn (ParDo), how does the Cloud Dataflow parallel workload? For example, in my ParDO, I send out one http request to an external server for one element. And I use 30 workers, each has 4vCPU.
Does that mean on each worker, there will be 4 threads at maximum?
Does that mean from each worker, only 4 http connections are necessary or can be established if I keep them alive to get the best performance?
How can I adjust the level of parallelism other than using more cores or more workers?
with my current setting (30*4vCPU worker), I can establish around 120 http connections on the http server. But both server and worker has very low resource usage. basically I want to make them work much harder by sending out more requests out per second. What should I do...
Code Snippet to illustrate my work:
public class NewCallServerDoFn extends DoFn<PreparedRequest,KV<PreparedRequest,String>> {
private static final Logger Logger = LoggerFactory.getLogger(ProcessReponseDoFn.class);
private static PoolingHttpClientConnectionManager _ConnManager = null;
private static CloseableHttpClient _HttpClient = null;
private static HttpRequestRetryHandler _RetryHandler = null;
private static String[] _MapServers = MapServerBatchBeamApplication.CONFIG.getString("mapserver.client.config.server_host").split(",");
#Setup
public void setupHttpClient(){
Logger.info("Setting up HttpClient");
//Question: the value of maxConnection below is actually 10, but with 30 worker machines, I can only see 115 TCP connections established on the server side. So this setting doesn't really take effect as I expected.....
int maxConnection = MapServerBatchBeamApplication.CONFIG.getInt("mapserver.client.config.max_connection");
int timeout = MapServerBatchBeamApplication.CONFIG.getInt("mapserver.client.config.timeout");
_ConnManager = new PoolingHttpClientConnectionManager();
for (String mapServer : _MapServers) {
HttpHost serverHost = new HttpHost(mapServer,80);
_ConnManager.setMaxPerRoute(new HttpRoute(serverHost),maxConnection);
}
// config timeout
RequestConfig requestConfig = RequestConfig.custom()
.setConnectTimeout(timeout)
.setConnectionRequestTimeout(timeout)
.setSocketTimeout(timeout).build();
// config retry
_RetryHandler = new HttpRequestRetryHandler() {
public boolean retryRequest(
IOException exception,
int executionCount,
HttpContext context) {
Logger.info(exception.toString());
Logger.info("try request: " + executionCount);
if (executionCount >= 5) {
// Do not retry if over max retry count
return false;
}
if (exception instanceof InterruptedIOException) {
// Timeout
return false;
}
if (exception instanceof UnknownHostException) {
// Unknown host
return false;
}
if (exception instanceof ConnectTimeoutException) {
// Connection refused
return false;
}
if (exception instanceof SSLException) {
// SSL handshake exception
return false;
}
return true;
}
};
_HttpClient = HttpClients.custom()
.setConnectionManager(_ConnManager)
.setDefaultRequestConfig(requestConfig)
.setRetryHandler(_RetryHandler)
.build();
Logger.info("Setting up HttpClient is done.");
}
#Teardown
public void tearDown(){
Logger.info("Tearing down HttpClient and Connection Manager.");
try {
_HttpClient.close();
_ConnManager.close();
}catch (Exception e){
Logger.warn(e.toString());
}
Logger.info("HttpClient and Connection Manager have been teared down.");
}
#ProcessElement
public void processElement(ProcessContext c) {
PreparedRequest request = c.element();
if(request == null)
return;
String response="{\"my_error\":\"failed to get response from map server with retries\"}";
String chosenServer = _MapServers[request.getHardwareId() % _MapServers.length];
String parameter;
try {
parameter = URLEncoder.encode(request.getRequest(),"UTF-8");
} catch (UnsupportedEncodingException e) {
Logger.error(e.toString());
return;
}
StringBuilder sb = new StringBuilder().append(MapServerBatchBeamApplication.CONFIG.getString("mapserver.client.config.api_path"))
.append("?coordinates=")
.append(parameter);
HttpGet getRequest = new HttpGet(sb.toString());
HttpHost host = new HttpHost(chosenServer,80,"http");
CloseableHttpResponse httpRes;
try {
httpRes = _HttpClient.execute(host,getRequest);
HttpEntity entity = httpRes.getEntity();
if(entity != null){
try
{
response = EntityUtils.toString(entity);
}finally{
EntityUtils.consume(entity);
httpRes.close();
}
}
}catch(Exception e){
Logger.warn("failed by get response from map server with retries for " + request.getRequest());
}
c.output(KV.of(request, response));
}
}
Yes, based on this answer.
No, you can establish more connections. Based on my answer, you can use a async http client to have more concurrent requests. As this answer also describes, you need to collect the results from these asynchronous calls and output it synchronously in any #ProcessElement or #FinishBundle.
See 2.
Since your resource usage is low, it indicates that the worker spends most of its time waiting for a response. I think with the described approach above, you can utilize your resources far better and you can achieve the same performance with far less workers.

401:Authentication credentials were invalid - Invalid or expired token. code - 89

This is the code and I am recieving the error 401: Authentication Error
public class Server {
// initialize socket and input stream
private Socket socket = null;
private ServerSocket server = null;
private DataInputStream in = null;
public void tweet() throws TwitterException {
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setDaemonEnabled(true).setOAuthConsumerKey("......")
.setOAuthConsumerSecret("......")
.setOAuthAccessToken("......")
.setOAuthAccessTokenSecret(".....");
TwitterFactory tf = new TwitterFactory();
twitter4j.Twitter twitter = tf.getInstance();
List status = twitter.getHomeTimeline();
for (Status st : status) {
System.out.println(st.getUser().getName() + "---- Tweets----" + st.getText());
}
}
// constructor with port
public Server(int port) throws TwitterException {
// starts server and waits for a connection
try {
server = new ServerSocket(port);
System.out.println("Server started");
System.out.println("Waiting for a client ...");
socket = server.accept();
System.out.println("Client accepted");
// takes input from the client socket
in = new DataInputStream(new BufferedInputStream(socket.getInputStream()));
String line = "";
// reads message from client until "Over" is sent
while (!line.equals("Over")) {
try {
line = in.readUTF();
System.out.println(line);
if (line.equalsIgnoreCase("Data")) {
tweet();
}
} catch (IOException i) {
System.out.println(i);
}
}
System.out.println("Closing connection");
// close connection
socket.close();
in.close();
} catch (IOException i) {
System.out.println(i);
}
}
public static void main(String args[]) throws TwitterException {
Server server = new Server(5000);
}
}
Please make sure that the tokens are valid.
Then, you could try enabling system proxies like so:
System.setProperty("java.net.useSystemProxies", "true");

Mqtt broker, closing old connection

mqtt client has same client id。
how to closing old connection?
just like this.
New connection from 192.168.3.57 on port 1883.
1479826181: Client paho166768969170988 already connected, closing old connection.
1479826181: Client paho166768969170988 disconnected.
1479826181: New client connected from 192.168.3.57 as paho166768969170988 (c1, k60, u'admin').
EDIT:
String clientId ="**public_cloud**";
try {
MqttClient sampleClient = new MqttClient(config.getBroker(), clientId);
sampleClient.setCallback(new PushCallback());
MqttTopic mtopic = sampleClient.getTopic(config.getTopic());
MqttConnectOptions connOpts = new MqttConnectOptions();
connOpts.setCleanSession(true);
connOpts.setConnectionTimeout(100);
connOpts.setKeepAliveInterval(200);
connOpts.setUserName(config.getUsername());
connOpts.setPassword(config.getPassword().toCharArray());
connOpts.setWill(mtopic, "close".getBytes(), 0, true);
sampleClient.connect(connOpts);
} catch (MqttException e) {
e.printStackTrace();
}
}
if the clientid is same,has error.
ConnectionLost,cause:(32109) - java.io.EOFException
Every connection to the broker MUST have a unique client id. This is part of the MQTT specification. How the broker handles a new connection with an existing client id, but most will disconnect the oldest connection.
The usual fix in the situation you are seeing is to use a randomly generated client id or the current timestamp
long time = new Date().getTime();
String clientId ="public_cloud" + time;
try {
MqttClient sampleClient = new MqttClient(config.getBroker(), clientId);
sampleClient.setCallback(new PushCallback());
MqttTopic mtopic = sampleClient.getTopic(config.getTopic());
MqttConnectOptions connOpts = new MqttConnectOptions();
connOpts.setCleanSession(true);
connOpts.setConnectionTimeout(100);
connOpts.setKeepAliveInterval(200);
connOpts.setUserName(config.getUsername());
connOpts.setPassword(config.getPassword().toCharArray());
connOpts.setWill(mtopic, "close".getBytes(), 0, true);
sampleClient.connect(connOpts);
} catch (MqttException e) {
e.printStackTrace();
}
}
String clientId ="**public_cloud**";
try {
MqttClient sampleClient = new MqttClient(config.getBroker(), clientId);
sampleClient.setCallback(new PushCallback());
MqttTopic mtopic = sampleClient.getTopic(config.getTopic());
MqttConnectOptions connOpts = new MqttConnectOptions();
connOpts.setCleanSession(true);
connOpts.setConnectionTimeout(100);
connOpts.setKeepAliveInterval(200);
connOpts.setUserName(config.getUsername());
connOpts.setPassword(config.getPassword().toCharArray());
connOpts.setWill(mtopic, "close".getBytes(), 0, true);
sampleClient.connect(connOpts);
} catch (MqttException e) {
e.printStackTrace();
}
}
If the clientId is the same, it has the error:
ConnectionLost,cause:(32109) - java.io.EOFException

Mqtt: Persist message on server side

We decided to use mqtt protocol for chat module in our mobile application. I want to save messages of topic in server side also. But i saw,mqtt client is global here,so one way is i have to subscribe single instance of mqtt client to all topics and save messages in database. but is it right approach to do it. i am just worring about it.
private void buildClient(){
log.debug("Connecting... "+CLIENT_ID);
try {
mqttClient = new MqttClient(envConfiguration.getBrokerUrl(), CLIENT_ID);
} catch (MqttException e) {
log.debug("build client stopped due to "+e.getCause());
}
chatCallback = new ChatCallback();
mqttClient.setCallback(chatCallback);
mqttConnectOptions = new MqttConnectOptions();
mqttConnectOptions.setCleanSession(false);
}
#Override
public void connect() {
if(mqttClient == null || !mqttClient.getClientId().equals(CLIENT_ID)){
buildClient();
}
boolean tryConnecting = true;
while(tryConnecting){
try {
mqttClient.connect(mqttConnectOptions);
} catch (Exception e) {
log.debug("connection attempt failed "+ e.getCause() + " trying...");
}
if(mqttClient.isConnected()){
tryConnecting = false;
}else{
pause();
}
}
}
#Override
public void publish() {
boolean publishCallCompletedErrorFree = false;
while (!publishCallCompletedErrorFree) {
try {
mqttClient.publish(TOPIC, "hello".getBytes(), 1, true);
publishCallCompletedErrorFree = true;
} catch (Exception e) {
log.debug("error occured while publishing "+e.getCause());
}finally{
pause();
}
}
}
#Override
public void subscribe() {
if(mqttClient != null && mqttClient.isConnected()){
try {
mqttClient.subscribe(TOPIC, 2);
} catch (MqttException e) {
log.debug("subscribing error.."+e.getCause());
}
}
}
#Override
public void disconnect() {
System.out.println(this.mqttClient.isConnected());
try {
mqttClient.disconnect();
log.debug("disconnected..");
} catch (MqttException e) {
log.debug("erro occured while disconneting.."+e.getCause());
}
}
There are two possibilities how to solve this issue:
Write a MQTT client that subscribes to all topics using a wildcard (# in MQTT)
Write a broker plugin that does the job for you, depending on the broker implementation you're using
There is a good description of how to implement both options at the HiveMQ website, also describing limitations of the first option.

SNMP4j - Cannot send RESPONSE PDU on some OID

I'm trying to respond to SNMP GET requests from SnmpB with SNMP4j 2.3.1 (running on Windows).
In "Discover" mode, SnmpB queries by broadcasting 255.255.255.255 (checked with Wireshark) and I receive a GET request with standard OID (sysDescr, sysUpTime, sysContact, sysName and sysLocation). It finds my instance with the information I coded ("My System", "Myself", ...) (note that it also works when I enter the IP address under the "IP networks" textboxes, though I don't see any traffic on Wireshark but I receive the GET request):
I did write a very simple MIB file that I imported into SnmpB. It defines a single Integer32 data that I want to retrieve using an SNMP GET request from SnmpB.
However, using the same code than for the standard sys* OID, SnmpB doesn't seem to receive that data ("Timeout" in red on the top-right):
I did try Wireshark to check network activity and I don't see anything, so I guess it takes place on localhost (which is not accessible with Wireshark on Windows)? But the traces below show it does not (peerAddress=192.168.56.1)...
Here is the MIB file (code follows):
MY-TEST-MIB DEFINITIONS ::= BEGIN
IMPORTS
enterprises, MODULE-IDENTITY, OBJECT-TYPE, Integer32
FROM SNMPv2-SMI;
myTest MODULE-IDENTITY
LAST-UPDATED "201412301216Z"
ORGANIZATION "My org"
CONTACT-INFO "Matthieu Labas"
DESCRIPTION "MIB Test"
REVISION "201412301216Z"
DESCRIPTION "Generated"
::= { enterprises 12121 }
myData OBJECT-TYPE
SYNTAX Integer32
MAX-ACCESS read-only
STATUS current
DESCRIPTION "My data for test"
::= { myTest 1 }
END
... and the code:
public class RespondGET implements CommandResponder {
public static final OID sysDescr = new OID("1.3.6.1.2.1.1.1.0");
public static final OID sysUpTime = new OID("1.3.6.1.2.1.1.3.0");
public static final OID sysContact = new OID("1.3.6.1.2.1.1.4.0");
public static final OID sysName = new OID("1.3.6.1.2.1.1.5.0");
public static final OID sysLocation = new OID("1.3.6.1.2.1.1.6.0");
public static final OID myData = new OID("1.3.6.1.4.1.12121.1.0");
private Snmp snmp;
public RespondGET() throws IOException {
MessageDispatcher dispatcher = new MessageDispatcherImpl();
dispatcher.addMessageProcessingModel(new MPv2c()); // v2c only
snmp = new Snmp(dispatcher, new DefaultUdpTransportMapping(new UdpAddress("192.168.56.1/161"), true));
snmp.addCommandResponder(this);
snmp.listen();
}
#Override
public void processPdu(CommandResponderEvent event) {
System.out.println("Received PDU "+event);
PDU pdu = event.getPDU();
switch (pdu.getType()) {
case PDU.GET:
List<VariableBinding> responses = new ArrayList<VariableBinding>(pdu.size());
for (VariableBinding v : pdu.getVariableBindings()) {
OID oid = v.getOid();
// Answer the usual SNMP requests
if (sysDescr.equals(oid)) {
responses.add(new VariableBinding(oid, new OctetString("My System description")));
} else if (sysUpTime.equals(oid)) {
responses.add(new VariableBinding(oid, new TimeTicks(ManagementFactory.getRuntimeMXBean().getUptime())));
} else if (sysContact.equals(oid)) {
responses.add(new VariableBinding(oid, new OctetString("Myself")));
} else if (sysName.equals(oid)) {
responses.add(new VariableBinding(oid, new OctetString("My System")));
} else if (sysLocation.equals(oid)) {
responses.add(new VariableBinding(oid, new OctetString("In here")));
} else if (myData.equals(oid)) { // MyData handled here
responses.add(new VariableBinding(oid, new Integer32(18)));
}
}
try {
CommunityTarget comm = new CommunityTarget(event.getPeerAddress(), new OctetString(event.getSecurityName()));
comm.setSecurityLevel(event.getSecurityLevel());
comm.setSecurityModel(event.getSecurityModel());
PDU resp = new PDU(PDU.RESPONSE, responses);
System.out.println(String.format("Sending response PDU to %s/%s: %s", event.getPeerAddress(), new String(event.getSecurityName()), resp));
snmp.send(resp, comm);
} catch (IOException e) {
System.err.println(String.format("Unable to send response PDU! (%s)", e.getMessage()));
}
event.setProcessed(true);
break;
default:
System.err.println(String.format("Unhandled PDU type %s.", PDU.getTypeString(pdu.getType())));
break;
}
}
public static void main(String[] args) throws IOException {
RespondGET rg = new RespondGET();
System.out.println("Listening...");
int n = 300; // 5 min
while (true) {
try { Thread.sleep(1000); } catch (InterruptedException e) { }
if (--n <= 0) break;
}
System.out.println("Stopping...");
rg.snmp.close();
}
}
It produces the following output when I click "discover" under SnmpB and right-click on myData in the MIB Tree and "Get" (slightly reformatted for readability):
Listening...
Received PDU CommandResponderEvent[securityModel=2, securityLevel=1, maxSizeResponsePDU=65535,
pduHandle=PduHandle[16736], stateReference=StateReference[msgID=0,pduHandle=PduHandle[16736],
securityEngineID=null,securityModel=null,securityName=public,securityLevel=1,
contextEngineID=null,contextName=null,retryMsgIDs=null], pdu=GET[requestID=16736, errorStatus=Success(0), errorIndex=0,
VBS[1.3.6.1.2.1.1.1.0 = Null; 1.3.6.1.2.1.1.3.0 = Null; 1.3.6.1.2.1.1.4.0 = Null; 1.3.6.1.2.1.1.5.0 = Null; 1.3.6.1.2.1.1.6.0 = Null]],
messageProcessingModel=1, securityName=public, processed=false, peerAddress=192.168.56.1/49561, transportMapping=org.snmp4j.transport.DefaultUdpTransportMapping#120d62b, tmStateReference=null]
Sending response PDU to 192.168.56.1/49561/public: RESPONSE[requestID=0, errorStatus=Success(0), errorIndex=0,
VBS[1.3.6.1.2.1.1.1.0 = My System description; 1.3.6.1.2.1.1.3.0 = 0:01:03.18; 1.3.6.1.2.1.1.4.0 = Myself; 1.3.6.1.2.1.1.5.0 = My System; 1.3.6.1.2.1.1.6.0 = In here]]
Received PDU CommandResponderEvent[securityModel=2, securityLevel=1, maxSizeResponsePDU=65535,
pduHandle=PduHandle[1047], stateReference=StateReference[msgID=0,pduHandle=PduHandle[1047],
securityEngineID=null,securityModel=null,securityName=public,securityLevel=1,
contextEngineID=null,contextName=null,retryMsgIDs=null], pdu=GET[requestID=1047, errorStatus=Success(0), errorIndex=0,
VBS[1.3.6.1.4.1.12121.1.0 = Null]], messageProcessingModel=1, securityName=public, processed=false, peerAddress=192.168.56.1/49560, transportMapping=org.snmp4j.transport.DefaultUdpTransportMapping#120d62b, tmStateReference=null]
Sending response PDU to 192.168.56.1/49560/public: RESPONSE[requestID=0, errorStatus=Success(0), errorIndex=0, VBS[1.3.6.1.4.1.12121.1.0 = 18]]
Stopping...
What am I missing here? Could that "just" be a network routing issue?
After setting up a VM and checking with Wireshark, it turned out I forgot to set, on the response PDU, the same request ID than the GET PDU.
It was solved by adding resp.setRequestID(pdu.getRequestID()); when building the response PDU
CommunityTarget comm = new CommunityTarget(event.getPeerAddress(), new OctetString(event.getSecurityName()));
comm.setSecurityLevel(event.getSecurityLevel());
comm.setSecurityModel(event.getSecurityModel());
PDU resp = new PDU(PDU.RESPONSE, responses);
resp.setRequestID(pdu.getRequestID()); // Forgot that!
snmp.send(resp, comm);
Thanks to #Jolta for his patience during New Year holiday and his insisting on using Wireshark for further checking. :)

Resources