I would like to create cloud service like arest.io to access MQTT broker (Mosquitto) via REST API like it is on cloud.arest.io service. The main goal is bridge between mosquitto mqtt broker and apache2 http server to getting access to sepecific mqtt topics like:
cloud.arest.io/47fd9g/digital/5/1
where 47fd9g/digital/5/1 means publish "digital/5/1" in "47fd9g" topic. In the http response I can get JSON data from my IoT device connected to "47fd9g" topic. If you ever used the arest arduino library and arest.io cloud, You know what I mean. I have ubuntu based server with apache, php, mosquitto, php, Mosquitto-PHP php extension. Mosquitto works fine with my IoT device but I don't know how to mosquitto and apache make working toghether like arest.io.
There are a number PHP MQTT clients that you can use to write service as you describe, but you don't directly connect apache to mosquitto. You will have to write the bridge.
But encoding data into the URL as you suggest is not going to work well, it assumes that topics are only ever a single level deep which really doesn't scale well and doesn't allow the use of wildcards.
It would be better to have a HTTP POST to http://example.com/some/multi/level/topic, with the message payload as the body of the post would publish to a the topic some/multi/level/topic makes a lot more sense.
Related
I am developing an app that's going to have two capabilities. First of all, it has to be able to ingest a huge load of events (think millions per minute). It's also going to have a typical REST frontend/backend where I'll have the classic authentication flows, some dashboard based on an analysis of the ingested events, etc. For the REST portion of the app, I'm using Django + Postgres and React on Docker. For the event-driven backend, since Django is too slow to handle clients sending me millions of requests, I was thinking of using Kafka + a streaming database like Materialize.
However, I'm still unclear on how I would route requests to these different endpoints. Do I need to write an endpoint in something fast like Golang/Rust to send client payloads to Kafka? Or do clients communicate with Kafka directly? Is something like Nginx the right way to route these requests? I would still need to have some sort of reference to the the Postgres DB in order to verify the API Key used in the client request is valid.
be able to ingest a huge load of events (think millions per minute).
Through HTTP, or Kafka? If Kafka, then you can use Kafka Connect to directly write into the database without needing any frontend web server.
Django is too slow to handle clients sending me millions of requests
Based on your own benchmarks? Using how many instances?
Do I need to write an endpoint in something fast like Golang/Rust to send client payloads to Kafka?
Python can send producer requests to Kafka. There's nothing preventing you from using Kafka libraries in Django, although none of your question seems very specific to Kafka vs other message queues, especially when you've only referenced needing one or two databases.
If you're going to pick a different "service worker" language, you may as well write your HTTP server using those as well...
Or do clients communicate with Kafka directly?
Web clients? No, at least not without an HTTP proxy.
Is something like Nginx the right way to route these requests?
Unclear what you mean by "requests" here. Kafka requests - no, database requests - probably not, load balancing of HTTP requests - yes.
have some sort of reference to the Postgres DB in order to verify the API Key used in the client request is valid
Okay fine, use a Postgres client in your backend code.
IBM has MQIPT (IBM MQ Internet Pass-Thru) that acts as MQ forwarder/reverse proxy to implement messaging solutions between remote sites across the internet. Is there such an equivalence for Solace?
Solace has all kinds of fancy advanced features for load balancing and hybrid/multi-site deployments like bridges and dynamic message routing, but I don't really know those, and where's the fun in having everything ready-made and pre-solved for you anyway? :-)
So here I am going to assume you want to roll your own solution and use an actual reverse proxy:
You can switch to HTTP-based protocols, and just use any regular HTTP reverse proxy. Solace message brokers have a REST message interface, or if your application already uses the Solace API for messaging (or needs its advanced features), you can switch over to HTTP streaming or WebSockets as a transport by modifying the scheme portion of the broker URL in your application configuration. (http:// or ws:// instead of tcp://) This will only allow you to balance sessions, not individual messages within a single elephant flow.
I just simply cant get it to work.
How can you push an incoming MQTT value to a php page in a Node-Red flow?
"1" is sent to topic "test", I then would like this to be pushed to a blank PHP page, just a simple "1". I do not want the PHP site needed to be refresh.
Appreciate all the inputs I can get! Thanks. :)
If you want truly instant updates then you have 2 real options
Skip Node-RED all together and just subscribe to the same MQTT topic using MQTT over Websockets and the Paho Javascript client. This requires the broker to be running a Websockets listener, but most of the major brokers support this these days.
Use the built in Websockets nodes to provide updates to the web pages. You will need to add a Websocket client to the page and then connect back to a pair of Websocket nodes (input/output) and wire in a MQTT subscriber node. An example Websockets flow can be found here
Is it possible to implement REST client ( not browser but ruby client works on arduino device ) to access resource if resource is available via publish subscribe strategy ? (for example: http server(rails application) informed client that resource is now available you can receive/download)
I want to build application for "communication directly between ruby client in arduino board to http server". I don't want to use middle layer like Rabbitmq, zeromq. If it is possible what can be the possible steps to follow ?
It is not possible to use HTTP for pub/sub.
If you don't want to use message queue(anyMQ), try using plain TCP/UDP sockets.
XMPP is not the best choice, it gives too much overhead.
We are looking at using MQTT as the messaging protocol on a new device we're building. We'd also like a web interface for the device. Does anyone know if you can implement a browser client app (without additional plugins) that talks MQTT?
Yes, as mentioned in Steve-o's comment MQTT via websockets is very possible.
There are 2 options at the moment
IBM's MQ 7.5 comes with websockets support, you can find details here.
The Mosquitto broker has a javascript client with an example running here.
To answer your second question lighttpd has a websockets module that can be used to do forwarding to an existing broker with details here.
I've not been able to find anything for Apache that doesn't need you to write your own library to do the forwarding.
To extend hardillbs answer:
There is a third option now:
HiveMQ supports native websockets without the need for any external web server. It works perfectly with the Javascript client of Eclipse Paho.
If you want to try it out, the public MQTTDashboard (which uses HiveMQ under the hoods) supports websockets on port 8000.
Disclosure: I am one of the developers of HiveMQ
IBM has released a WebSockets based JavaScript client. It is open-source on Eclipse Paho.
Git repository:
http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.javascript.git/
Tutorial is here.
https://www.ibm.com/developerworks/community/blogs/c565c720-fe84-4f63-873f-607d87787327/entry/how_to_prog_javascript?lang=en
We've had good experience using WSS, the WebSocket to TCP Gateway (for MQTT). It means running a separate process, but that does avoid having to change Apache's configuration. WSS also has support for TLS.
Not Apache, but you can follow this guide to install lighttpd with mod_websocket,
http://oriolrius.cat/blog/2013/09/25/server-send-push-notifications-to-client-browser-without-polling/
but for easy of use I highly recommend HiveMQ, as #Dominik recommends.
The blog by jpmens is a good starting point to get mosquitto setup with websocket feature. He's client javascript code based on Paho is on github.
This blog "How to run your web server and MQTT WebSockets broker on the same por" explains two ways for the server side:
mosquitto with http configuration
apache+mod_websocket_mosquitto
Why not nodejs + socket.io? They come with all gadgets included (webscokets, pub/sub, webserver,mqtt broker...).
SocketCluster is another good starting point.
This link (Download codes) is useful. Download it and modify this file mosq-mqttws31.html. Instead of the embedded java script library use this cdn
https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js
and the client instantiation should be client = new Paho.MQTT.Client("test.mosquitto.org", 8080,"myclien256tiddd_" + parseInt(Math.random() * 100, 10));