In my current IOT project. I will be using multiple ESP8266s (3) to send data and receive actions. Each MCU will be in charge of monitoring different aquariums around the house. I have thought of structuring my topics like the following:
"Data" topics will follow the same structure, for example to retrieve temperature data:
esp8266/aquarium/aquarium_id/temperature/dht11
"Action topics", the topics the MCU subscribes to receive commands, for example:
aquariumcontroller/aquarium_id/action/water
The topic which the aquarium subscribes to to updte the MCU's params:
aquariumcontroller/aquarium/aquarium_id/params
The aquariumcontroller is the MQTT client written in Python. This is the entity that will be sending actions and handling messages received. I have two questions, are my topics correctly named and structured to handle multiple aquariums? Also as I will have a Controller, isn't it better to also have a database which will contain the aquariums info like the topics for this specific aquarium and the params or will i run into problems if i persist topics when i change the aquarium ID?
Thank you
What you mentioned will work.
You can also have a common topic that everyone subscribes to and use JSON format to send data. In JSON format you can mention necessary identifiers and actions.
{
"aquarium_id": "xxx",
"operation": "temperature_read",
"value": "24.5"
}
For every aquarium_id you can have an individual topic. The advantage of this scheme is that aquarium controller will only receive message that are associated with it however their will be a added complexity where by you have to pre-populate aquarium_id in your server.
{
"operation": "temperature_read",
"value": "24.5"
}
You should ideally have a database to store aquarium_id and its relevant properties (something like master record). With database you can also store you readings and action to get an historical view of your data. You can use postgres as DBMS.
Related
I am trying to extract a large amount of details out of our Eloqua system using it's API and got this API to work perfectly for single IDs: https://docs.oracle.com/en/cloud/saas/marketing/eloqua-rest-api/op-api-rest-1.0-data-contact-id-get.html
The problem is that I need to run this for a large number of IDs and it will require alot in order to run it for the entire population. Is there any bulk APIs that can extract all of the following details out of Eloqua/Contact for the entire population? I don't see any on that pages documentation that meet this need under the Bulk section.
contactid, company, employees, company_revenue, business_phone, email_address, web_domain, date_created, date_modified, address_1, address_2, city, state_or_province, zip_or_postal_code, mobile_phone, first_name, last_name, title
It's a multi-step process with the Bulk API, typically in the following fashion:
Get a list of the current internal field names - useful for creating your export definition
Create an export definition and post it here. There is a useful example on the page, you do not need a filter criteria. Store the export ID somewhere
Using your export definition id, create a sync. It will gather the data in the background and prepare it for you. Take note of the sync ID provided in the initial response.
Check on the sync status with your sync ID here. It should only take a couple of minutes - and there is a callback url option as well in the previous step, if you don't want to keep polling.
Once your data is ready, use that sync id and request the data. Depending on how many rows were retrieved, you might need to paginate through the results using the offset query param. By default it will give you JSON, but I usually choose CSV (specify in the header).
If you need updated data, feel free to create a new sync using the same export definition id. You do not need to create a new export definition each time.
I am trying to set up the Cumulocity smartREST response templates to supply info from the device object stored by an app. I am thinking it is not possible but I want to confirm because the documentation is brief and not clear.
I have successfully implemented MEASUREMENT POST templates previously but I am trying to do INVENTORY GET now. I have set some values in the device object and I can see them from tenant.cumulocity.com/inventory/managedObjects/id/. This could look like:
"custom_values": {
"val1": "abcde",
"val2": "fghij"
}
Now I go to the smartREST template web interface editor and create a template (eg t1) and set a message (eg m001) and set it to 'inventory' and 'get' and tick 'includes response'. I'm not sure if I need custom fields here. Then I create a response (eg r001) and I now have to fill in base pattern with a condition and some number of other patterns. I assume in one of those fields I have $.custom_values.val1 etc but all permutations I try result in 'no access to object' error from topic 's/e'.
My ideal result (from the above example) would be to publish to '/s/uc/t1' with 'm001' and receive 'r001,abcde' (ie custom_values.val1) and publish 'm002' and receive 'r001,fghij' (ie custom_values.val2).
Thanks for taking the time to read.
Your approach is correct but your MQTT user probably is just missing the rights to the object as it is created by the app (I guess the mqtt client is a device user).
The best approach here is to give this object you are creating from the app the fragment c8y_Global
{
"name": "testObject3",
"c8y_Global": {},
"custom_values": {
"val1": "abcde",
"val2": "fghij"
}
}
This fragment will make this object available to all registered users. This is common practice for app meta data.
I am very new with mqtt design.
As I see from some tutorials in the internet, common mqtt topic has this format: /home/room/device_type/device_id
I could not see the benefit to do that. And have no idea how to use this kind of design.
From my point of view, the device (dev) might subscribe (sub) to control topic and publish (pub) to status topic. Like this:
pub: clients/dev/devid/stat
sub: clients/dev/devid/ctrl
In this way, it seems sub,pub logic is very simple for both clients and devices
Could someone please tell me some good way to design mqtt topic ?
(!) Please do not start topic with '/' (This one has been recommended by HiveMQ Team)
EDIT:
I just figured out that for whatever kind of design, the model must serve-able at least:
Individual control: send control command to a particular device.
Group control: send control command to a group of devices: type, defined group
Able to recieve the status of device.
Thank you very much
I found that the following topic split scheme works very well in multiple applications
protocol_prefix / src_id / dest_id / message_id / extra_properties
protocol_prefix is used to differentiate between different protocols / application that can be used at the same time
src_id is the ID of the mqtt client that publishes the message. It is expected to be the same as "client ID" used to connect to MQTT broker. It allows quick ACL control to check whether the client is allowed to publish specific topic.
dest_id is client ID of the "destination" unit, i.e. to whom the message is intended. Also allows quick ACL control on the broker of whether client is allowed to subscribe to a particular topic. There can be reserved "destination" strings to specify that the message is broadcasted to anyone who is interested. For example all.
message_id is actual ID of the message within used protocol. I usually use numeric value (as string of course), because the IOT or other embedded system that is connected to MQTT broker can have other I/O links and I would like to use the same protocol (but with different transport framing) to control the device using these other I/O links. I usually use numeric message IDs in such communication links.
extra_properties is an optional subtopic which can be used to communicate other MQTT specific extra information (comma separated key=value pairs for example). Good example would be reporting timestamp of the message when it was actually sent by the client. In case of "retained" messages it can help to identify the relevance of the received message. With MQTTv5 protocol that is expected to arrive soon, the need for this subtopic may disappear because there will be other way to communicate extra properties.
Hope it helps.
We have done some work on that for the domain of manufacturing (Industrial-IoT, not IoT!).
In our scenario there are lots of serverside apps of different companies communicating through MQTT. Thus we needed some overall structure. We call this "Manufacturing Message Stack".
The bottom layer is MQTT, then there is the "Messaging Layer". It consists mainly of
basic topic specs
basic payload specs
On top of the messaging layer there are domain message layers covering various domain specific topics as system messages, alerting, physical device / digital twin messages or other manufacturing related messages.
Topics
The topics are roughly defined as <senderapp>/<app-id>/<message-name>/<args> e.g. pacman/pacman-1/gameover (this is a sample for illustration only!)
The developer of an app which publishes MQTT messages defines <message-name> and <args> depending on the semantics of the payload.
The <senderapp> and <app-id> refers to the sending App and allows to quickly select messages from a defines origin (publisher). We deploy Apps in a microservice environment built with Docker, Rancher - and soon - Kubernetes.
Payload
The payload is specified in JSON format. There is a JSON schema reference URL in each
build an URL to an API of the publishing app which holds further information (e.g. JSON schema) of the message sent. Thus a subscriber can get meta data of a MQTT message on demand. Static meta data is not sent with a message to reduce the payload size.
Payload sample:
{
"$schema": "http://app/api/messages/message1.json",
"score": 1234,
"highscore": false
}
Publisher's message meta data
The publishing app holds an index of all messages which can be sent in an API at
http://<app>/api/messages/index.json:
{
"message1": "message1.json",
...
}
Each message is described by its JSON schema message1.json:
{
"$schema": "http://json-schema.org/draft-06/schema#",
"title": "Pacman end of game",
"properties": {
"score": {
"description": "Players score at the end of game",
"type": "integer"
},
...
}
}
Unfortunately we did not publish our manufacturing message stack yet. Publishing is planned in the next months. Feedback is welcome.
I think if topics are to reflect the physical world, we should look at something like Signal K.
EDIT:
That spec is also still maturing, but it includes concepts like "self" for the server/broker, and a tree that can start at the current vessel/home, but easily extends upwards to other vessels/aircraft/things.
My two cents:
All topics are read-only unless they end in "/set"
Ideally, topics are reasonably normalized and granular. I can understand grouping values up into a group topic. IMHO, this kind of decision should be application-specific.
Payloads should be strings, to avoid endian-ness issues
Here's one suggested tree:
broker = information of this specific broker
broker/clients
broker/clients/count
broker/clients/0/name or broker/clients[0]/name
broker/topics
home = this current location (could also be "here" or something)
home/kitchen/temperature "19C"
home/kitchen/temperature/hardware/type "ESP8266"
home/garage/maindoor/set "closed"
locations = list of all known locations
locations/0/uuid
locations/0/name
locations/0/address
I'm using System Center Orchestrator 2012, and I have a generic error handler which is connected to all the activities in my runbook.
I have subscribed to a list of published data as a parameter in this error handler, which looks like this:
Activity Name : {Activity Name from "Monitor File"}{Activity Name from "Move file"}.......
What I need to do is copy this list of subscribed data and do a search and replace to change the Activity Name parameter to something else, say Error Message. Whenever I copy this list of variables, I get gibberish when I paste it
\`d.T.~Ed/{598EBDFA-BF5B-4B77-8156-E6FA6ECD0CE1}.Object.Name\`d.T.~Ed/\`d.T.~Ed/{6838D741-DF8E-4C25-8C28-D06A52F67D36}.Object.Name\`d.T.~Ed/\`d.T.~Ed/{D1D9BBE6-5AAF-4D8F-A98A-1A8BDD977E7E}.Object.Name\`d.T.~Ed/\`d.T.~Ed/{9EC92323-1B9B-4D06-88E9-A97BA525CF5A}.Object.Name\`d.T.~Ed/\`d.T.~Ed/{FAD32B4C-92CC-40BD-837A-4C5F22C2E018}.Object.Name\`d.T.~Ed/\`d.T.~Ed/{DFF7A110-ACFD-4377-AFEC-16B5BEC8BFF4}.Object.Name\`d.T.~Ed/\`d.T.~Ed/{139ACC4E-CF6D-4EEE-BD88-9DC1E0FC2038}.Object.Name\`d.T.~Ed/\`d.T.~Ed/{1484789C-BB37-4507-AD21-E367665E0BE6}.Object.Name\`d.T.~Ed/\`d.T.~Ed/{01EA8BD0-69C5-4959-86DB-29FAD34D144A}.Object.Name\`d.T.~Ed/\`d.T.~Ed/{A60C582B-5DD0-41F7-BB0A-B5D71C3B9ECB}.Object.Name\`d.T.~Ed/\`d.T.~Ed/{1FA1E2D6-813D-4A4A-A5CD-07EB2AD4AC9B}.Object.Name\`d.T.~Ed/
Is there a way to simply copy the text? If not, I will have to click on all the activities over and over again to subscribe to different types of data.
Thanks in advance
The gibberish you're seeing is how Orchestrator translates the published data and variables to the database guids. Copying this out will always translate to the DB guids, so there's no way to do a copy/replace with "activity name" for "Error Summary". You'd need to subscribe to each activity's data or look up all of the guids which probably wouldn't be easier.
Unfortunately, setting up the error handling process in SCOrch becomes tedious to be effective.
I use a separate email linked to each activity to give more information based on the error message. I don't like to use generic error emails with published data from multiple activities since it doesn't necessarily give all the useful information you need (or an end user will be able to understand).
What I'm trying to do:
Be able to have users subscribed to a number of different 'chat rooms' and use reverse AJAX / comet to send messages from a chat room to everyone logged into that room. (a bit more complicated but this is a similar use case).
What I'm doing:
Using Grails with JMS and Atmosphere. When a message is sent, I'm using JMS to send the message object which is received by a Grails service which is then broadcasted to the atmosphere URL (i.e. atmosphere/messages).
Obviously JMS is a bit redundant there but I though I could use it to help me filter who should retrieve the message although that doesn't really look it'll work (given that the subscriber is basically a singleton service...).
Anyway, what I need to be able to do is only send out a message to the correct subset of people listening to atmosphere/messages. A RESTful-type URL will be perfect here (i.e. atmosphere/messages/* where * is the room ID) however I have no idea how to do that with Atmosphere.
Any ideas / suggestions on how I can achieve what I want? Nothing is concrete at all here so feel free to suggest almost anything. I've even been thinking (based on the response to another question), for example, if I could do something like send out messages to a Node.js server and have that handle the reverse AJAX / comet part.
If I understand your requirements correctly the following should work (jax-rs + scala code):
1) Everyone who wants to get messages from a chat room registers for it:
#GET
#Path(choose/a/path)
def register(#QueryParam("chatroomId") chatroomId: Broadcaster) {
// alternatively, the Suspend annotation can be used
new SuspendResponse.SuspendResponseBuilder[String]()
.resumeOnBroadcast(false).broadcaster(chatroomId).scope(SCOPE.REQUEST)
.period(suspendTimeout, TimeUnit.MINUTES)
.addListener(new AtmosphereEventsLogger()).build
}
2) To broadcast a message for all the registered users, call the following method:
#POST
#Broadcast
#Path(choose/a/path/{chatroomId})
def broadcast(#PathParam("chatroomId") id: String) {
// first find your broadcaster with the BroadcasterFactory
BroadcasterFactory.getDefault().lookupAll() // or maybe there is a find by id?
broadcaster = ...
broadcaster.broadcast(<your message>)
}
I also recommend reading the atmosphere whitepaper, have a look at the mailing list and at Jeanfrancois Arcand's blog.
Hope that helps.
There is a misunderstaning of the concept of comet. Its just another publish/subscribe implementation. If you have multiple chat-rooms, then you need to have multiple "topics", i.e. multiple channels the user can register to. E.g.:
broadcaster['/atmosphere/chatRoom1'].broadcast('Hello world!')
broadcaster['/atmosphere/chatRoom2'].broadcast('Hello world!')
So I would advance you to creaet multiple channels and do not filter manually the set of users, which should retrieve messages (which is definitely not the way it should be done). You do not need to create anything on the server side on this, since the user will just register for a specific channel and receive messages, which anyone is putting into it.
I would recommend you create an AtmosphereHandler for one URL like /atmosphere/chat-room and then use the AtmosphereResource and bind an BroadcastFilter with it, lets say name it ChatRoomBroadcastFilter.
Whenever a user subscribes to a new chat room, a message would be sent to the server (from the client) telling the server about the subscription. Once subscribed, maintain the list of users <> chat room bindings somewhere on the server.
Whenever a message is broadcasted, broadcast it with the chat room id with it. The in the ChatRoomBroadcastFilter (You probably need to make this a PerRequestBroacastFilter) propagate the message to the user only if the user subscribed to the chat room. I am not sure if this clears it out. If you need code example please mention in the comments. I'll put that but that needs some time so ain't putting it right now ;).