I'm still learning about SQS, I did some successful tests with SQS and a Ruby application with Shoryuken that picks up the messages. This works fine. We want to get 1000 IoT devices installed in people's homes. These devices will send events to the backend, which will be picked up. So far so good.
Now we also need to send messages back to the devices (less frequent, but still). Will I be able to use SQS for this too? How can I target the device?
Do I need to add a MessageAttribute to identity the destination? Or create a queue per device?
What are the best practises here?
SQS is definitely not the right tool to use for sending messages to individual devices, and while a queue per device may work, I would not recommend it (1000+ devices regularly polling for messages can get expensive).
Take a look at Amazon IoT, in particular the message broker component.
Related
Take an iOS app like Instagram. Instagram is fundementally a real-time application that updates its UI whenever a user interacts with you. For example, if someone likes your post and you are using the app, the UI is updated to trigger dopamine release and inform you that something has hapened to one of your posts. Similarly, when someone sends you a direct message on instagram, while using the app, you see the message spring down from the top, all in real-time.
In terms of implementing such real time features it is obvious that a naive HTTPS polling approach is far too inefficient. Thus this leaves two strategies:
1.) APNS Push Notifications:
When a user likes a post, sends a direct message, comments (etc.), send an HTTP POST to a backend server that will then update the database and send a silent Apple push-notification to the device of the recipient. The recipient, which is using the app, will receive the pushed payload and will send an HTTP GET to the backend server to fetch the needed data (ie. the contents of the direct message sent). The UI is updated in quasi "real-time".
2.) Websockets:
Whenever any user opens the iOS app, connect the user to the server via a websocket. This means, that all users currently using the app are connected to the server via their own websocket. When a user likes a post, sends a direct message, comments (etc.), the app sends a message to the server through the socket indicating the action. The server, before updating the database, finds the socket associated with the recipient and forwards the message through the socket to the recipient. Upon reception of the message, the UI is updated in real-time
Which of these approaches is scalable and better suited for a production environment?
TL;DR You'll love websockets + Combine framework, event driven to update your UI smoothly. APNS can be somewhat unreliable in terms of deliverability and resource/database costs, and to me is more of an information center.
Reasons to like websockets:
Decreases DB costs (e.g. finding device to send to)
You know when the user is not using the app, decreasing outbound data costs
Latency and faster practically speaking (more in depth in the websocket paragraph)
To answer the question: websockets are scalable based on your user count obviously. APNS is not server side testable in the CI, not cross platform, and not exactly feasible in terms of resource consumption.
I have a bias with websockets. But to present why I like it more, think about the efficiency given by your Instagram example.
To me, websockets == Event Driven, APNS is a simple information center.
APNS:
You sign up with APNS and you save the device in your backend. Great, every time an action is performed, like someone liking your post, query your backend, find the device that is linked to the OP, then send it outbound (which costs money if you think cloud computing costs). And you have to do that every single time someone likes your post (obviously you can aggregate, but why bother when you have websockets?). So database costs is one thing to think about. Additionally, a user could be muting their notifications. I don't have Instagram, but something efficiency wise they could have done (if not by sockets) is updating their UI based on incoming notifications for likes/hearts rather than a websocket connection. That's slow in terms of latency, costly, and unreliable if marked as spam. However, APNS has the benefit of not needing authorization unlike websockets, but...
Websockets:
When we approach websockets, you only authorize once (at least for mobile applications). You're removing outbound data costs (in terms of $$ and latency) by removing stuff like headers. You want to send small chunks of data, and sockets are just sending text/binary (I like to create commands out of them using JSON. A notable example is GitHub). When your user is done with the app, you close the connection and you don't need to send anymore data via APNS. Your server itself can be communicating with a single websocket via something like Redis PubSub in order to update your UI when someone uses a POST request to heart a post without needing to send some push notification. That's a win for the liker (who doesn't need to wait for that push notification to be sent) or the other way if you offload it to a background task (the OP doesn't need to wait). Websockets == Event Driven:
Small data chunks that are constantly delivered (as in a lot) is much better than having APNS in which it can be slow to actually deliver, especially if you're flooding Apple/your user with notifications, marking you as spam. Although, disclaimer, that's just my personal belief and speculation depending on your use case.
You're removing those unnecessary database costs.
A downside would be keeping the connection alive.
Personally, I've used websockets with the combine framework recently for a chat application, but it can be used in so many different circumstances. Updating a Facebook feed/comment section, live notifications via the websocket instead of APNS, even posting content.
APNS is not guaranteed to be delivered, especially if you have a lot of notifications - I don't remember where I read that, but after some threshold per min it'll stop working. Also, you'll have to send them to all your users, not just online ones. It's possible to send silent notifications, but it's still not optimal.
That's why websocket is a preferred way. Also you can use something like https://github.com/centrifugal/centrifugo, which is a helper for your server, that'll hold all the connections, and is very stable.
It is not one out of two, even the web-socket isn't failsafe, but all of above should be considered for an effective communication, when the app is in foreground use a web-socket to listen for any updates from the server, expect a confirmation from client when something is delivered by socket, if the web-socket connection is not active or the there is no confirmation response deliver the update through APNS/FCM, as APNS/FCM it is not guaranteed to deliver, deliver the updates when next Socket connection is successful.
I am using Google IOT core with mongoose os. I wanted to update device connection status to firestore. But i am unable to find event which reports mqtt connection status to pub/sub like when device disconnects or reconnect i.e if device is offline or not.
I am stuck on this problem for days.Any help will be appreciated
Update
As #devunwired mentioned in this response it is now possible to monitor Stackdriver logs for disconnect events. You must have at a minimum enabled INFO level logging on your project in IoT Core > Registries > [your registry] > Edit Registry > Select "Info" log level > Click save.
Original Response
There are a few values you can look at that are tracked in device configuration metadata that you could use to know when a device last was online:
Last Configuration Send time - sent anytime your device connects /
configuration is posted
Last Event Time - Last time an event was sent from the device
Last State Time - Last time state was sent from the device
Last Heartbeat time - Last time MQTT heartbeat was sent
To get you started, here is an example using API explorer that you can fill-in with your project ID, region, registry, and device to query for a specific device's metadata.
For 1...3 you have control over these through device manager and by publishing data. MQTT heartbeat is updated if your device sends an MQTT_PINGREQ message during the "ping period" without other messages getting sent.
At any rate, you could use any of these update time values to see the last time a device was online / functioning. You could query the states of your devices after listing the devices in a registry and could update a Firebase RTDB periodically if that's how you want to report (e.g. using AppEngine TaskQueue). Note that you also just can get these "last connected" values from the Google Cloud Console.
It was said before but we don't have an event for disconnect, just configuration ack, which generally is the connection event. If you want to share state between a device and the device manager, use state messages.
Unfortunately, there's no built in way to do this right now as there aren't events on this state.
However, you could implement a hack by sending a message on connect/disconnect from the device that you have a Cloud Function subscribed to the Pub/Sub topic listening for. It's not perfect as it would fail in the case where the device disconnected unexpectedly.
There currently is no way to do this, that i've been able to find (a year later after this original post). I posted a question here on SO regarding this as well, with more details and link to example code I had to use for handling this:
Google Core IoT Device Offline Event or Connection Status
The AWS IoT platform publishes messages on a special MQTT topic (prefixed with $aws) when your device connects/disconnects. You can easily use these to monitor these events - however, you should be aware that the MQTT protocol is designed to be robust to a poor networking conditions and the broker on the AWS side probably doesn't think it's a bit deal to disconnect a client. The broker expects that the client will just reconnect and queue messages for a moment during that process (which can be a big deal on a microcontroller).
All that being said, the AWS topics you would watch are:
$aws/events/presence/connected/{clientId}
and
$aws/events/presence/disconnected/{clientId}
and the documentation for these (and other) lifecycle events are located: https://docs.aws.amazon.com/iot/latest/developerguide/life-cycle-events.html
If I want to make sure that a device receives a message, over the network, for an app used by sellers in a shop for example.
I heard push notifications are not 100% reliable, sometimes some of the notifications don't arrive, or not on time.
The app could be in a shop, where the staff communicate with one another, and there could be 10 devices connected. (ipad, iphones)
edit 1: I heard about sockets, is it the right direction to go?
EDIT 2:
I am not sure why a socket should be used, rather than a webserver for example, I found these 2 sentences (source raywenderlich) :
You can send connected clients data whenever you want, rather than requiring the clients to poll.
You can write socket servers without a dependency of a web server, and can write in the language of your choice : don't understand what the "dependency" is?
Does it also mean :
Sockets let 2 (or more) specific devices to connect one another in a private connection, compared to webservers where everybody could connect if there is no login/password?
EDIT 3 : Maybe a bluetooth solution with MultiPeerConnectivity would be better...
Sockets, and push notifications in general, are only as reliable as the network the user is connected to. If your looking to circumvent network reliability where a 100% success rate is guaranteed, in a Shop environment where users are in close proximity, you can look into GKSession as part of the GameKit.framework
Or you could look into Bonjour or client/service discovery protocols that makes the process of "knowing" peers in a network
I see you tagged Parse.com, again, the reliability of Push Notifications is highly dependent on the reachability, however, most issues that arise with Parse is dev-end related, not product related.
EDIT I forgot to mention MultiPeerConnectivity
If you use TCP sockets, you are guaranteed the message will be delivered, and very quickly too.
However, the application would have to be open (and most likely in the foreground) to receive the messages. You can always have the server wait till the client connects to send the message.
I would suggest using a combination of both TCP sockets and push notifications (for when the app is closed).
I have a chat application developed by JS. I want to send PING to server once in a while. Its not a problem if app runs on fore ground. The problem is when user minimizes it or open another app. My app looses its focus and gets into suspended state.
I have following two use-cases.
To keep the chat session open I need to send PING to server (Its an IRC server) every X minutes even the app runs in background.
We also need to check for new messages (by ajax on a local http server) and add a local notification to the notification queue so when user clicks on it app can resume
I have found apple does not allow running apps in the background. if they allow they require special permission. I found some apps does it by requesting finite length execution time.
What is the best way to get highest possible background execution time? As a chat app can I request permission for voip, location or any other way ?
Note: the app will be running in an environment where there is no Internet. Hence push notification will not work here.
Update: After doing a lot searching I found background fetch. It seem background fetch will suite it. But still the problem remains, its not called in a timely manner.
This sounds like an interesting problem. From reading the various comments, it sounds like you want this to work when you're on a local network - so you have wifi, but the wifi router/base station isn't connected to the actual internet?
Because background refresh isn't going to be predictable - you'll never know when it is going to update - you might want to get creative.
You could look into exploiting iOS VOIP support, only without the Voice! Apple has some tips on VOIP here. VOIP basically uses something called SIP (Session Initiation Protocol), which is signalling layer of the call, and a lot like HTTP. It's this SIP layer that you want to take advantage of.
This isn't going to be terribly easy, but it should be achievable. Setup your app to use VOIP, and then look into something like PJSip as your SIP library. Then, on your local network have a SIP Server (I'm sure there are plenty open source implementations) that you can register your iPhone against (so your server knows where your phone is, pretending to be a VOIP phone). This should work, because it doesn't need to go through Apple as far as I am aware... And will run happily on your local network.
Then, the server can send a message via SIP to the handset, as if it were instigating a VOIP session. You app is awoken, gets the messages - ideally from the SIP message if possible - and then just doesn't start the session. SIP was designed just for creating sessions, not just VOIP. When I worked in Telecoms R&D (a long time ago) we were using it to swap between Text/Voice/Video, all using local servers.
You'll have to jump a lot of hoops to make this work, but it would be pretty awesome. I have never tried this actual use case - especially with iOS, but I'm fairly sure it will work. It is a bit of a fudge, but should get you where you need to go.
Good luck!
You can use something like PubNub to build this chat app with iOS using native Objective-C code, or with the Phonegap (Cordova) libs.
The beauty with using a real-time messaging network like PubNub is that when the app goes to the background, you can easily have the chat messages come in on APNS.
When the app is in the foreground, it can just receive them as the native (PubNub) message. And if it needs to "catch-up" with the messages it missed while in the background (but received via APNS), its trivial to implement.
Also, PubNub is platform agnostic -- so you can easily also use it on Web, Android, BB, Windows Phone, etc.
http://www.pubnub.com/blog/build-real-time-chat-10-lines-code/
http://www.pubnub.com/blog/html5-websockets-beautiful-real-time-chat-on-mobile-using-pubnubs-channel-presence/
https://github.com/pubnub/objective-c/tree/master/iOS
https://github.com/pubnub/javascript/tree/master/phonegap
geremy
I have to add e-mail notifications to a client server application.
Notifications happen as the user do some particular action on the client UI.
If I had a middle tier or a service running at server I can imagine how to do it:
1) I simply create a DB tables with "pending notifications"
2) as a user does an action that generates a notification I add a record to the table
3) serverside I would continuously try to send those mails and removing them from the table once sending is succesful
Now I cannot do this now, I have a plan to add a service later on, but for now I must go the quick and dirty way.
So somehow what I was thinking to is to implement something like this:
1) as a notify-worth event occurs at client, the same client (my exe) tries to send the notification, upon failure it will log the notification in the "pending notifications" table (failure can be becuase lack of internet connection or any other problem)
2) I add a Timer that will work from any client machine to check for pending notifications. If there are any the client will try to send the e-mail (using a transaction: I will mark a field as "TryngToSendFromClientX" and in case of failure I will reset that field to NULL)
I think this approach would work, it has obvious limitations (if after failure no one logs into the system, no notification will be sent - same would be if service goes "down"). But can you comment on this approach and suggest a better one?
Additional notes (to better understand the scenario):
a) Note: all notifications are sent from the same e-mail account.
b) I don't need to keep track of who sent the e-mail.
c) the problem of creating the service now is that it will basically complicate significantly deployment and I need to create tools for monitoring the status of the service. Something that I will do in future but not now, in future I have plan to add more functionality (not only sending notifications) to the service, so in that case it makes more sense to create it.
d) I will send e-mails by using Indy components and SMTP server.
If you are not willing to create the service now, I think you are stuck with the scenario you describe. There are some things though you could do to circumvent the problem of no user firing up the client anymore while there are still pending messages.
You could add a commandline utility (or commandline parameter as bepe4711 suggested) that will only check for pending messages and try to send them.
Add this commandline utility to the StartUp folder or Run key in the registry. This way messages will at least get sent when the computer restarts, even if the user does not fire up the your app.
Add a scheduled task to run this utility at least once every day. The scheduled task can be added by code or by your installer.
If you do both, you will only have to worry about pending messages of users that never start their computer again.
Perhaps you can add a parameter to your client which causes it to just look at the pending notifications and send them. After this it can terminate itself. It will just act like some kind of service.
Then you install the client on the server and start it every x minutes.
I do something very similar to the approach you describe. Instead of sending emails I need to call a web service. My application is installed on several laptops and they are commonly not connected to any network.
When my application raises an exception I collect various bits of information including user comments and screen shots. Then I attempt to send this to our web service. If by chance the web service is not available. (i.e. not connected to the internet or web service is down) I write the results to an XML file on disk in the User Profile (App_Data) directory.
The one major difference is I don't poll to check to see if the server is up. I attempt to send them again on the startup of the application.
If both Systems are running on Windows, have a look at MS Message Queue. It is designed to send notifications to systems, which are not allways online. I did it in .Net, there are already easy to use classes implemented. Not sure about Delphi.
Latest version of Windows uses much more the Windows Task Scheduler, and now task can be fired on event (i.e. when a network card gets connected...). You could write a separate utility that tries to send pending notification, even if noone is logged in.