I have to create a POC to demonstrate that Rebus will work for this scenario -
Multiple subscribers listening to event(s) and handling them accordingly.
How can I configure this using the BuiltInContainer and the config?
Do I need to have multiple queues and multiple endpoints configured and also single instance of the container or multiple?
I suggest you take a look at the pub/sub sample that demonstrates a scenario with three separate applications, one publisher and two subscribers.
In most scenarios I recommend you create one container for each endpoint, which implies one queue per endpoint.
Related
I am new to using loopback, and I'm using loopback4 (which I think is referred to as loopback-next)
I have set up my controllers, models & respositories in order to be able to support CRUD operations to mysql, and that is all fine.
I want my loopback application to also connect to an MQTT server, so that I can subscribe to messages from MQTT, and react to those messages by creating entities in my repositories. In addition, I want to be able to have existing controller methods drop messages onto the MQTT (publish)
I am struggling to understand the right way to do this in the loopback eco-system.
I don't think I want to create a Server - because the documentation describes a server as including a listen port. I don't want my loopback application to be a MQTT server. I just want it to interact with one.
Similarly, I don't think this would be an MQTT bridge, or a datasource.
I suspect, what I want is a service. But I'm not certain.
I would appreciate any advice on how to achieve this integration.
Thanks
LB4 is highly extensible and a very good choice for such integrations. What you need in this case is to have a MQTT connector component. You can refer to the documentation for how to create a component in LB4 here and here.
You can refer to an example component implementation for authorization as well for quicker understanding.
Iam working on a project of big data, where Iam trying to get tweets from Twitter and analyse these tweets and make predictions out of it.
I have followed this tutorial : http://blog.cloudera.com/blog/2012/10/analyzing-twitter-data-with-hadoop-part-2-gathering-data-with-flume/
for getting the tweets. Now Iam planning to build a microservice which can replicate itself as I increase the number of topics on which I want tweets. Now whatever code I have written to gather the tweets with that I want to make a microservice that can take a keyword and create a instance of that code for that keyword and gather tweets, for each keyword an instance should be created.
It will also be helpful if you inform me what tools to use for such application.
Thank you.
I want to make a microservice that can take a keyword and create a instance of that code for that keyword and gather tweets, for each keyword an instance should be created.
You could use kubernetes as an underlying cluster/deployment infrastructure. It has an API that allows you to deploy new services programmatically. So what you would have to do is:
Set up a basic service container for your twitter-service that is available in a container repository.
Then you deploy a first service based on your container. The service configuration will contain the keyword that the service uses as well as information about the kubernetes cluster (how to access the cluster API and where to find the container in the repository).
Now your first service has all the information it needs to automatically create additional service descriptions for kubernetes (with other key words) and deploy those additional services by calling the kubernetes cluster API.
Since the additional services will be passed all the necessary information as well, they themselves can then start even more services and so on.
You probably need to put some effort into figuring out the cluster provisioning, but that can also be done automatically with auto-scaling (available for Google or AWS clouds for example).
A different approach would be to run a horizontally scaled cluster of your basic twitter services that use a self organization algorithm to involve all the keywords put into a database or event queue.
I am making a broker that I want to deploy on three servers and I want to run it on only one one by one.flow that is running called active flow and others are passive.when it runs it send an email that flow is running on server x.
can we do this broker through a flag.
i.e. to implement a distributed deployment approach
The best approach would be to use a multi instance broker so that you only have one node active at any one time, you could then have either in the onInitialize() method of a JCN, or in a timeout control driven flow a branch of subflow which creates an email based on the local hostname.
I'm developing an application that has various types of Notifications. Examples of notifications:
Message Created
Listing Submitted
Listing Approved
I'd like to tie all of these up to SignalR so that any connected clients get updates in real-time.
As far as architecture goes - right now the application is entirely within a single solution hosted on an Azure Website. The triggers for each of these notification types live within this application.
When a trigger is hit, I'd like to tell signalR, "Hey, send this message to the following clients" along with a list of userIds. I'm assuming that it's possible to identify connected clients based on userId... and I'm assuming that the process of send message to clients should be executed outside of the web application, so as to not slow down the MVC app or risk losing data in a broken async call. First question - are these assumptions correct?
Assuming so, this means that I'll need something like a dedicated web/worker role to be sending messages to clients. I could pass messages from my web application directly to this process, but what happens if the process dies? The resiliency concerns lead me to believe that the proper way to pass messages would be via a queue of some sort. Second question - is this a valid train of thought?
Assuming so, this means that I can either use a good ol' Azure SQL database as a queue, but it seems like there are some specialized (and maybe cheaper) services to handle message queueing, such as this:
http://www.windowsazure.com/en-us/develop/net/how-to-guides/queue-service/
Third question: Should this be used as a queueing mechanism for signalR? I'm interested in using Redis for caching in the future... would Redis be better or worse than the queue service?
Final Question:
I've attempted to illustrate my proposed architecture here:
What I'm most unclear on here is how the MVC app will know when to queue, or how the SignalR processes will know when to broadcast. Should the MVC app queue blindly, without caring about connected clients? This seems to introduce a lot of wasted space on the queue, and wasted cycles in the worker roles, since a very small percentage of clients will ever be connected.
The only other approach I can think of is to somehow give the MVC app visibility into the SignalR processes to see if the client is connected... and if they are, then Enqueue. This makes me uncomfortable though because it means I have to hit that red line on the diagram for every trigger that gets hit, which - even if done async - gets me worrying about performance and reliability.
What is the recommended architecture for scalable, performant SignalR message broadcasting? Performance is top priority, followed closely by cost.
Bonus question:
What if some messages are of higher priority than others? Should two queues be used, one of which always gets checked before the other?
If you want to target some users, you'll have to come up with a mechanism, off the top of my head I can give an example, if any user hits a page, you can create a group for that page and push to all users in that group/in that page.
It's not clear to me why you need the queues. Usually users subscribe to some events when hitting a page or by some action like join a chat room, and the server pushes data using those events/functions when appropriate.
For scalability, you can run signalr in different servers, in which case you should use sql server, or service bus or redis as a backplane.
Firstly you need to create a SignalR server to which all the users can connect to. This SignalR server can be created either in the web role or worker role. If you have a huge user base then its better to create the SignalR server on a separate role.
Then wherever the trigger is hit and you want to send messages to users, you have to create a SignalR client (.NET or javascript) and then connect to SignalR server. Then you can send the message to SignalR server which in turn will broadcast to all the other users connected. After that you can disconnect the connection with SignalR server. This way you dont have to use queues to communicate with the SignalR role.
And also to send messages to specific users you can store the socket id's along with their user id's in a table (azure table storage should do) when they connect to SignalR server. Then using socket id you can send messages to specific user.
So we currently have a website that was created using Django. Now, we would like to create a native iOS app that uses the same backend, so we don't have to re-code the whole thing. From my understanding, there are two alternative routes:
1) Call directly Django URLs, which then calls a function. Within that function, create a HTTPResponse, with encoded JSON data and send that back.
2) Create a REST Service from the Django server with something like Tastypie. However, aside from doing straight-forward GET calls to an object, I don't see how we can call custom functions in our Django Models from TastyPie. Can we even do that?
I find it surprising that there is not a lot of information about consuming a web service from iOS with existing backends like Django or RoR. For example, I know that instagram uses Django, but how do they communicate from iOS to their servers?!
Thanks a lot!
I am currently working on an iOS app for iPhone, with Django / Tastypie in the backend. We do both 1 and 2. The resources are offered REST-style (after auth) via Tastypie, and any custom function calls (for example, creating a new user) are handled by views.py at various REST endpoints, which returns JSON.
When you can you should try to use a common way of doing something instead of reinventing the wheel. Given that, REST is a standard style of software architecture for distributed systems and it works very well when you work with entities/objects.
If you have an API where you interact with entities, it is recommended to use REST interfaces. On python you have Tastypie or the newer Django Rest Framework that does almost all the work. As you propose in 2)
If you have an API where you interact with services, like a login, then you should build an RPC service, basically a function with remote access as you explain on 1).
Normally you will need both ways on a robust application. And YES, it is possible to do that. I agree with #sampson-chen, we are doing the same. We have a REST interface with tastypie, and other methods are done with custom RPC services.
The performance in our case is still good, but mostly depends on the methods you call inside your services, for example, a DB query. You have a lot of ways to improve speed, for example using Celery to queue heavy jobs.
Hope it helps.
REST APIs, while very useful, limit you to GET, POST, PUT, DELETE actions, which are performed upon resources. This can make it difficult to express other action types, such as sending an email. There are a few ways I've found to handle this within django/tastypie:
Issue a PUT/PATCH request on an existing resource, setting a flag that lets your backend know to trigger an action. Detecting if a flag was set can be done inside post_save signal handlers (use django-model-utils FieldTracker to see if a field was changed from False to True); this also helps make sure your application logic works the same outside your REST API (such as changes via the admin site, a celery task, an HTML based view, or the Python shell).
Create a non-ORM Resource (e.g. /api/v1/email/) and override the post_list() method, calling your function there.
As mentioned elsewhere, create a subordinate resource (/api/v1/myresource/send/).