I am facing a problem with masstransit. I want to use SQS and one queue only. I want masstransit to always send the messages to that queue and receive them on a specific endpoint. (So not publishing).
The sending to the queue works but for some reason I'm having problems receiving messages. The documentation examples do not work for me when connecting a consumer to an endpoint.. It complains about SNS (which I am not using)
Does anyone have a simple example?
Ok, so I found out that if you want to send messages to a queue, than SQS should be sufficient in combination with Masstransit. But when you want to recieve those messages, then Masstransit expects credentials that allow the creation of SNS topics.
I thought I had more say in what queues Masstransit will use for communication. But when you use Masstransit with amazon it will create it's own queues and SNS topics based on it's needs.
So when using Masstransit, like i did in my case. Give it AWS IAM user credentials that have the proper permissions to create SQS queues and SNS topics.
Related
I have an application running Laravel 6.1. There are clients which connect to it via laravel websockets and listen for events. I have an external service which sends post requests to this server which will then raise an event, and the websocket clients see it. I am in the dev stage, and it's not been deployed yet, this is what I'm currently researching. I use Docker, so there's an nginx container, a php container, and a Mysql container(in production, the containers will use RDS though)
This works fine in development, but the plan is to deploy in ECS, with Elastic Beanstalk, as it enables multiple containers per EC2 instance. I was planning on having these instances auto scale with a load balancer, so my question is how can I make the incoming events be raised and visible on all the servers? For example, the post request may hit one instance and the clients connected to that instance would see that the event was raised, but the clients connected to another instance will not see the raised event. Is this accurate? I'd imagine the events will have to be sent to some kind of "queue" which is monitored by all instances, but not sure how to implement that with Laravel or if there's a simpler faster way.
Based on the comments.
The proposed solution involves the use of sns instead of the SQS.
The reason is that sns allows delivery of messages to multiple recipients at the same time. In contrast, SQS is designed for one delivery of messages to only one recipient, unless used in fan out architecture.
Can someone tell me if service workers (https://developers.google.com/web/fundamentals/primers/service-workers/) support server sent events (SSE)?
As per the service worker specification's issue tracker, service workers should be able to intercept EventSource.
That being said, I've never seen an example of a service worker that did anything meaningful with server-sent events.
According to what I read of DataFlow, the Pub/Sub datasource only gives the message body to work with in the pipeline. We have a use-case where we want to inspect the attributes of the message to make certain decisions. Is there any way of achieving this currently? I'm open to extending the Pub/Sub I/O to incorporate this if required.
Currently, there is no way to access the message attributes of your messages via the PubsubIO connector, but it would clearly be useful to do so. This is tracked in Apache Beam (incubating) as the issue BEAM-404.
I recommend following this issue to keep abreast of new developments.
I have a scenario where a request is sent to a service via my client now the response comes inside a message queue in azure, How can I poll the queue at client end and update the view when the response comes say I have to update a label when data is recieved in the queue.
Azure has two types of queues - Azure Queue and Service Bus Queue. Although in theory you can access them from client side (I assume JavaScript) because CORS has been introduced some time ago (Not sure about CORS support for ServiceBusQueue), this might not be the best option.
Problems you might face:
Lot's of clients trying to process messages (locking and releasing), Azure Queue does not support sessions so you would have to either create queue per client or use Service Bus Queue (as I said earlier not sure about CORS) with sessions
What should happen when your client is not online anymore? Does the message stays in the queue? Till when? Expiration?
Different approach
You can do message processing on the server and only notify user about the change using SinglalR. This gives you much better flexibility (one message can trigger notification for many users etc).
SignalR Scaleout with Azure Service Bus
Using SignalR with Azure Table Storage - What architecture?
I'm developing an application that has various types of Notifications. Examples of notifications:
Message Created
Listing Submitted
Listing Approved
I'd like to tie all of these up to SignalR so that any connected clients get updates in real-time.
As far as architecture goes - right now the application is entirely within a single solution hosted on an Azure Website. The triggers for each of these notification types live within this application.
When a trigger is hit, I'd like to tell signalR, "Hey, send this message to the following clients" along with a list of userIds. I'm assuming that it's possible to identify connected clients based on userId... and I'm assuming that the process of send message to clients should be executed outside of the web application, so as to not slow down the MVC app or risk losing data in a broken async call. First question - are these assumptions correct?
Assuming so, this means that I'll need something like a dedicated web/worker role to be sending messages to clients. I could pass messages from my web application directly to this process, but what happens if the process dies? The resiliency concerns lead me to believe that the proper way to pass messages would be via a queue of some sort. Second question - is this a valid train of thought?
Assuming so, this means that I can either use a good ol' Azure SQL database as a queue, but it seems like there are some specialized (and maybe cheaper) services to handle message queueing, such as this:
http://www.windowsazure.com/en-us/develop/net/how-to-guides/queue-service/
Third question: Should this be used as a queueing mechanism for signalR? I'm interested in using Redis for caching in the future... would Redis be better or worse than the queue service?
Final Question:
I've attempted to illustrate my proposed architecture here:
What I'm most unclear on here is how the MVC app will know when to queue, or how the SignalR processes will know when to broadcast. Should the MVC app queue blindly, without caring about connected clients? This seems to introduce a lot of wasted space on the queue, and wasted cycles in the worker roles, since a very small percentage of clients will ever be connected.
The only other approach I can think of is to somehow give the MVC app visibility into the SignalR processes to see if the client is connected... and if they are, then Enqueue. This makes me uncomfortable though because it means I have to hit that red line on the diagram for every trigger that gets hit, which - even if done async - gets me worrying about performance and reliability.
What is the recommended architecture for scalable, performant SignalR message broadcasting? Performance is top priority, followed closely by cost.
Bonus question:
What if some messages are of higher priority than others? Should two queues be used, one of which always gets checked before the other?
If you want to target some users, you'll have to come up with a mechanism, off the top of my head I can give an example, if any user hits a page, you can create a group for that page and push to all users in that group/in that page.
It's not clear to me why you need the queues. Usually users subscribe to some events when hitting a page or by some action like join a chat room, and the server pushes data using those events/functions when appropriate.
For scalability, you can run signalr in different servers, in which case you should use sql server, or service bus or redis as a backplane.
Firstly you need to create a SignalR server to which all the users can connect to. This SignalR server can be created either in the web role or worker role. If you have a huge user base then its better to create the SignalR server on a separate role.
Then wherever the trigger is hit and you want to send messages to users, you have to create a SignalR client (.NET or javascript) and then connect to SignalR server. Then you can send the message to SignalR server which in turn will broadcast to all the other users connected. After that you can disconnect the connection with SignalR server. This way you dont have to use queues to communicate with the SignalR role.
And also to send messages to specific users you can store the socket id's along with their user id's in a table (azure table storage should do) when they connect to SignalR server. Then using socket id you can send messages to specific user.