I'm using the spring-boot-starter-amqp library to:
Wait indefinitely until a message can be received.
Process the message, which may take several minutes.
Terminate the program
I also would like to the set the consumer priority. However, I don't know how to set the x-priority consumer argument, as the documentation only mentions how to do this with an asynchronous consumer, not with AmqpTemplate or RabbitTemplate. Any help would be appreciated.
ApplicationRunner class:
private final AmqpTemplate template;
private final ApplicationContext applicationContext;
...
#Override
public void run(ApplicationArguments args) {
Message message = template.receive("analyses", -1);
if (message == null) throw new IllegalStateException("No message received");
byte[] body = message.getBody();
// todo: process the message
System.out.printf("[x] Received '%s'%n", new String(body, StandardCharsets.UTF_8));
context.close();
}
In a configuration class:
#Bean
public RabbitTemplate amqpTemplate(final ConnectionFactory connectionFactory) {
return new RabbitTemplate(connectionFactory);
}
It is not currently supported; the consumer is created in a private method createConsumer() which doesn't pass any arguments to basicConsume().
Please open a new feature issue on GitHub; it won't be difficult to implement and we have releases planned for next week.
https://github.com/spring-projects/spring-amqp/issues
Related
I have an app that's using Boot 2.0 with webflux, and has an endpoint returning a Flux of ServerSentEvent. The events are created by leveraging spring-amqp to consume messages off a RabbitMQ queue. My question is: How do I best bridge the MessageListener's configured listener method to a Flux that can be passed up to my controller?
Project Reactor's create section mentions that it "can be very useful to bridge an existing API with the reactive world - such as an asynchronous API based on listeners", but I'm unsure how to hook into the message listener directly since it's wrapped in the DirectMessageListenerContainer and MessageListenerAdapter. Their example from the create section:
Flux<String> bridge = Flux.create(sink -> {
myEventProcessor.register(
new MyEventListener<String>() {
public void onDataChunk(List<String> chunk) {
for(String s : chunk) {
sink.next(s);
}
}
public void processComplete() {
sink.complete();
}
});
});
So far, the best option I have is to create a Processor and simply call onNext() each time in the RabbitMQ listener method to manually produce an event.
I have something like this:
#SpringBootApplication
#RestController
public class AmqpToWebfluxApplication {
public static void main(String[] args) {
ConfigurableApplicationContext applicationContext = SpringApplication.run(AmqpToWebfluxApplication.class, args);
RabbitTemplate rabbitTemplate = applicationContext.getBean(RabbitTemplate.class);
for (int i = 0; i < 100; i++) {
rabbitTemplate.convertAndSend("foo", "event-" + i);
}
}
private TopicProcessor<String> sseFluxProcessor = TopicProcessor.share("sseFromAmqp", Queues.SMALL_BUFFER_SIZE);
#GetMapping(value = "/sseFromAmqp", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> getSeeFromAmqp() {
return this.sseFluxProcessor;
}
#RabbitListener(id = "fooListener", queues = "foo")
public void handleAmqpMessages(String message) {
this.sseFluxProcessor.onNext(message);
}
}
The TopicProcessor.share() allows to have many concurrent subscribers which we get when we return this TopicProcessor as a Flux to our /sseFromAmqp REST request via WebFlux.
The #RabbitListener just delegates its received messages to that TopicProcessor.
In the main() I have a code to confirm that I can publish to the TopicProcessor even if there is no subscribers.
Tested with two separate curl sessions and published messages to the queue via RabbitMQ Management Plugin.
By the way I use share() because of: https://projectreactor.io/docs/core/release/reference/#_topicprocessor
from multiple upstream Publishers when created in the shared configuration
That' because that #RabbitListener really can be called from different ListenerContainer threads, concurrently.
UPDATE
Also I moved this sample to my Sandbox: https://github.com/artembilan/sendbox/tree/master/amqp-to-webflux
Let's suppose you want to have a single RabbitMQ listener that somehow puts messages to one or more Flux(es). Flux.create is indeed a good way how to create such a Flux.
Let's start with Messaging with RabbitMQ Spring guide and try to adapt it.
The original Receiver would have to be modified in order to be able to put received messages to a FluxSink.
#Component
public class Receiver {
/**
* Collection of sinks enables more than one subscriber.
* Have to keep in mind that the FluxSink instance that the emitter works with, is provided per-subscriber.
*/
private final List<FluxSink<String>> sinks = new ArrayList<>();
/**
* Adds a sink to the collection. From now on, new messages will be put to the sink.
* Method will be called when a new Flux is created by calling Flux.create method.
*/
public void addSink(FluxSink<String> sink) {
sinks.add(sink);
}
public void receiveMessage(String message) {
sinks.forEach(sink -> {
if (!sink.isCancelled()) {
sink.next(message);
} else {
// If canceled, don't put any new messages to the sink.
// Sink is canceled when a subscriber cancels the subscription.
sinks.remove(sink);
}
});
}
}
Now we have a receiver that puts RabbitMQ messages to sink. Then, creating a Flux is rather simple.
#Component
public class FluxFactory {
private final Receiver receiver;
public FluxFactory(Receiver receiver) { this.receiver = receiver; }
public Flux<String> createFlux() {
return Flux.create(receiver::addSink);
}
}
Receiver bean is autowired to the factory. Of course, you don't have to create a special factory. This only demonstrates the idea how to use the Receiver to create the Flux.
The rest of the application from Messaging with RabbitMQ guide may stay the same, including the bean instantiation.
#SpringBootApplication
public class Application {
...
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(queueName);
container.setMessageListener(listenerAdapter);
return container;
}
#Bean
MessageListenerAdapter listenerAdapter(Receiver receiver) {
return new MessageListenerAdapter(receiver, "receiveMessage");
}
...
}
I used similar design to adapt Twitter streaming API sucessfuly. Though, there may be a nicer way how to do it.
I have an application that does not recieve ordinary HTTP requests through a controller, instead it listens to and receives messages (AMQP protocol) in order to initiate it's logic flow.
My application may receive and handle more than 1 message at a time. I have an object that will be collecting information/data throughout the process, in several different services/classes, in order for me to use it at the end.
But I need the data to be seperated per message received, as a "Scoped" injection would seperate the injected instance from other HTTP requests.
My usecase is therefor very similar to how I would use a Scoped injected object in an ordinary API, but instead of a new HTTP request, I receive a message in my listeners.
Is there any way that I can create a custom scope, for every message received, either through some kind of configuration, or having the code create a new scope as the first thing in my Listener.MessageReceived(Message message) method?
Imagine a flow like this:
public class Listener {
ServiceClassA serviceClassA //injected in constructor
CustomLogger customLogger // (HAS TO BE SAME OBJECT INJECTED INTO ServiceClassA, ServiceClassB and Listener)
public void ReceiveMessage(Message message) {
using (var scope = CreateNewScope()) {
try {
serviceClassA.DoStuff();
} catch(Exception e) {
Console.Write(customLogger.GetLogs())
}
}
}
}
public class ServiceClassA {
ServiceClassB serviceClassB //injected in constructor
CustomLogger customLogger //(HAS TO BE SAME OBJECT INJECTED INTO ServiceClassA, ServiceClassB and Listener)
public void DoStuff() {
customLogger = ResolveCustomLogger(); // how do I make sure I can get/resolve the same object as in Listener (without having to pass parameters)
var data = // does stuff
customLogger.Log(data);
serviceClassB.DoStuff();
}
}
public class ServiceClassB {
CustomLogger customLogger //(HAS TO BE SAME OBJECT INJECTED INTO ServiceClassA, ServiceClassB and Listener)
public void DoStuff() {
customLogger = ResolveCustomLogger(); // how do I make sure I can get/resolve the same object as in Listener (without having to pass parameters)
var data = // does other stuff
customLogger.Log(data);
}
}
My CustomLogger may not only be used 1 or 2 service layers down, there might be many layers, and I might only want to use the CustomLogger in the bottom on, yet I want it accessible in the top level afterwards, to retrieve the data stored in it.
Thank you very much.
You can inject a ServiceScopyFactory in the class that reacts to messages from the queue, then for each message it receives it can create a scope, from which it requests a MessageHandler dependency.
The code sample below does exactly this (and it also deals with sessions on the queue, but that should make no difference for creating the scope).
public class SessionHandler : ISessionHandler
{
public readonly string SessionId;
private readonly ILogger<SessionHandler> Logger;
private readonly IServiceScopeFactory ServiceScopeFactory;
readonly SessionState SessionState;
public SessionHandler(
ILogger<SessionHandler> logger,
IServiceScopeFactory serviceScopeFactory,
string sessionId)
{
Logger = logger;
ServiceScopeFactory = serviceScopeFactory;
SessionId = sessionId
SessionState = new SessionState();
}
public async Task HandleMessage(IMessageSession session, Message message, CancellationToken cancellationToken)
{
Logger.LogInformation($"Message of {message.Body.Length} bytes received.");
// Deserialize message
bool deserializationSuccess = TryDeserializeMessageBody(message.Body, out var incomingMessage);
if (!deserializationSuccess)
throw new NotImplementedException(); // Move to deadletter queue?
// Dispatch message
bool handlingSuccess = await HandleMessageWithScopedHandler(incomingMessage, cancellationToken);
if (!handlingSuccess)
throw new NotImplementedException(); // Move to deadletter queue?
}
/// <summary>
/// Instantiate a message handler with a service scope that lasts until the message handling is done.
/// </summary>
private async Task<bool> HandleMessageWithScopedHandler(IncomingMessage incomingMessage, CancellationToken cancellationToken)
{
try
{
using IServiceScope messageHandlerScope = ServiceScopeFactory.CreateScope();
var messageHandlerFactory = messageHandlerScope.ServiceProvider.GetRequiredService<IMessageHandlerFactory>();
var messageHandler = messageHandlerFactory.Create(SessionState);
await messageHandler.HandleMessage(incomingMessage, cancellationToken);
return true;
}
catch (Exception exception)
{
Logger.LogError(exception, $"An exception occurred when handling a message: {exception.Message}.");
return false;
}
}
private bool TryDeserializeMessageBody(byte[] body, out IncomingMessage? incomingMessage)
{
incomingMessage = null;
try
{
incomingMessage = IncomingMessage.Deserialize(body);
return true;
}
catch (MessageDeserializationException exception)
{
Logger.LogError(exception, exception.Message);
}
return false;
}
}
Now whenever a MessageHandlerFactory is instantiated (which happens for each message received from the queue), any scoped dependencies requested by the factory will live until the MessageHandler.HandleMessage() task finishes.
I created a message handler factory so that the SessionHandler could pass non-DI-service arguments to the constructor of the MessageHandler (the SessionState object in this case) in addition to the DI-services. It is the factory who requests the (scoped) dependencies and passes them to the MessageHandler. If you are not using sessions then you might not need the factory, and you can instead fetch a MessageHandler from the scope directly.
I have the following two cases
In case of ExceptionA : retrying for finite number of times
and finally when number of retrials exhausted, message is written in
a dead letter queue
In case of ExceptionB : simply, message should be written to
dead letter queue
I want to support the two cases on the same listener container factory and the same queue.
I already have the following configuration to support case 1 successfully:
#Bean
public RetryOperationsInterceptor workMessagesRetryInterceptor() {
return RetryInterceptorBuilder.stateless()
.maxAttempts(5)
.backOffOptions(1000, 2, 10000)
.recoverer(new RejectAndDontRequeueRecoverer())
.build();
}
#Bean
public SimpleRabbitListenerContainerFactory myRabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMaxConcurrentConsumers(8);
factory.setAdviceChain(workMessagesRetryInterceptor());
return factory;
}`
Now I want to extend the previous configuration to support case 2 too.
Edit, thanks Gary for your fast response.
Here you are my new configuration, but I still get retrials on both the two exceptions : ListenerExecutionFailedException , AmqpRejectAndDontRequeueException
#Bean
public SimpleRetryPolicy rejectionRetryPolicy(){
Map<Class<? extends Throwable> , Boolean> exceptionsMap = new HashMap<Class<? extends Throwable> , Boolean>();
exceptionsMap.put(ListenerExecutionFailedException.class, true); //retriable
exceptionsMap.put(AmqpRejectAndDontRequeueException.class, false);//not retriable
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy(5 , exceptionsMap );
return retryPolicy;
}
#Bean
public RetryOperationsInterceptor workMessagesRetryInterceptor() {
return RetryInterceptorBuilder.stateless().retryPolicy(rejectionRetryPolicy())
//.backOffOptions(1000, 2, 10000)
//.recoverer(new RejectAndDontRequeueRecoverer())
.build();
}
Provide a SimpleRetryPolicy with a map of exceptions and booleans (whether or not to retry). You can optionally traverse the exception cause tree to find the specific exception. See the Javadocs for SimpleRetryPolicy.
I am using MessageListener to read messages from the specified queue and want to perform processing once i got message on queue.
Here is my code
#Bean
public MessageListenerContainer messageListenerContainer()
{
SimpleMessageListenerContainer messageListenerContainer = new SimpleMessageListenerContainer();
messageListenerContainer.setConnectionFactory(connectionFactory());
messageListenerContainer.setQueueNames("queue");
messageListenerContainer.setMessageListener(vceListener());
return messageListenerContainer;
}
#Bean
public VListener vceListener()
{
return new VListener();
}
#Component
public class VListener implements MessageListener
{
#Override
public void onMessage(Message message)
{
//start process
}
}
My processing time is more for one message and when one message process is going on I cant process another message which is coming on the queue.
so, cant I process multiple messages in parallel
Add
messageListenerContainer.setConcurrentConsumers(5);
(or whatever concurrency you need).
See the documentation for other container properties.
I am creating a websocket server that interfaces with a web service endpoint on one side and another which receives web socket connection requests from multiple clients. Here are two approaches that I found:
Implement a web socket configurer and web socket handler as such:
Configurer
#Configuration
#EnableWebSocket
public class TestConfig implements WebSocketConfigurer {
#Override
public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
registry.addHandler(testHandler(), "/testHandler")
.addInterceptors(new HttpSessionHandshakeInterceptor())
.withSockJS();
}
#Bean
public WebSocketHandler testHandler() {
return new TestHandler();
}
Handler
public class TestHandler extends TextWebSocketHandler {
#Override
public void afterConnectionEstablished(WebSocketSession session) throws Exception {
//Take request params and check if a current subscription to external webservice exists, if yes then directly add this session to a map cache repository with the subscription id as key
//If it is a new request then add session to a map cache repository and make new subscription to the external webservice
}
#Override
public void handleTextMessage(WebSocketSession session, TextMessage message) {
}
Configure a message broker endpoint to be subscribed to called /subscribe
public class TestWebSocketConfig implement WebSocketMessageBrokerConfigurer {
#Override
public void addArgumentResolvers(List<HandlerMethodArgumentResolver> arg0) {}
#Override
public void addReturnValueHandlers(List<HandlerMethodReturnValueHandler> arg0) {}
#Override
public void configureClientInboundChannel(ChannelRegistration arg0) {
System.out.println("");
}
#Override
public void configureClientOutboundChannel(ChannelRegistration arg0) {
}
#Override
public void configureMessageBroker(MessageBrokerRegistry registry) {
}
#Override
public boolean configureMessageConverters(List<MessageConverter> arg0) {
return true;
}
#Override
public void configureWebSocketTransport(WebSocketTransportRegistration arg0) {}
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/subscribe").withSockJS();
}
Create controller where websocket clients can communicate with
#Controller
public class SubscriptionController {
#Autowired
private SimpMessagingTemplate template;
#MessageMapping("/subscribe1")
#SendTo("/subscribe")
public void addSubscription(String message) {
System.out.println("hi");
}
Here is my question, am I misunderstanding somewhere where these two methods I speak of meant to be combined together? I was using a tomcat implementation of websocket before which matches method 1 which gives me easy direct control over sessions as I would like to be able to reuse web service subscriptions to avoid duplicate request from distinct clients and also a single requests may map to more than one subscription requests to the external webservice. Yet it seems method 2 would push all data requests to the same "/subscribe" endpoint and all connected clients would be receiving the same data, which is not what I am trying to accomplish. It also seems like the message broker api is limited as it does not allow me access to the subscribed sessions where I can control which sessions the receiving data will be sent to. I realized I had to switch to spring websocket as I needed built in browser compatibility fallback offered by SockJS and automatic heartbeat function offered by Stomp.js.
i think i found my answer, method 1 and 2 can be used side by side but not together. Method 2 is used when i want to implement a message broker that can create multiple channel destinations which many users can subscribe to the same destination. Now the question is how i can check whether i can check the number of subscriptions periodically for each existing destination