I've been experiencing troubles with spring-boot consumer. I compared the work of two consumers.
First consumer:
import com.rabbitmq.client.*;
import java.io.IOException;
public class Recv {
private final static String QUEUE_NAME = "hello";
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
Consumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope,
AMQP.BasicProperties properties, byte[] body) throws IOException {
}
};
channel.basicConsume(QUEUE_NAME, true, consumer);
}
}
Second consumer:
#Controller
public class Consumer {
#RabbitListener(queues = "hello")
public void processMessage(Message message) {
}
}
There are no config files for spring-boot consumer installed, everything goes by default.
On my computer first one works 10 times faster. What might be the problem?
The default prefetch (basicQos) for Spring AMQP consumers is 1 which means only 1 message is outstanding at the consumer at any one time; configure the rabbitListenerContainerFactory #Bean to set the prefetchCount to something larger.
You will have to override the default boot-configured #Bean.
Related
Little background: I am working on a topology using Apache Storm, I thought why not use dependency injection in it, but I was not sure how it will behave on cluster environment when topology deployed to cluster. I started looking for answers on if DI is good option to use in Storm topologies, I came across some threads about Apache Spark where it was mentioned serialization is going to be problem and saw some responses for apache storm along the same lines. So finally I decided to write a sample topology with google guice to see what happens.
I wrote a sample topology with two bolts, and used google guice to injects dependencies. First bolt emits a tick tuple, then first bolt creates message, bolt prints the message on log and call some classes which does the same. Then this message is emitted to second bolt and same printing logic there as well.
First Bolt
public class FirstBolt extends BaseRichBolt {
private OutputCollector collector;
private static int count = 0;
private FirstInjectClass firstInjectClass;
#Override
public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
collector = outputCollector;
Injector injector = Guice.createInjector(new Module());
firstInjectClass = injector.getInstance(FirstInjectClass.class);
}
#Override
public void execute(Tuple tuple) {
count++;
String message = "Message count "+count;
firstInjectClass.printMessage(message);
log.error(message);
collector.emit("TO_SECOND_BOLT", new Values(message));
collector.ack(tuple);
}
#Override
public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
outputFieldsDeclarer.declareStream("TO_SECOND_BOLT", new Fields("MESSAGE"));
}
#Override
public Map<String, Object> getComponentConfiguration() {
Config conf = new Config();
conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 10);
return conf;
}
}
Second Bolt
public class SecondBolt extends BaseRichBolt {
private OutputCollector collector;
private SecondInjectClass secondInjectClass;
#Override
public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
collector = outputCollector;
Injector injector = Guice.createInjector(new Module());
secondInjectClass = injector.getInstance(SecondInjectClass.class);
}
#Override
public void execute(Tuple tuple) {
String message = (String) tuple.getValue(0);
secondInjectClass.printMessage(message);
log.error("SecondBolt {}",message);
collector.ack(tuple);
}
#Override
public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
}
}
Class in which dependencies are injected
public class FirstInjectClass {
FirstInterface firstInterface;
private final String prepend = "FirstInjectClass";
#Inject
public FirstInjectClass(FirstInterface firstInterface) {
this.firstInterface = firstInterface;
}
public void printMessage(String message){
log.error("{} {}", prepend, message);
firstInterface.printMethod(message);
}
}
Interface used for binding
public interface FirstInterface {
void printMethod(String message);
}
Implementation of interface
public class FirstInterfaceImpl implements FirstInterface{
private final String prepend = "FirstInterfaceImpl";
public void printMethod(String message){
log.error("{} {}", prepend, message);
}
}
Same way another class that receives dependency via DI
public class SecondInjectClass {
SecondInterface secondInterface;
private final String prepend = "SecondInjectClass";
#Inject
public SecondInjectClass(SecondInterface secondInterface) {
this.secondInterface = secondInterface;
}
public void printMessage(String message){
log.error("{} {}", prepend, message);
secondInterface.printMethod(message);
}
}
another interface for binding
public interface SecondInterface {
void printMethod(String message);
}
implementation of second interface
public class SecondInterfaceImpl implements SecondInterface{
private final String prepend = "SecondInterfaceImpl";
public void printMethod(String message){
log.error("{} {}", prepend, message);
}
}
Module Class
public class Module extends AbstractModule {
#Override
protected void configure() {
bind(FirstInterface.class).to(FirstInterfaceImpl.class);
bind(SecondInterface.class).to(SecondInterfaceImpl.class);
}
}
Nothing fancy here, just two bolts and couple of classes for DI. I deployed it on server and it works just fine. The catch/problem though is that I have to initialize Injector in each bolt which makes me question what is side effect of it going to be?
This implementation is simple, just 2 bolts.. what if I have more bolts? what impact it would create on topology if I have to initialize Injector in all bolts?
If I try to initialize Injector outside prepare method I get error for serialization.
i am not sure my understanding to errorHandler and returnExceptions is right or not.
but here is my goal: i sent a message from App_A, use #RabbitListener to receive message in App_B.
according to the doc
https://docs.spring.io/spring-amqp/docs/2.1.3.BUILD-SNAPSHOT/reference/html/_reference.html#annotation-error-handling
i assume if APP_B has a business exception during process the message,through set errorHandler and returnExceptions in a right way on #RabbitListener can let the exception back to App_A.
do I understood correctly?
if i am rigth, how to use it in a right way?
with my code, i get nothing in APP_A .
here is my code in APP_B
errorHandler:
#Component(value = "errorHandler")
public class ErrorHandler implements RabbitListenerErrorHandler {
#Override
public Object handleError(Message arg0, org.springframework.messaging.Message<?> arg1,
ListenerExecutionFailedException arg2) throws ListenerExecutionFailedException {
throw new ListenerExecutionFailedException("msg", arg2, null);
}
}
RabbitListener:
#RabbitListener(
bindings = #QueueBinding(
value = #Queue(value = "MRO.updateBaseInfo.queue", durable = "true"),
exchange = #Exchange(name = "MRO_Exchange", type = ExchangeTypes.DIRECT, durable = "true"),
key = "baseInfoUpdate"
),
// errorHandler = "errorHandler",
returnExceptions = "true"
)
public void receiveLocationChangeMessage(String message){
BaseUpdateMessage newBaseInfo = JSON.parseObject(message, BaseUpdateMessage.class);
dao.upDateBaseInfo(newBaseInfo);
}
and code in APP_A
#Component
public class MessageSender {
#Autowired
private RabbitTemplate rabbitTemplate;
public void editBaseInfo(BaseUpdateMessage message)throws Exception {
//and i am not sure set RemoteInvocationAwareMessageConverterAdapter in this way is right
rabbitTemplate.setMessageConverter(new RemoteInvocationAwareMessageConverterAdapter());
rabbitTemplate.convertAndSend("MRO_Exchange", "baseInfoUpdate", JSON.toJSONString(message));
}
}
i am very confuse with three points:
1)do i have to use errorHandler and returnExceptions at the same time? i thought errorHandler is something like a postprocessor that let me custom exception.if i don't need a custom exception can i just set returnExceptions with out errorHandler ?
2)should the method annotated with #RabbitListener return something or void is just fine?
3)in the sender side(my situation is APP_A), does have any specific config to catch the exception?
my workspace environment:
Spring boot 2.1.0
rabbitMQ server 3.7.8 on docker
1) No, you don't need en error handler, unless you want to enhance the exception.
2) If the method returns void; the sender will end up waiting for timeout for a reply that will never arrive, just in case an exception might be thrown; that is probably not a good use of resources. It's better to always send a reply, to free up the publisher side.
3) Just the RemoteInvocationAwareMessageConverterAdapter.
Here's an example:
#SpringBootApplication
public class So53846303Application {
public static void main(String[] args) {
SpringApplication.run(So53846303Application.class, args);
}
#RabbitListener(queues = "foo", returnExceptions = "true")
public String listen(String in) {
throw new RuntimeException("foo");
}
#Bean
public ApplicationRunner runner(RabbitTemplate template) {
template.setMessageConverter(new RemoteInvocationAwareMessageConverterAdapter());
return args -> {
try {
template.convertSendAndReceive("foo", "bar");
}
catch (Exception e) {
e.printStackTrace();
}
};
}
}
and
org.springframework.amqp.AmqpRemoteException: java.lang.RuntimeException: foo
at org.springframework.amqp.support.converter.RemoteInvocationAwareMessageConverterAdapter.fromMessage(RemoteInvocationAwareMessageConverterAdapter.java:74)
at org.springframework.amqp.rabbit.core.RabbitTemplate.convertSendAndReceive(RabbitTemplate.java:1500)
at org.springframework.amqp.rabbit.core.RabbitTemplate.convertSendAndReceive(RabbitTemplate.java:1433)
at org.springframework.amqp.rabbit.core.RabbitTemplate.convertSendAndReceive(RabbitTemplate.java:1425)
at com.example.So53846303Application.lambda$0(So53846303Application.java:28)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:794)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:324)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
at com.example.So53846303Application.main(So53846303Application.java:15)
Caused by: java.lang.RuntimeException: foo
at com.example.So53846303Application.listen(So53846303Application.java:20)
As you can see, there is a local org.springframework.amqp.AmqpRemoteException with the cause being the actual exception thrown on the remote server.
In a Spring AMQP project, I would like to get the number of messages in a certain queue (to make decisions based on that number of messages) in RabbitMQ in real time (I can't use the management plugin).
The basic configuration is this:
#Bean(name="managementServerHandler")
public ManagementServerHandler managementServerHandler(){
return new ManagementServerHandler();
}
#Bean
public MessageListenerAdapter broadcastManagementServerHandler() {
return new MessageListenerAdapter(managementServerHandler(), "handleMessage");
}
#Bean(name="broadcastManagementMessageListenerContainer")
public SimpleMessageListenerContainer broadcastManagementMessageListenerContainer()
{
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(_connectionFactory());
container.setQueueNames( REQUEST_MANAGEMENT_QUEUE );
container.setMessageListener(broadcastManagementServerHandler());
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
container.setAutoDeclare(true);
container.setAutoStartup(true);
container.setConcurrentConsumers(1);
container.setRabbitAdmin((RabbitAdmin)_amqpAdmin());
container.setPrefetchCount(50);
container.setDeclarationRetries(3);
container.setMissingQueuesFatal(true);
container.setFailedDeclarationRetryInterval(1000);
container.setRecoveryInterval(400);
return container;
}
Where the "ManagementServerHandler" is just:
public class ManagementServerHandler implements ServletContextAware, MessageListener
{
#Override
public void onMessage(Message msg)
{....}
}
I need the number of queued messages in the onMessage method, but I can't find the way to do it.
I asked this question, but I don't know how to get the AMQP channel:
RabbitMQ and queue data
Thanks!
Use RabbitAdmin.getQueueProperties(queue)
/**
* Returns 3 properties {#link #QUEUE_NAME}, {#link #QUEUE_MESSAGE_COUNT},
* {#link #QUEUE_CONSUMER_COUNT}, or null if the queue doesn't exist.
*/
#Override
public Properties getQueueProperties(final String queueName) {
I'm using Spring-AMQP-rabbit in one of applications which acts as a message-consumer. The queues are created and subscribed to the exchange at startup.
My problem:
When the RabbitMq server is restarted or removed and added completely, the Queue's are not recreated. The connection to the RabbitMq server is re-stored, but not the Queues.
I've tried to do the queue admin within a ConnectionListener but that hangs on startup. I guess the admin is connection aware and should do queue management upon connection restore isn't?
My Queues are created by a service:
#Lazy
#Service
public class AMQPEventSubscriber implements EventSubscriber {
private final ConnectionFactory mConnectionFactory;
private final AmqpAdmin mAmqpAdmin;
#Autowired
public AMQPEventSubscriber(final AmqpAdmin amqpAdmin,
final ConnectionFactory connectionFactory,
final ObjectMapper objectMapper) {
mConnectionFactory = connectionFactory;
mAmqpAdmin = amqpAdmin;
mObjectMapper = objectMapper;
}
#Override
public <T extends DomainEvent<?>> void subscribe(final Class<T> topic, final EventHandler<T> handler) {
final EventName topicName = topic.getAnnotation(EventName.class);
if (topicName != null) {
final MessageListenerAdapter adapter = new MessageListenerAdapter(handler, "handleEvent");
final Jackson2JsonMessageConverter converter = new Jackson2JsonMessageConverter();
converter.setJsonObjectMapper(mObjectMapper);
adapter.setMessageConverter(converter);
final Queue queue = new Queue(handler.getId(), true, false, false, QUEUE_ARGS);
mAmqpAdmin.declareQueue(queue);
final Binding binding = BindingBuilder.bind(queue).to(Constants.DOMAIN_EVENT_TOPIC).with(topicName.value());
mAmqpAdmin.declareBinding(binding);
final SimpleMessageListenerContainer listener = new SimpleMessageListenerContainer(mConnectionFactory);
listener.setQueues(queue);
listener.setMessageListener(adapter);
listener.start();
} else {
throw new IllegalArgumentException("subscribed Event type has no exchange key!");
}
}
}
Part of my handler app:
#Component
public class FooEventHandler implements EventHandler<FooEvent> {
private final UserCallbackMessenger mUserCallbackMessenger;
private final HorseTeamPager mHorseTeamPager;
#Autowired
public FooEventHandler(final EventSubscriber subscriber) {
subscriber.subscribe(FooEvent.class, this);
}
#Override
public void handleEvent(final FooEvent event) {
// do stuff
}
}
I wonder why out-of-the-box feature with the RabbitAdmin and beans for Broker entities doesn't fit your requirements:
A further benefit of doing the auto declarations in a listener is that if the connection is dropped for any reason (e.g. broker death, network glitch, etc.) they will be applied again the next time they are needed.
See more info in the Reference Manual.
Even though this is not described in the Spring documentation, a websocket connect should lead to a connection upgrade response (101 status).
#Configuration
#EnableWebSocketMessageBroker
public class WebSocketConfig<S extends ExpiringSession> extends AbstractSessionWebSocketMessageBrokerConfigurer<S>{
#Override
public void configureMessageBroker(MessageBrokerRegistry config) {
config.enableSimpleBroker("/topic", "/queue");
config.setApplicationDestinationPrefixes("/mobile-server");
config.setUserDestinationPrefix("/mobile-user");
}
#Override
public void configureStompEndpoints(StompEndpointRegistry registry) {
registry
.addEndpoint("/ws")
.setHandshakeHandler(new DefaultHandshakeHandler(new TomcatRequestUpgradeStrategy()))
.setAllowedOrigins("*")
.withSockJS()
.setSessionCookieNeeded(false)
;
}
}
However, I get a 200 status with a "Welcome to SockJS" message which is generated by TransportHandlingSockJsService in stead of the WebSocketHttpRequestHandler which would generate the upgrade AFAIK
#Configuration
public class WebSocketSecurity extends AbstractSecurityWebSocketMessageBrokerConfigurer{
#Override
protected boolean sameOriginDisabled() {
return true;
}
#Override
protected void configureInbound(MessageSecurityMetadataSourceRegistry messages) {
messages
.nullDestMatcher().permitAll()
.simpSubscribeDestMatchers("/user/queue/errors").permitAll()
.simpDestMatchers("/mobile-server/**").hasRole("ENDUSER")
.simpSubscribeDestMatchers("/user/**", "/topic/**").hasRole("ENDUSER")
.anyMessage().denyAll();
}
}
When I change the config to
#Override
public void configureStompEndpoints(StompEndpointRegistry registry) {
registry
.addEndpoint("/ws")
.setHandshakeHandler(new DefaultHandshakeHandler(new TomcatRequestUpgradeStrategy()))
.setAllowedOrigins("*");
}
to my surprise a call to /ws does lead to a connection upgrade 101. I'm surprised, since the documentation and all examples uniformly use the withSockJS() and the start of any websocket connection AFAIK is a request upgrade.
I can choose to force the upgrade by connecting to /ws/websocket (also not documented). So, I'm not sure what is best.
Any suggestions?
This is expected behavior. It's how the SockJS protocol works:
http://sockjs.github.io/sockjs-protocol/sockjs-protocol-0.3.3.html. There is an initial "greeting" request and then the client starts trying transports one at a time.