Queues not recreated after broker failure - spring-amqp

I'm using Spring-AMQP-rabbit in one of applications which acts as a message-consumer. The queues are created and subscribed to the exchange at startup.
My problem:
When the RabbitMq server is restarted or removed and added completely, the Queue's are not recreated. The connection to the RabbitMq server is re-stored, but not the Queues.
I've tried to do the queue admin within a ConnectionListener but that hangs on startup. I guess the admin is connection aware and should do queue management upon connection restore isn't?
My Queues are created by a service:
#Lazy
#Service
public class AMQPEventSubscriber implements EventSubscriber {
private final ConnectionFactory mConnectionFactory;
private final AmqpAdmin mAmqpAdmin;
#Autowired
public AMQPEventSubscriber(final AmqpAdmin amqpAdmin,
final ConnectionFactory connectionFactory,
final ObjectMapper objectMapper) {
mConnectionFactory = connectionFactory;
mAmqpAdmin = amqpAdmin;
mObjectMapper = objectMapper;
}
#Override
public <T extends DomainEvent<?>> void subscribe(final Class<T> topic, final EventHandler<T> handler) {
final EventName topicName = topic.getAnnotation(EventName.class);
if (topicName != null) {
final MessageListenerAdapter adapter = new MessageListenerAdapter(handler, "handleEvent");
final Jackson2JsonMessageConverter converter = new Jackson2JsonMessageConverter();
converter.setJsonObjectMapper(mObjectMapper);
adapter.setMessageConverter(converter);
final Queue queue = new Queue(handler.getId(), true, false, false, QUEUE_ARGS);
mAmqpAdmin.declareQueue(queue);
final Binding binding = BindingBuilder.bind(queue).to(Constants.DOMAIN_EVENT_TOPIC).with(topicName.value());
mAmqpAdmin.declareBinding(binding);
final SimpleMessageListenerContainer listener = new SimpleMessageListenerContainer(mConnectionFactory);
listener.setQueues(queue);
listener.setMessageListener(adapter);
listener.start();
} else {
throw new IllegalArgumentException("subscribed Event type has no exchange key!");
}
}
}
Part of my handler app:
#Component
public class FooEventHandler implements EventHandler<FooEvent> {
private final UserCallbackMessenger mUserCallbackMessenger;
private final HorseTeamPager mHorseTeamPager;
#Autowired
public FooEventHandler(final EventSubscriber subscriber) {
subscriber.subscribe(FooEvent.class, this);
}
#Override
public void handleEvent(final FooEvent event) {
// do stuff
}
}

I wonder why out-of-the-box feature with the RabbitAdmin and beans for Broker entities doesn't fit your requirements:
A further benefit of doing the auto declarations in a listener is that if the connection is dropped for any reason (e.g. broker death, network glitch, etc.) they will be applied again the next time they are needed.
See more info in the Reference Manual.

Related

mosquito MQTT message handler getting messages with some delay not real time

Following is the MQTT configuration to listening event.
For high message load around 100 message per second I noticed messages not received realtime on handler.
public class VehicleEventMqttConfig {
#Value("${mqtt.auto-startup.vehicleEvent:false}")
private boolean autoStartup;
#Value("${mqtt.completion-timeout.vehicleEvent:30000}")
private int completionTimeout;
#Bean
public MessageChannel vehicleMqttInputChannel() {
return new DirectChannel();
}
#Bean
public MessageProducer inboundVehicleEvent(
final MqttPahoClientFactory mqttPahoClientFactory,
final MqttAdapters adapters,
#Value("${mqtt.topic.vehicleEvent}") final String topic) {
log.info("Register vehicleEvent mqtt");
if (StringUtils.isEmpty(topic)) {
log.warn("vehicleEvent disabled!");
return null;
}
final MqttPahoMessageDrivenChannelAdapter adapter =
new MqttPahoMessageDrivenChannelAdapter(
getClientIdWithHost("inboundVehicleEvent"), mqttPahoClientFactory, topic);
adapter.setCompletionTimeout(completionTimeout);
adapter.setConverter(new DefaultPahoMessageConverter());
adapter.setOutputChannel(vehicleMqttInputChannel());
adapter.setAutoStartup(autoStartup);
adapter.setQos(1);
adapters.add(adapter);
return adapter;
}
#Bean
#ServiceActivator(inputChannel = "vehicleMqttInputChannel")
public MessageHandler vehicleEventHandler() {
return new VehicleEventMessageHandler();
}
}

Dependency Injection in Apache Storm topology

Little background: I am working on a topology using Apache Storm, I thought why not use dependency injection in it, but I was not sure how it will behave on cluster environment when topology deployed to cluster. I started looking for answers on if DI is good option to use in Storm topologies, I came across some threads about Apache Spark where it was mentioned serialization is going to be problem and saw some responses for apache storm along the same lines. So finally I decided to write a sample topology with google guice to see what happens.
I wrote a sample topology with two bolts, and used google guice to injects dependencies. First bolt emits a tick tuple, then first bolt creates message, bolt prints the message on log and call some classes which does the same. Then this message is emitted to second bolt and same printing logic there as well.
First Bolt
public class FirstBolt extends BaseRichBolt {
private OutputCollector collector;
private static int count = 0;
private FirstInjectClass firstInjectClass;
#Override
public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
collector = outputCollector;
Injector injector = Guice.createInjector(new Module());
firstInjectClass = injector.getInstance(FirstInjectClass.class);
}
#Override
public void execute(Tuple tuple) {
count++;
String message = "Message count "+count;
firstInjectClass.printMessage(message);
log.error(message);
collector.emit("TO_SECOND_BOLT", new Values(message));
collector.ack(tuple);
}
#Override
public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
outputFieldsDeclarer.declareStream("TO_SECOND_BOLT", new Fields("MESSAGE"));
}
#Override
public Map<String, Object> getComponentConfiguration() {
Config conf = new Config();
conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 10);
return conf;
}
}
Second Bolt
public class SecondBolt extends BaseRichBolt {
private OutputCollector collector;
private SecondInjectClass secondInjectClass;
#Override
public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
collector = outputCollector;
Injector injector = Guice.createInjector(new Module());
secondInjectClass = injector.getInstance(SecondInjectClass.class);
}
#Override
public void execute(Tuple tuple) {
String message = (String) tuple.getValue(0);
secondInjectClass.printMessage(message);
log.error("SecondBolt {}",message);
collector.ack(tuple);
}
#Override
public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
}
}
Class in which dependencies are injected
public class FirstInjectClass {
FirstInterface firstInterface;
private final String prepend = "FirstInjectClass";
#Inject
public FirstInjectClass(FirstInterface firstInterface) {
this.firstInterface = firstInterface;
}
public void printMessage(String message){
log.error("{} {}", prepend, message);
firstInterface.printMethod(message);
}
}
Interface used for binding
public interface FirstInterface {
void printMethod(String message);
}
Implementation of interface
public class FirstInterfaceImpl implements FirstInterface{
private final String prepend = "FirstInterfaceImpl";
public void printMethod(String message){
log.error("{} {}", prepend, message);
}
}
Same way another class that receives dependency via DI
public class SecondInjectClass {
SecondInterface secondInterface;
private final String prepend = "SecondInjectClass";
#Inject
public SecondInjectClass(SecondInterface secondInterface) {
this.secondInterface = secondInterface;
}
public void printMessage(String message){
log.error("{} {}", prepend, message);
secondInterface.printMethod(message);
}
}
another interface for binding
public interface SecondInterface {
void printMethod(String message);
}
implementation of second interface
public class SecondInterfaceImpl implements SecondInterface{
private final String prepend = "SecondInterfaceImpl";
public void printMethod(String message){
log.error("{} {}", prepend, message);
}
}
Module Class
public class Module extends AbstractModule {
#Override
protected void configure() {
bind(FirstInterface.class).to(FirstInterfaceImpl.class);
bind(SecondInterface.class).to(SecondInterfaceImpl.class);
}
}
Nothing fancy here, just two bolts and couple of classes for DI. I deployed it on server and it works just fine. The catch/problem though is that I have to initialize Injector in each bolt which makes me question what is side effect of it going to be?
This implementation is simple, just 2 bolts.. what if I have more bolts? what impact it would create on topology if I have to initialize Injector in all bolts?
If I try to initialize Injector outside prepare method I get error for serialization.

Spring AMQP and messages in queue

In a Spring AMQP project, I would like to get the number of messages in a certain queue (to make decisions based on that number of messages) in RabbitMQ in real time (I can't use the management plugin).
The basic configuration is this:
#Bean(name="managementServerHandler")
public ManagementServerHandler managementServerHandler(){
return new ManagementServerHandler();
}
#Bean
public MessageListenerAdapter broadcastManagementServerHandler() {
return new MessageListenerAdapter(managementServerHandler(), "handleMessage");
}
#Bean(name="broadcastManagementMessageListenerContainer")
public SimpleMessageListenerContainer broadcastManagementMessageListenerContainer()
{
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(_connectionFactory());
container.setQueueNames( REQUEST_MANAGEMENT_QUEUE );
container.setMessageListener(broadcastManagementServerHandler());
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
container.setAutoDeclare(true);
container.setAutoStartup(true);
container.setConcurrentConsumers(1);
container.setRabbitAdmin((RabbitAdmin)_amqpAdmin());
container.setPrefetchCount(50);
container.setDeclarationRetries(3);
container.setMissingQueuesFatal(true);
container.setFailedDeclarationRetryInterval(1000);
container.setRecoveryInterval(400);
return container;
}
Where the "ManagementServerHandler" is just:
public class ManagementServerHandler implements ServletContextAware, MessageListener
{
#Override
public void onMessage(Message msg)
{....}
}
I need the number of queued messages in the onMessage method, but I can't find the way to do it.
I asked this question, but I don't know how to get the AMQP channel:
RabbitMQ and queue data
Thanks!
Use RabbitAdmin.getQueueProperties(queue)
/**
* Returns 3 properties {#link #QUEUE_NAME}, {#link #QUEUE_MESSAGE_COUNT},
* {#link #QUEUE_CONSUMER_COUNT}, or null if the queue doesn't exist.
*/
#Override
public Properties getQueueProperties(final String queueName) {

Very slow performance of spring-amqp comsumer

I've been experiencing troubles with spring-boot consumer. I compared the work of two consumers.
First consumer:
import com.rabbitmq.client.*;
import java.io.IOException;
public class Recv {
private final static String QUEUE_NAME = "hello";
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
Consumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope,
AMQP.BasicProperties properties, byte[] body) throws IOException {
}
};
channel.basicConsume(QUEUE_NAME, true, consumer);
}
}
Second consumer:
#Controller
public class Consumer {
#RabbitListener(queues = "hello")
public void processMessage(Message message) {
}
}
There are no config files for spring-boot consumer installed, everything goes by default.
On my computer first one works 10 times faster. What might be the problem?
The default prefetch (basicQos) for Spring AMQP consumers is 1 which means only 1 message is outstanding at the consumer at any one time; configure the rabbitListenerContainerFactory #Bean to set the prefetchCount to something larger.
You will have to override the default boot-configured #Bean.

How to add multiple Simplemessagelistenercontainer dynamically through application.properties

Below is our program, we create multiple containers for different queue through property in application.properties. But now it is static, when add another property, we must change the code.
I want add containers dynamically. I investigate several solutions.
1.use BeanFactory.registerSingleton method, but it cannot receive lifecycle callback,so i'm not sure the container can shutdown gracefully.
2.use BeanFactoryPostRegistor, but it need build a BeanDefinition, i have no idea how can construct a BeanDefinition for SimpleMessageListenerContainer, because it will be created by SimpleMessageListenrContainerFactory.
Can anybody give me better solution both add beans dynamically and the SimpleMessageListenerContainer can be started and shutdown normally?
#Bean
#ConditionalOnProperty(name = "pmc.multiple.hypervisor.reply.routerkey.kvm")
public SimpleMessageListenerContainer kvmReplyQueueConsumer() {
return getSimpleMessageListenerContainer(environment
.getProperty("pmc.multiple.hypervisor.reply.routerkey.kvm"));
}
#Bean
#ConditionalOnProperty(name = "pmc.multiple.hypervisor.reply.routerkey.vmware")
public SimpleMessageListenerContainer vmwareReplyQueueConsumer() {
return getSimpleMessageListenerContainer(environment
.getProperty("pmc.multiple.hypervisor.reply.routerkey.vmware"));
}
#Bean
#ConditionalOnProperty(name = "pmc.multiple.hypervisor.reply.routerkey.powervc")
public SimpleMessageListenerContainer powervcReplyQueueConsumer() {
return getSimpleMessageListenerContainer(environment
.getProperty("pmc.multiple.hypervisor.reply.routerkey.powervc"));
}
#Autowired
private SimpleRabbitListenerContainerFactory simpleRabbitListenerContainerFactory;
private SimpleMessageListenerContainer getSimpleMessageListenerContainer(String queueName){
return simpleRabbitListenerContainerFactory.createContainerInstance();
}
Take all properties you need (for example by regexp) and then register beans you want. There are 2 separate task there (1) how to get Spring properties (2) how register bean dynamically
1) To iterate over properties in 'Spring way':
#Autowired
Properties props;
....
for(Entry<Object, Object> e : props.entrySet()) {
if( /*some code to match*/ ){
//dispatch bean creation
}
}
2) you can either create beans dynamically by:
public MyClassPostRegister implements BeanFactoryPostProcessor {
public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) {
//create bean definition:
GenericBeanDefinition beanDefinition = new GenericBeanDefinition();
beanDefinition.setBeanClass(MyBeanClass.class);
beanDefinition.setLazyInit(false);
beanDefinition.setAbstract(false);
beanDefinition.setAutowireCandidate(true);
beanDefinition.setScope("prototype");
beanFactory.registerBeanDefinition("dynamicBean",beanDefinition);
}
Appendix after comment #GrapeBaBa:
Actually I use simpleRabbitListenerContainerFactory.createContainerInstance() to create container, so how to transform to use beanDefinition - please pay attention to lines marked with (!!!)
Create you own component
#Component
public class MyClassPostRegister implements BeanFactoryPostProcessor {
#Autowired
Properties props; //this gives you access to all properties
//following is example of filter by name
static final Pattern myInterestingProperties =
Pattern.compile("pmc\\.multiple\\.hypervisor\\.reply\\.routerkey\\..+");
Add post-process handler:
public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) {
//iterate through properties
for(Entry<Object, Object> e : props.entrySet()) {
Matcher m = myInterestingProperties.matcher(e.key);
if( !m.matches() )
continue;
//create bean definition:
GenericBeanDefinition beanDefinition = new GenericBeanDefinition();
beanDefinition.setBeanClass(SimpleMessageListenerContainer.class);
beanDefinition.setLazyInit(false);
beanDefinition.setAbstract(false);
beanDefinition.setAutowireCandidate(true);
beanDefinition.setScope("prototype");
//!!! Now specify name of factory method
beanDefinition.setFactoryMethodName("getSimpleMessageListenerContainer");
//!!! Now specify factory arguments:
ConstructorArgumentValues v = new ConstructorArgumentValues();
v.addGenericArgumentValue( e.getKey() ); //string
beanDefinition.getConstructorArgumentValues().add( v );
beanFactory.registerBeanDefinition("dynamicBean",beanDefinition);
}
}

Resources