I am trying to trace Spring cloud integration aws + AWS SQS app with Sleuth. The receiver receives message from SQS after a message is added to queue. The log has app name, but no trace id and span id while receiving message from sql queue. Here is a line from the log:
2017-07-28 16:24:02.352 INFO [sqs-sleuth-demo,,,] 9706 --- [enerContainer-2] com.example.demo.SQSMessageReceiver : dequeued message: hello world
I use Spring Boot '1.5.4.RELEASE' and Spring Cloud 'Dalston.SR1'. Here is the dependencies:
dependencies {
compile("org.springframework.boot:spring-boot")
compile("org.springframework.boot:spring-boot-starter")
compile("org.springframework.boot:spring-boot-starter-web")
compile("org.springframework.cloud:spring-cloud-aws-messaging")
compile("org.springframework.cloud:spring-cloud-aws-autoconfigure")
compile('org.springframework.cloud:spring-cloud-starter-sleuth')
compile("org.springframework.integration:spring-integration-aws:1.0.0.RELEASE")
compile("com.amazonaws:aws-java-sdk-sqs")
}
dependencyManagement {
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
}
}
AppConfig.java
#Configuration
public class AppConfig {
#Value("${amazon.aws.accesskey}")
private String amazonAWSAccessKey;
#Value("${amazon.aws.secretkey}")
private String amazonAWSSecretKey;
#Value("${amazon.sqs.endpoint}")
private String amazonSqsEndpoint;
#Value("${cloud.aws.region.static}")
private String awsRegion;
#Bean
#Primary
public AWSCredentialsProviderChain credentialsProviderChain() {
return new DefaultAWSCredentialsProviderChain();
}
}
SQSMessageReceiver.java
#Component
public class SQSMessageReceiver {
private static final Logger LOGGER = LoggerFactory.getLogger(SQSMessageReceiver.class);
#Autowired
private RestTemplate restTemplate;
#SqsListener(value="${amazon.sqs.queue.name}", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void receive(String message) throws Exception {
LOGGER.info("dequeued message: " + message);
}
}
and, DemoApplication.java
#SpringBootApplication
#EnableSqs
public class DemoApplication {
private static final Logger LOGGER = LoggerFactory.getLogger(DemoApplication.class);
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
Is it possible to trace SQS events with Sleuth or anything wrong with the setting/code?
Thanks,
Related
I am using spring-amqp, and using consumerBatchEnabled to receive batch of events as mentioned in below link:
https://docs.spring.io/spring-amqp/docs/2.2.5.RELEASE/reference/html/#receiving-batch
and registering the listener as below:
import org.springframework.messaging.Message;
#RabbitListener(queues = "batch.2", containerFactory = "consumerBatchContainerFactory")
public void consumerBatch2(List<org.springframework.messaging.Message<Invoice>> messages) {
//code here to process events
}
Also have config class defined as below
#Bean
#Autowired
public ConnectionFactory connectionFactory() {
CachingConnectionFactory factory = new CachingConnectionFactory();
return factory;
}
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory){);
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(messageConverter());
return rabbitTemplate;
}
#Bean
public SimpleRabbitListenerContainerFactory consumerBatchContainerFactory(
SimpleRabbitListenerContainerFactoryConfigurer configurer,
ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
factory.setPrefetchCount(1000);
factory.setBatchListener(true);
factory.setBatchSize(1000);
factory.setConsumerBatchEnabled(true);
factory.setReceiveTimeout(1000l);
configurer.configure(factory, connectionFactory);
return factory;
}
#Bean
public MessageConverter messageConverter(){
return new Jackson2JsonMessageConverter();
}
however when I publish the event, i get ClassCastException, java.util.ArrayList cannot be cast to org.springframework.amqp.core.Message:
stack trace
Execution of Rabbit message listener failed.","logger_name":"org.springframework.amqp.rabbit.listener.ConditionalRejectingErrorHandler","thread_name":"org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1","level":"WARN","level_value":30000,"stack_trace":"java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.springframework.amqp.core.Message\n\tat brave.spring.rabbit.TracingRabbitListenerAdvice.invoke(TracingRabbitListenerAdvice.java:75)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)\n\tat org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)\n\tat org.springframework.amqp.rabbit.listener.$Proxy210.invokeListener(Unknown Source)\n\tat org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1537)\n\tat org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1532)\n\tat org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1472)\n\tat org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.executeWithList(SimpleMessageListenerContainer.java:1037)\n\tat org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1026)\n\tat org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:923)\n\tat org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1600(SimpleMessageListenerContainer.java:83)\n\tat org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.mainLoop(SimpleMessageListenerContainer.java:1298)\n\tat org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1204)\n\tat java.lang.Thread.run(Thread.java:748)\n"}
{"timestamp":"2021-06-10T13:47:56.851+0000","message":"Restarting Consumer
There is no issue without consumerBatchEnabled, and able to receive and process Message
#RabbitListener(queues = "batch.2")
public void consumerBatch2(org.springframework.messaging.Message<Invoice> message) {
//code here
}
its a known issue https://github.com/openzipkin/brave/issues/1240.
Bug is resolved in library version 5.12.4
I have multiple virtual hosts each with a request queue and a response queue. These virtual hosts serve different clients. The names for the request queue and the response queue remain the same across the virtual hosts.
I have created a SimpleRoutingConnectionFactory with the clientName()+"ConnectionFactory" as the lookup key and a corresponding CachingConnectionFactory as the value in the map. I'm able to publish message to the request queues by binding and the RabbitTemplate to a virtual host before convertAndSend and then unbinding it.
I'm not able to consume messages from the response queues from different virtual hosts. I have created a SimpleRabbitListenerContainerFactory for each client. I implemented RabbitListenerConfigurer and registered a SimpleRabbitListenerEndpoint for each SimpleRabbitListenerContainerFactory. I also set the connectionFactory on each SimpleRabbitListenerContainerFactory as the client's CachingConnectionFactory.
#Configuration
public class RabbitConfiguration implements RabbitListenerConfigurer {
#Autowired
private ApplicationContext applicationContext;
#Autowired
private ClientList clients;
#Bean
#Primary
public SimpleRoutingConnectionFactory routingConnectionFactory() {
final var routingConnectionFactory = new SimpleRoutingConnectionFactory();
final Map<Object, ConnectionFactory> routeMap = new HashMap<>();
applicationContext.getBeansOfType(ConnectionFactory.class)
.forEach((beanName, bean) -> {
routeMap.put(beanName, bean);
});
routingConnectionFactory.setTargetConnectionFactories(routeMap);
return routingConnectionFactory;
}
#Bean
public RabbitTemplate rabbitTemplate() {
return new RabbitTemplate(routingConnectionFactory());
}
#Bean
public DirectExchange orbitExchange() {
return new DirectExchange("orbit-exchange");
}
#Bean
public Queue requestQueue() {
return QueueBuilder
.durable("request-queue")
.lazy()
.build();
}
#Bean
public Queue responseQueue() {
return QueueBuilder
.durable("response-queue")
.lazy()
.build();
}
#Bean
public Binding requestBinding() {
return BindingBuilder.bind(requestQueue())
.to(orbitExchange())
.with("orbit-request");
}
#Bean
public Binding responseBinding() {
return BindingBuilder.bind(responseQueue())
.to(orbitExchange())
.with("orbit-response");
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
clients.get()
.stream()
.forEach(client -> {
var endpoint = createEndpoint(client);
var listenerContainerFactory = applicationContext.getBean(client.getName() + "ListenerContainerFactory");
listenerContainerFactory.setConnectionFactory((ConnectionFactory)applicationContext.getBean(client.getName() + "ConnectionFactory"));
registrar.registerEndpoint(endpoint, listenerContainerFactory);
});
}
}
private SimpleRabbitListenerEndpoint createEndpoint(Client client) {
var endpoint = new SimpleRabbitListenerEndpoint();
endpoint.setId(client.getName());
endpoint.setQueueNames("response-queue");
endpoint.setMessageListener(new MessageListenerAdapter(new MessageReceiver(), "receive"));
return endpoint;
}
}
However, I get org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer: Failed to check/redeclare auto-delete queue(s). java.lang.IllegalStateException: Cannot determine target ConnectionFactory for lookup key [null]
I'm not able to figure out whats causing this as I'm not using the SimpleRoutingConnectionFactory for message consumption at all.
EDIT:
Full stack trace below -
ERROR [2020-07-09T04:12:38,028] [amdoListenerEndpoint-1] [TraceId:] org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer: Failed to check/redeclare auto-delete queue(s).
java.lang.IllegalStateException: Cannot determine target ConnectionFactory for lookup key [null]
at org.springframework.amqp.rabbit.connection.AbstractRoutingConnectionFactory.determineTargetConnectionFactory(AbstractRoutingConnectionFactory.java:120)
at org.springframework.amqp.rabbit.connection.AbstractRoutingConnectionFactory.createConnection(AbstractRoutingConnectionFactory.java:98)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.createConnection(ConnectionFactoryUtils.java:214)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2089)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2062)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2042)
at org.springframework.amqp.rabbit.core.RabbitAdmin.getQueueInfo(RabbitAdmin.java:407)
at org.springframework.amqp.rabbit.core.RabbitAdmin.getQueueProperties(RabbitAdmin.java:391)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.attemptDeclarations(AbstractMessageListenerContainer.java:1836)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.redeclareElementsIfNecessary(AbstractMessageListenerContainer.java:1817)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.initialize(SimpleMessageListenerContainer.java:1349)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1195)
at java.base/java.lang.Thread.run(Thread.java:834)
EDIT2:
I used the routingConnectionFactory with every listener and used the setLookUpKeyQualifier. No more exceptions but, the listeners don't seem to be doing anything i.e., the queues are not being listened to.
#Import(MqConfig.class)
//This is to import CachingConnectinFactory beans and SimpleRabbitListenerContainerFactory beans for all clients
#Configuration
public class RabbitConfiguration implements RabbitListenerConfigurer {
#Autowired
private ApplicationContext applicationContext;
#Autowired
private ClientList clients;
#Bean
#Primary
public SimpleRoutingConnectionFactory routingConnectionFactory() {
final var routingConnectionFactory = new SimpleRoutingConnectionFactory();
final Map<Object, ConnectionFactory> routeMap = new HashMap<>();
applicationContext.getBeansOfType(ConnectionFactory.class)
.forEach((beanName, bean) -> {
routeMap.put(beanName+"[response-queue]", bean);
});
routingConnectionFactory.setTargetConnectionFactories(routeMap);
return routingConnectionFactory;
}
#Bean
public RabbitTemplate rabbitTemplate() {
return new RabbitTemplate(routingConnectionFactory());
}
#Bean
public DirectExchange orbitExchange() {
return new DirectExchange("orbit-exchange");
}
#Bean
public Queue requestQueue() {
return QueueBuilder
.durable("request-queue")
.lazy()
.build();
}
#Bean
public Queue responseQueue() {
return QueueBuilder
.durable("response-queue")
.lazy()
.build();
}
#Bean
public Binding requestBinding() {
return BindingBuilder.bind(requestQueue())
.to(orbitExchange())
.with("orbit-request");
}
#Bean
public Binding responseBinding() {
return BindingBuilder.bind(responseQueue())
.to(orbitExchange())
.with("orbit-response");
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
clients.get()
.stream()
.forEach(client -> {
var endpoint = createEndpoint(client);
var listenerContainerFactory = getListenerContainerFactory(Client client);
listenerContainerFactory.setConnectionFactory((ConnectionFactory)applicationContext.getBean(client.getName() + "ConnectionFactory"));
registrar.registerEndpoint(endpoint, listenerContainerFactory);
});
}
}
private SimpleRabbitListenerEndpoint createEndpoint(Client client) {
var endpoint = new SimpleRabbitListenerEndpoint();
endpoint.setId(client.getName());
endpoint.setQueueNames("response-queue");
endpoint.setMessageListener(new MessageListenerAdapter(new MessageReceiver(), "receive"));
return endpoint;
}
private SimpleRabbitListenerContainerFactory getListenerContainerFactory(Client client) {
var listenerContainerFactory = (SimpleRabbitListenerContainerFactory) applicationContext.getBean(client.getName() + "ListenerContainerFactory");
listenerContainerFactory.setConnectionFactory(routingConnectionFactory());
listenerContainerFactory.setContainerCustomizer(container -> {
container.setQueueNames("response-queue");
container.setLookupKeyQualifier(client.getName());
container.setMessageListener(message -> log.info("Received message"));
});
return listenerContainerFactory;
}
}
There is something very strange going on; [null] implies that when we call getRoutingLookupKey() the cf is not a routing cf but when we call getConnectionFactory() it is.
It's not obvious how that can happen. Perhaps you can figure out why in a debugger?
One solution would be to inject the routing cf and use setLookupKeyQualifier(...).
The lookup key will then be clientId[queueName].
When I create Task in Spring Cloud Dataflow and edit properties in Spring Cloud Dataflow Dashboard I only see standard properties label despite being configured
ConfigurationProperties. And I do not know what I've set up wrongly. Below the code.
JobProps:
#Component
#ConfigurationProperties("job")
public class JobProps {
private String ux;
//getter and setter
}
JobDoing:
#Component
public class JobDoing {
public JobDoing() {
doing();
}
#Value("${job.ux:}")
private String test;
private static final Log logger = LogFactory.getLog(JobConfiguration.class);
public void doing(){
logger.info("Props: " + test);
}
}
DemoApplication:
#EnableConfigurationProperties({JobProps.class })
#EnableTask
#SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
You need to whitelist your application's custom configuration properties so that they get picked up by the Spring Cloud Data Flow server when extracting and displaying the application configuration properties.
To whitelist the configuration properties, you can refer this documentation.
I'm currently trying to understand why my ConfirmCallback is called before I invoked Channel.basicAck / Channel.basicNack on the ChannelAwareMessageListener.
Please find below my current setup
#Component
public class MyMessageListener implements ChannelAwareMessageListener {
private Logger LOGGER = LoggerFactory.getLogger(MyMessageListener.class);
#Override
public void onMessage(Message message, Channel channel) throws Exception {
Thread.sleep(1000L);
String arg = String.valueOf(message.getBody());
LOGGER.info("Received message {}", arg);
Thread.sleep(1000L);
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
}
}
#Component
public class LoggingConfirmCallback implements RabbitTemplate.ConfirmCallback{
private Logger LOGGER = LoggerFactory.getLogger(LoggingConfirmCallback.class);
#Override
public void confirm(CorrelationData correlationData, boolean ack, String cause) {
LOGGER.info("Received confirm with result {}", ack);
}
}
#SpringBootApplication
#Configuration
public class Application {
#Autowired
RabbitTemplate rabbitTemplate;
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(Application.class, args);
context.getBean(Application.class).doIt();
}
public void doIt() {
rabbitTemplate.convertAndSend(null, null, "hello", new CorrelationData(UUID.randomUUID().toString()));
}
#Bean
#Qualifier("confirmConnectionFactory")
ConnectionFactory confirmConnectionFactory() {
CachingConnectionFactory factory = new CachingConnectionFactory();
factory.setPublisherConfirms(true);
factory.setHost("192.168.59.103");
factory.setChannelCacheSize(5);
return factory;
}
#Bean
#Primary
RabbitTemplate firstExchange(#Qualifier("confirmConnectionFactory") ConnectionFactory connectionFactory, LoggingConfirmCallback loggingConfirmCallback) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setConfirmCallback(loggingConfirmCallback);
rabbitTemplate.setExchange("first");
rabbitTemplate.setMandatory(true);
return rabbitTemplate;
}
#Bean
MessageListenerAdapter myMessageListenerAdapter(MyMessageListener receiver) {
return new MessageListenerAdapter(receiver);
}
#Bean
SimpleMessageListenerContainer myQueueListener(#Qualifier("confirmConnectionFactory")ConnectionFactory connectionFactory, MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setAcknowledgeMode(AcknowledgeMode.MANUAL);
container.setQueueNames("first.queue");
container.setMessageListener(listenerAdapter);
return container;
}
What I see in the logging is this
2015-07-17 08:56:32.352 INFO 5892 --- [ main] com.coderskitchen.rmqdct.Application : Started Application in 1.393 seconds (JVM running for 1.704)
2015-07-17 08:56:32.372 INFO 5892 --- [168.59.103:5672] c.c.rmqdct.LoggingConfirmCallback : Received confirm with result true
2015-07-17 08:56:33.373 INFO 5892 --- [cTaskExecutor-1] c.c.rmqdct.MyMessageListener : Received message [B#67962299
But I expected to the the message
Received confirm with result true
after
Received message [B#67962299
Thanks in advance
Peter
Publisher confirms have nothing to do with message reception.
The broker confirms that it has taken responsibility for the message by successfully delivering it to the configured queue(s).
It is quite independent of the consumer acknowledging reception. If you need that, you will have to send an application-level message back to the producer.
I'm trying to write a "Hello, World" with Spring Data Neo4j in a standalone app. It runs and actually creates the Neo4j database, but my #Autowired repo is not being initialized. I suspect the problem is in my main class, but I don't know what to try. Unsurprisingly, almost all the Spring tutorials I've found are about web apps.
What am I doing wrong?
config bean:
#Configuration
#EnableNeo4jRepositories(basePackages = "test2")
public class ConfigBean extends Neo4jConfiguration {
private static final String DB_PATH = "/home/kevin/tmp/hello-spring-data-neo4j/";
public ConfigBean() {
setBasePackage("test2");
}
#Bean
public GraphDatabaseService graphDatabaseService() {
return new GraphDatabaseFactory().newEmbeddedDatabase(DB_PATH);
}
}
node entity:
#NodeEntity
public class Foo {
#GraphId
private Long id;
}
repository:
public interface FooRepository extends GraphRepository<Foo> { }
main class:
#Component
public class Test2 {
#Autowired
FooRepository repo;
public void doStuff() {
System.out.println("repo: " + repo); // null!
}
public static void main(String[] args) {
ApplicationContext context = new AnnotationConfigApplicationContext("test2");
new Test2().doStuff();
}
}
It logs about 350 lines of output. These are the last few lines. I searched for this error message, but the impression I got is that it's unrelated to my problem.
20:44:30.630 [main] DEBUG o.s.c.e.PropertySourcesPropertyResolver - Searching for key 'spring.liveBeansView.mbeanDomain' in [systemProperties]
20:44:30.631 [main] DEBUG o.s.c.e.PropertySourcesPropertyResolver - Searching for key 'spring.liveBeansView.mbeanDomain' in [systemEnvironment]
20:44:30.635 [main] DEBUG o.s.c.e.PropertySourcesPropertyResolver - Could not find key 'spring.liveBeansView.mbeanDomain' in any property source. Returning [null]
repo: null
Via the magical "ask a question, find the answer" effect, my main class now looks like this, and the repo is being assigned:
#Component
public class Test2 {
#Autowired
FooRepository repo;
public void doStuff() {
System.out.println("repo: " + repo);
}
public static void main(String[] args) {
ApplicationContext context = new AnnotationConfigApplicationContext("test2");
Test2 test2 = context.getBean(Test2.class);
test2.doStuff();
}
}