I have one SQS queue and one listener method running in the task of ECS Fargate service. What I needed is when one message processed successfully, I want to trigger cloud watch alarm ? I could not be able to find any SQS metric for that.
#SqsListener(value = "StageQueue",deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void listen(String message) {
log.info("!!!! received message {} {}", message.toString());
String message = doSomeProcessing();
// I want to trigger cloudwatch alarm here, that this message processed successfully
}
Related
I have a question / problem about Persistent Session and Queuing Messages.
Here is the scenario:
I have a publisher (java server) which is publish message and I have a receiver (android client). When android client it online it gets the messages which amazing, working very well.
However, when I kill the android app and keep sending message from server and when I open android app, android does not receive previous messages.
Server side:
final Mqtt5BlockingClient client = MqttClient.builder()
.useMqttVersion5()
.serverHost(host)
.serverPort(8883)
.sslWithDefaultConfig()
.buildBlocking();
// connect to HiveMQ Cloud with TLS and username/pw
client.connectWith()
.simpleAuth()
.username(username)
.password(UTF_8.encode(password))
.applySimpleAuth()
.noSessionExpiry()
.cleanStart(false)
.send();
// This code is running every 15 sec
String now = LocalDateTime.now().toString();
String message = String.format("Hello: %s", now);
// publish a message to the topic "my/test/topic"
client.publishWith()
.topic("hasan-device/sayHello")
.payload(UTF_8.encode(message))
.retain(true)
.qos(MqttQos.AT_LEAST_ONCE)
.noMessageExpiry()
.send();
Client side:
// create an MQTT client
final Mqtt5BlockingClient client = MqttClient.builder()
.identifier("my-device-1")
.useMqttVersion5()
.serverHost(host)
.serverPort(8883)
.sslWithDefaultConfig()
.automaticReconnectWithDefaultConfig()
.buildBlocking();
// connect to HiveMQ Cloud with TLS and username/pw
client.connectWith()
.simpleAuth()
.username(username)
.password(UTF_8.encode(password))
.applySimpleAuth()
.noSessionExpiry()
.cleanStart(false)
.send();
// subscribe to the topic "my/test/topic"
client.subscribeWith()
.topicFilter("hasan-device/sayHello")
.retainHandling(Mqtt5RetainHandling.SEND)
.send();
// set a callback that is called when a message is received (using the async API style)
client.toAsync().publishes(ALL, publish -> {
byte[] message = publish.getPayloadAsBytes();
LOGGER.info("Received message: {} -> {}, ", publish.getTopic(), new String(message, UTF_8));
});
Expecting to message arrive when device back to online
When the Android app restarts with the persistent session, brokers will send down pending messages immediately. This can happen before the application callbacks get initialised.
Here is an example from when I did some testing with this:
To fix, move this bit of code to execute just before the connectWith call:
// set a callback that is called when a message is received (using the async API style)
client.toAsync().publishes(ALL, publish -> {
byte[] message = publish.getPayloadAsBytes();
LOGGER.info("Received message: {} -> {}, ", publish.getTopic(), new String(message, UTF_8));
});
I am trying to apply subscription filter policy for SQS Queue.
1st condition
{
"insurance_type":"car",
"event_type":["created","updated"]
}
2nd condition
{
"event_type:["deleted"]
}
I want messages coming to the SNS topic should filter according to these policies and comes to SQS queue.
for ex:-
{
"insurance_type":"car",
"event_type":"created"
}
and
{
"event_type":"deleted"
}
both should end up in the same queue.
How can I achieve this?
With a MQTT shared subscription, the message on the subscirbed topic would only be sent to one of the subscribing clients. Then, how the other clients of the group receive the message as they also subscribe to the same topic?
With a MQTT shared subscription, the message on the subscribed topic would only be sent to one of the subscribing clients.
Correct. With a normal (non‑shared) subscription the messages are sent to ALL subscribers; with shared subscriptions they are sent to ONE subscriber.
Prior to the introduction of shared subscriptions it was difficult to handle situations where you want multiple servers (for fault tolerance, load balancing etc) but only want to process each message once. Shared subscriptions provide a simple way of accomplishing this.
Then, how the other clients of the group receive the message as they also subscribe to the same topic?
If they are subscribing to the same shared subscription (with the same ShareName) then only one will receive the message; this is by design. If you want them all to receive the message then don't use a shared subscription. Alternatively you can establish multiple subscriptions (so all subscribers receive the message but only one processes it - but note the "could") as per the spec:
If a Client has a Shared Subscription and a Non‑shared Subscription and a message matches both of them, the Client will receive a copy of the message by virtue of it having the Non‑shared Subscription. A second copy of the message will be delivered to one of the subscribers to the Shared Subscription, and this could result in a second copy being sent to this Client.
There is an interesting bug in Java Paho (1.2.5) client that prevents working with shared topics that contains wildcards (#, +) https://github.com/eclipse/paho.mqtt.java/issues/827
Long story short, this will not work:
mqttClient.subscribe("$shared/group/some_topic/#", 1, (topic, message) -> System.out.println(topic));
instead it's required to use callbacks:
mqttClient.subscribe("$shared/group/some_topic/#", 1);
mqttClient.setCallback(new MqttCallback() {
#Override
public void connectionLost(final Throwable cause) {
}
#Override
public void messageArrived(final String topic, final MqttMessage message) throws Exception {
System.out.println(topic);
}
#Override
public void deliveryComplete(final IMqttDeliveryToken token) {
}
});
I am trying to create a subscriber for my durable topic endpoint in solace via .NET APIs.
I have the below method where in I am trying to create a flow for my durable topic endpoint. I don't understand what is the need for ISubscription instance in the Session.CreateFlow method. (https://docs.solace.com/API-Developer-Online-Ref-Documentation/net/html/a548a98a-9134-c167-2517-192a26ceed77.htm)
How do I create an instance of ISubscription and what should it be?
public void Start()
{
//Create a instance of a durable topic endpoint
topic = ContextFactory.Instance.CreateDurableTopicEndpointEx(topicName);
FlowProperties flowProps = new FlowProperties();
flowProps.FlowStartState = false;
if (selector != null)
{
flowProps.Selector = selector;
}
flowProps.AckMode = ackMode == AckMode.ClientAcknowledge ? MessageAckMode.ClientAck : MessageAckMode.AutoAck;
if (windowSize.HasValue)
{
flowProps.WindowSize = windowSize.Value;
}
flow = session.CreateFlow(flowProps, topic, null, HandleFlowMessageEvent, HandleFlowEvent);
flow.Start();
}
I am currently passing it as null and I get the error
subscription must be non-null when endpoint is of type ITopicEndpoint
Secondly, I have a message handler for my flow event and session event, so when a message comes through which handler would it invoke. I would expect that I handle the message in the FlowMessageHandler as I am connecting to a durable topic endpoint. Please can someone shed more light on this?
//session message event handler
session = context.CreateSession(sessionProperties, HandleSessionMessageEvent, HandleSessionEvent);
//Flow message event handler
flow = session.CreateFlow(flowProps, topic, null, HandleFlowMessageEvent, HandleFlowEvent);
For a durable topic endpoint, if you have multiple subscribers then each would have to register with a unique subscription name (you can give any name as you like) so that the broker can maintain a durable connection with the specific subscriber. Even if the subscriber is down for a while and reconnects, it will not lose out on messages in the interim. The broker will push the messages to the subscriber. You can create a subscriber via the Solace Admin client or via code. Solace Admin -> Select the VPN -> Choose Endpoint tab -> select Durable Topic Endpoint -> click the + sign to create a subscriber -> ensure that consume is chosen for 'All other Permissions'.
The messages are sent to the session handler first and then to the flow handler. So you can access the messages from both the handlers.
My application only listens to a certain queue, the producer is the 3rd party application. I receive the messages but sometimes based on some logic I need to send fail message to the producer so that the message is resend to my listener again until I decide to consume it and acknowledge it. My current implementation of this process is just throwing some custom exception. But this is not a clean solution, therefore can any one help me to send FAIL to producer without throwing exception.
My JMS Listener Factory settings:
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactoryForQexpress(SQSErrorHandler errorHandler) {
SQSConnectionFactory connectionFactory = SQSConnectionFactory.builder()
.withRegion(RegionUtils.getRegion(StaticSystemConstants.getQexpressSqsRegion()))
.withAWSCredentialsProvider(new ClasspathPropertiesFileCredentialsProvider(StaticSystemConstants.getQexpressSqsCredentials()))
.build();
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setDestinationResolver(new DynamicDestinationResolver());
factory.setConcurrency("3-10");
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
factory.setErrorHandler(errorHandler);
return factory;
}
My Listener Settings:
#JmsListener(destination = StaticSystemConstants.QUEXPRESS_ORDER_STATUS_QUEUE, containerFactory = "jmsListenerContainerFactoryForQexpress")
public void receiveQExpressOrderStatusQueue(String text) throws JSONException {
LOG.debug("Consumed QExpress status {}", text);
//here i need to decide either acknowlege or fail
...
if (success) {
updateStatus();
} else {
//todo I need to replace this with explicit FAIL message
throw new CustomException("Not right time to update status");
}
}
Please, share your experience on this. Thank you!
SQS -- internally speaking -- is fully asynchronous and completely decouples the producer from the consumer.
Once the producer successfully hands off a message to SQS and receives the message-id in response, the producer only knows that SQS has received and committed the message to its internal storage and that the message will be delivered to a consumer at least once.¹ There is no further feedback to the producer.
A consumer can "snooze" a message for later retry by simply not deleting it (see setSessionAcknowledgeMode docs) or by actively resetting the visibility timeout on the message instead of deleting it, which triggers SQS to leave the message in the in flight status until the timer expires, at which point it will again deliver the message for the consumer to retry.
Note, too, that a single SQS queue can have multiple producers and/or multiple consumers, as long as all the producers ask for and consumers provide identical services, but there is no intrinsic concept of which consumer or which producer. There is no consumer-to-producer backwards communication channel, and no mechanism for a producer to inquire about the status of an earlier message -- the design assumption is that once SQS has received a message, it will be delivered,² so no such mechanism should be needed.
¹at least once. Unless the queue is a FIFO queue, SQS will typically deliver the message exactly once, but there is not an absolute guarantee that the message will not be delivered more than once. Because SQS is a massive, distributed system that stores redundant copies of messages, it is possible in some edge case conditions for messages to be delivered more than once. FIFO queues avoid this possibility by leveraging stronger internal consistency guarantees, at a cost of reduced throughput of 300 TPS.
²it will be delivered assuming of course that you actually have a consumer running. SQS does not block the producer, and will allow you to enqueue an unbounded number of messages waiting for a consumer to arrive. It accepts messages from producers regardless of whether there are currently any consumers listening. The messages are held until consumed or until the MessageRetentionPeriod (default 4 days, max 14 days) timer expires for each message, whichever comes first.