I am using SpringIntegration's IntegrationFlows to define the message flow, and used Jms.messageDrivenChannelAdapter to get the message from the MQ, now I need to parse it, send it to KAFKA and update couchbase.
IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(this.acarsMqListener)) //MQ Listener with session transacted=true
.wireTap(ACARS_WIRE_TAP_CHNL) // Logging the message
.transform(agmTransformer, "parseXMLMessage") // Parse the xml message
.filter(acarsFilter,"filterMessageOnSmi") // Filter the message based on condition
.transform(agmTransformer, "populateImi") // Parse and Populate based on the message payload
.filter(acarsFilter,"filterMessageOnSmiImi") // Filter the message based on condition
.handle(acarsProcessor, "processEvent") // Create the message
.handle(Kafka.outboundChannelAdapter(kafkaTemplate).messageKey(MESSAGE_KEY).topic(acarsKafkaTopic)) //send it to kafka
.handle(updateCouchbase, "saveToDB") // Update couchbase
.get();
As per the application logic - the message should be stored in kafka and couchbase, if there is any exception in storing the message into kafka and couchbase the message should be rolled back to the queue. Is the above message flow cater the expected behavior? Can you please suggest if any improvement can be done?
Related
I want to send a message to all Participant nodes of my Helix cluster from Controller node. I tried following piece of code to send message to all registered participants of my cluster but their registered Participant message listeners are not receiving the message notification from Helix.
Message msg = new Message(factory.getMessageTypes().get(0), msgId);
msg.setMsgId(msgId);
msg.setSrcName(hostSrc);
msg.setTgtSessionId("*");
msg.setMsgState(MessageState.NEW);
msg.getRecord().setSimpleField("TestMessage", "Message from controller");
Criteria recipientCriteria = new Criteria();
recipientCriteria.setRecipientInstanceType(InstanceType.PARTICIPANT);
recipientCriteria.setInstanceName("%"); // To all recipients
recipientCriteria.setSessionSpecific(true); // To deliver only live participants
recipientCriteria.setSessionSpecific("DEV_CLUSTER"); // To only participants of this cluster
messagingService.send(recipientCriteria,msg);
Note that, when I am sending this message there is no resource exists in the cluster.
After debugging further what I have observed is CriteriaEvaluator.evaluateCriteria(....) operation is returning an empty list which further results into 0 messages to be sent to Participants nodes.
Kindly let me know if I am missing anything here while defining my criteria for Participants.
Thanks !
Update-1: our observation on this issue is as follows:
The received message at the participant side is read by both the Participant message listener(Say L1) and the handler created through MessageHandlerFactory(Which internally creating a listener HelixtaskExecutor (Say L2)).
In case if the message is read by HelixTaskExecutor(L2) first, it then immediately deletes the Znode in Zookeeper and the additionally configured message listener(L1) doesn't receive this message.
In case if the message is first read my additional message listener i.e. L1 then in such scenarios we don't face this problem as this additionally added listener doesn't delete the Znode from ZooKeeper.
We are still not sure how can we handle this problem as we want to use both the listener and MessageHandlers but facing the same problem I stated above.
Any inputs are appreciated.
I have published a message in the solace interface and got the messageId generated for that.
From SolAdmin, When I inspect the queue, I can able to see one new messages received, but the message id which generated is not same.
TextMessage txtMsg = jmsSession.createTextMessage();
messageID = txtMsg.getJMSMessageID();
The above messageID generated the output as
ID:2eaaf46d-b9ff-4aeb-a385-fbc2e6cced0a:1:1:1-1
But in SolAdmin, the message shows as 5985824677
The "Message ID" that is shown in the endpoints tab of SolAdmin is internal to the Solace Message Broker and is not equivalent to the "JMS Message ID".
You can use it for operations such as deleting some messages via the CLI or SEMP.
For example:
solace(admin/message-spool)# delete-messages queue my_sample_queue message 123456789 to 123456790
There's no way to display the JMS Message ID in SolAdmin.
Instead, you will need to make use of a queue browser to browse messages in the queue.
This can be a custom application that you write, sdkperf (use the -qb and -md flags), or a third party graphical JMS queue browser such as HermesJMS.
I am looking to retrieve some Solace queue stats e.g. the current messages spooled count out of the maximum limit for us to set a threshold to stop publishing more messages to the queue.
Also, to subscribe to vpn events to track message discard rates.
By the time we receive errors e.g. MaxMsgUsageExceeded/SpoolOverQuota, it will be too late.
I can't seem to find any of these on SolaceSystems.Solclient.Messaging API
https://docs.solace.com/API-Developer-Online-Ref-Documentation/net/html/7f10bcf6-19f4-beff-0768-ced843e35168.htm
Would be great if someone could help
(using C# for this)
To poll for Solace queue stats from your C# application, you could use legacy SEMP over the message bus to make a SEMP request for the details that you want. Semp (Solace Element Management Protocol) is a request/reply protocol that uses an XML schema to identify all managed objects available in a message broker. Applications can use SEMP to manage and monitor a message broker.
To allow for legacy SEMP to be used over the message bus, as opposed to the management interface, it first needs to be enabled on the Solace PubSub+ message broker at the VPN level.
To publish a SEMP request with the Solace .Net Messaging API, perform the following steps:
Create a Session.
Create the message topic. “#SEMP//SHOW”
ITopic topic = ContextFactory.Instance.CreateTopic( “#SEMP/<router name>/SHOW”);
Create a request message and set its Destination to the topic in Step 2:
IMessage requestMsg = ContextFactory.Instance.CreateMessage();
requestMsg.Destination = topic;
Set the SEMP request string as the binary attachment.
string SOLTR_VERSION = "8_4_0" //change to the message-broker's version
string SEMP_SHOW_QUEUE = "<rpc semp-version=\"soltr/" + SOLTR_VERSION +
"<show><queue><name>queueName</name><detail></detail></queue></show></rpc>";
requestMsg.BinaryAttachment = Encoding.UTF8.GetBytes(SEMP_SHOW_QUEUE);
Call the SendRequest(…) method on Session.
IMessage replyMsg;
ReturnCode rc = session.SendRequest(requestMsg, out replyMsg, timeout);
The SEMP response is returned in replyMsg.
Obtain the binary attachment data from the reply message:
replyMsg.BinaryAttachment
The binary attachment contains the SEMP reply for the command topic in the publish request.
The Solace PubSub+ message broker does raise an event when an egress message is discarded. However, it is only sent out approximately once every 60 seconds for the specified client so it is not possible to get these exact rates.
It is possible for your .NET application to subscribe to VPN-level events over the message-bus. To do this, you must first enable the Solace PubSub+ message broker to publish the events. You can then subscribe to the special topic and receive the events as messages.
The topic to subscribe to is:
#LOG/<level>/VPN/<routerName>/<eventName>/<vpnName>
The different levels can use the * wildcard. For example, if you wish to subscribe to all VPN events of all levels for the VPN apple on router QA-NY1, the topic string would be:
#LOG/*/VPN/QA-NY1/*/apple
SEMP (starting in v2) is a RESTful API for configuring, monitoring, and administering a Solace PubSub+ broker.
1-The swapper page link is SEMP V2 API
2-The Swagger metadata definitions URL is located # http://{solace-sever-url}/SEMP/v2/config/spec
3- From Visual studio, add REST API Client
4-In the configuration dialog pass swagger metadata URL (defined at step 2), for code purpose I choose SolaceSemp as input value parameter for client namespace input.
4 Once you click ok, VS will create the client along with the models under SolaceSemp namespace
5 Start using the client as per following
using SolaceSemp;
using Microsoft.Rest;
var credentials = new BasicAuthenticationCredentials();
credentials.UserName = "place user name";
credentials.Password = "place password";
using (var client = new SolaceSempClient(credentials))
{
var model = client.GetAboutApi();
}
In .NET Framework Microsoft.ServiceBus.Messaging had a class used to receive messages from Service Bus, BrokeredMessage. However, in .NET Standard 2.0, in order to receive messages from a Service Bus, class Message is used, from Microsoft.Azure.ServiceBus.Core.
BrokeredMessage has a method, CompleteAsync(), used to complete the receive operation of a message and indicates that the message should be marked as processed and deleted. I can't find a method for the Message class that does the same thing. Do you guys know any solution in order to mark a message as processed and deleted for the Message class?
To complete the message in the Queue using Microsoft.Azure.ServiceBus.Core, there is a method CompleteAsync available in the QueueClient, through which the messages will be received.
The lock token of the message should be passed as the parameter for the CompleteAsync method.
Example: queueClient.CompleteAsync(message.SystemProperties.LockToken)
How do I break/split the message received from mqtt broker(mosca)? The whole message come with packet, topic, messageid, payload, etc. I just need the payload {"T":"t"} displayed at the debug node. I tried the split and switch node, it doesn't seem to work, no response at the output.
mqtt device
mqtt broker
You should probably be using the MQTT-in node to subscribe to the topics you want rather than the output of the Mosca broker node, which will include EVERY message sent to the broker (with all the internal detail that you don't want.
But you can move the msg.packet.payload to msg.payload with the change node. Then run that output through the JSON node which will parse the String representation of the JSON object back into a proper object.
(If you use the MQTT-in node you will still need to use the JSON node)