I'm trying to create a Flux from incoming messages received from a queue.
For instance, if I'm using Amazon SQS how do I achieve to write the following code:
Flux<String> messages = connectionToSQS.receiveFromQueue(queueName);
messages.map(s -> log.info("message: {}", s).subscribe();
After experimentation, I found the following issues:
How do I keep requesting messages from the queue (loop forever)? Do I create one thread that has a loop that keeps on requesting from the queue?
How do I make the Flux cold? I don't want to request messages from SQS unless the consumer asks for it. This allows me to use backpressure.
First pass over this problem yielded something like the following code as per Reactor documentation:
Flux<String> bridge = Flux.create(sink -> {
myEventProcessor.register(new MyEventListener<String>() {
public void onDataChunk(List<String> chunk) {
for(String s : chunk) {
sink.next(s);
}
}
public void processComplete() {
sink.complete();
}
});
});
The idea being to create a single thread that keeps on requesting for messages in a loop and then using an observer pattern like above to do a next() on each message received.
Related
I have integration test with docker using test containers. On container I run jms. In test I am putting message on queue.
How I can wait in test to make it populated on jms?
On local machine it works, but on jenkins it fails, so I have to add
Thread.sleep(3000);
but this is nasty.
org.awaitility seems to be missed usage:
await().atMost(2, TimeUnit.SECONDS).until(() -> return true));
I just need to do a pause to make jms propagate (put on jms queue) and wait for listener to act, which is putting message to database.
Then I have to call get rest endpoint to see it worked.
With topic it would be easier, because I would create test listener on topic.
But it is queue, there can be on listener that will get message.
Use org.awaitility with a JMS QueueBrowser, e.g.:
#Test
public void myTest() throws Exception {
...
await().atMost(2, TimeUnit.SECONDS).until(() -> return queueIsEmpty(queueName)));
...
}
private boolean queueIsEmpty(String queueName) {
ConnectionFactory cf = new MyBrokersConnectionFactory();
Connection connection = cf.createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
QueueBrowser browser = session.createBrowser(session.createQueue(queueName));
Enumeration enumeration = senderBrowser.getEnumeration();
while (enumeration.hasMoreElements()) {
return false;
}
return true;
}
A QueueBrowser is read only so there is no danger that it will actually consume the message.
Another potential option would be to create a consumer with a transacted session and then try to receive the message. If you actually did receive a message you could rollback the transaction and close the consumer.
Use retries (e.g. Spring RetryTemplate or Failsafe Retry Policy) to improve integration test execution time:
Retry the SQL query until record is present
Retry the REST endpoint until it is successful
Here an example to wait for a DB record; tweak the policies to your needs:
RetryTemplate retryTemplate = new RetryTemplate();
retryTemplate.setBackOffPolicy(new FixedBackOffPolicy());
retryTemplate.setRetryPolicy(new SimpleRetryPolicy(
10, Collections.singletonMap(AssertionError.class, true)));
retryTemplate.execute(retryContext -> {
List<MyRecord> records = jdbcTemplate.query("select ...");
Assert.assertEquals(1, records.size());
return null;
});
My solution is to use org.awaitility lib and replace asserts with return statement:
await().atMost(30, TimeUnit.SECONDS).until(
() -> {
//
// assertTrue(condition);
return condition == true;
}
I have requirement where workflow should start execution but it should wait for next subsequent events to do the execution. Events are generated based of API calls that the service receives from client. The events semantics is based on a state machine. Please let me know how this can be implemented in java. Please provide any sample code for reference.
If you are using AWS Flow Framework an external event should use Signal API to notify a workflow instance. Inside a workflow it becomes a handler method like:
#Workflow
#WorkflowRegistrationOptions(
defaultExecutionStartToCloseTimeoutSeconds = 60,
defaultTaskStartToCloseTimeoutSeconds = 10)
public interface MyWorkflow
{
#Execute(version = "1.0")
void startMyWF();
#Signal
void signal1();
}
public class MyWFImpl implements MyWorkflow
{
MyActivitiesClient client = new MyActivitiesClientImpl();
// Used to block the workflow until a signal is received.
Settable<Void> signal1Called = new Settable<Void>();
#Override
public void startMyWF(){
Promise<Integer> result = client.activity1();
// Continues when both result and signal1 are ready.
client.activity2(result, signal1);
}
#Override
public void signal1() {
//Process signal
signal1.set(null);
}
}
http://docs.aws.amazon.com/amazonswf/latest/awsflowguide/features.workflow.html describes how to write a workflow interface.
http://docs.aws.amazon.com/amazonswf/latest/awsflowguide/workflowimpl.html describes how to implement workflows.
http://docs.aws.amazon.com/amazonswf/latest/awsflowguide/clients.html describes how to send signals using external client.
I encapsulated JCSMP API in a class and provide methods for other classes to send and consume messages.
I have the following method:
public void send(byte[] data, String queueName) throws JCSMPException {
// Acquire a message producer
if (producer == null) { //producer is an instance of XMLMessageProducer
producer = session.getMessageProducer(new PublishCallback());
}
Queue queue = JCSMPFactory.onlyInstance().createQueue(queueName);
BytesMessage msg = JCSMPFactory.onlyInstance().createMessage(BytesMessage.class);
msg.setData(data);
msg.setDeliveryMode(DeliveryMode.PERSISTENT);
logger.info("Sending to \"{}\"", queueName);
producer.send(msg, queue);
}
When I have multiple threads calling this method, once in a while, I notice the following exception:
com.solacesystems.jcsmp.ClosedFacilityException: Tried to perform operation on a closed XML message producer
I wonder whether the producer is closed after each call to send(). How should I make this method thread-safe?
Thank you.
XMLMessageProducer does not close itself after each call to send.
The first step here is to investigate why your XMLMessageProducer is closed. The easiest option to do this is to enable the Solace API logging to INFO (or even DEBUG), and edit your question to include the Solace API logs prior to the first ClosedFacilityException.
One possible reason is that your application has disconnected, but was not able to automatically reconnect to the Solace appliance/VMR.
I need to implement a listener that receives multiple messages then groups them base on a certain criteria then send the grouped messages to the next queue. In other words batching them in a logical way.
I'm using Grails and JMS integration plugin.
In my mind, it can be implemented in a listener that runs a single thread that runs an infinite loop that forever consume messages from a queue then do the grouping logic there.
Runnable runnable = new Runnable() {
#Override
public void run() {
while(true) {
def obj = consumer.receive()
//group the messages and store on a list then send them in groups to the next queue
producer.send(groupedObjList)
}
}
}
def thread = new Thread(runnable);
thread.start();
I'm not sure if this is a clean way to do it. So I'm looking for alternatives on how to implement this.
I have a Struts2 Action class that places a JMS Fetch request for a list of Trade in a JMS Queue. This JMS Fetch message is processed by an external process and can take either a few seconds or even few minutes depending on the number of Trade files to be processed by the external task processing app.
I want to know how to handle this HTTP Request with an appropriate response. Does the client wait till the list of Trades is returned? (client (UI) has to action on it and has nothing else to do meanwhile).
The way I approached it is
HTTP Request -->
Struts2 Action -->
Invokes a Runnable to run in a separate Thread (separate from Action class)
UI waits
Action class thread sleeps till runnable does it's job
When Task completed, return list of Trades to UI
Flow is as follows:
Place JMS Fetch Request on Queue1
ExecutorService for Runnable
CClass cclass = new CClass();
final ExecutorService execSvc = Executors.newFixedThreadPool(1);
execSvc.execute(cclass);
Where CClass implements runnable returning a list of Trades:
List<Trade> tradesList = new ArrayList<Trade>();
#Override
public void run() {
while (true) {
try {
Message message = msgConsumer.receive(); // SYNCHRONOUS / NO MDB
if (message == null){
break;
}
if (message instanceof TextMessage) {
TextMessage txtMessage = (TextMessage) message;
Trade trade = TradeBuilder.buildTradeFromInputXML(txtMessage);
if (trade != null) {
tradesList.add(trade); // tradeList is a CClass class variable
}
}
} catch (JMSException e) {
logger.error("JMSException occurred ", e);
}
}
closeConnection();
}
And while this runnableis executing, I do a Thread.sleep in Action class (to let the Runnable execute in the separate Thread)
// In Action class
try {
Thread.sleep(5000); // some time till when the runnable will get executed
} catch (InterruptedException e) {
e.printStackTrace();
}
execSvc.shutdown();
Problem is If I use Callable with a FutureTask and do a get() , that will be blocking till any result is returned. If I do a Runnable, I have to put Action class Thread to sleep till runnable has executed and tradeList is available.
Using Runnable approach, I am able to get couple of hundred records back to UI giving a 5 second Thread.sleep() in main Action class, but only partially constructed tradeList when thousands of records are to be fetched and shown in UI.
This is clearly Not a fail-proof approach.
Any better approach to suggest ? Please elucidate steps for processing in one complete request - response flow.
Yes there is a much better approach when making a standard HTTP request (with ajax you can do other things).
You want to look at the Struts2 Execute and Wait Interceptor Which has most of the functionality you've already implemented. Also look at the token interceptor... which could be useful (it prevents duplicate requests, but doesn't provide a happy wait screen like exec and wait does).