I am trying to make kind of a polling service towards a activemq queue using camel routes.
I am using routing and routing-jsm plugins for grails.
I have my route configuration set like this.
class QueueRoute {
def configure = {
from("activemq:daemon").routeId("daemonRoute")
.noAutoStartup()
.shutdownRunningTask(ShutdownRunningTask.CompleteCurrentTaskOnly)
.to('bean:daemonCamelService?method=receive')
.end()
}
}
and I am basically trying to do .suspendRoute("daemonRoute") and .resumeRoute("daemonRoute") with some time inbetween. Though after issuing suspendRoute the route is not stopped.
Anyone have tried this?, I have read something about needing to kill the exchange in progress or something similar.
if you are just trying to periodically process all messages in a queue, then another option (instead of starting and stopping the route) is to use a timer and a polling consumer bean to do retrieve all the messages in the queue...
from("timer://processQueueTimer?fixedRate=true&period=30000")
.to("bean:myBean?method=poll");
public class MyBean {
public void poll() {
// loop to empty queue
while (true) {
// receive the message from the queue, wait at most 3 sec
Object msg = consumer.receiveBody("activemq:queue:daemon", 3000);
if (msg == null) {
// no more messages in queue
break;
}
// send it to the next endpoint
producer.sendBody("bean:daemonCamelService?method=receive", msg);
}
}
}
See this FAQ how to stop/suspend a route from a route
http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
An alternative is to use a route policy
http://camel.apache.org/routepolicy
For example as we do with the throttling route policy that is provided out of the box, take a look at how its implemented, you can do similar for your route as well.
Related
I have integration test with docker using test containers. On container I run jms. In test I am putting message on queue.
How I can wait in test to make it populated on jms?
On local machine it works, but on jenkins it fails, so I have to add
Thread.sleep(3000);
but this is nasty.
org.awaitility seems to be missed usage:
await().atMost(2, TimeUnit.SECONDS).until(() -> return true));
I just need to do a pause to make jms propagate (put on jms queue) and wait for listener to act, which is putting message to database.
Then I have to call get rest endpoint to see it worked.
With topic it would be easier, because I would create test listener on topic.
But it is queue, there can be on listener that will get message.
Use org.awaitility with a JMS QueueBrowser, e.g.:
#Test
public void myTest() throws Exception {
...
await().atMost(2, TimeUnit.SECONDS).until(() -> return queueIsEmpty(queueName)));
...
}
private boolean queueIsEmpty(String queueName) {
ConnectionFactory cf = new MyBrokersConnectionFactory();
Connection connection = cf.createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
QueueBrowser browser = session.createBrowser(session.createQueue(queueName));
Enumeration enumeration = senderBrowser.getEnumeration();
while (enumeration.hasMoreElements()) {
return false;
}
return true;
}
A QueueBrowser is read only so there is no danger that it will actually consume the message.
Another potential option would be to create a consumer with a transacted session and then try to receive the message. If you actually did receive a message you could rollback the transaction and close the consumer.
Use retries (e.g. Spring RetryTemplate or Failsafe Retry Policy) to improve integration test execution time:
Retry the SQL query until record is present
Retry the REST endpoint until it is successful
Here an example to wait for a DB record; tweak the policies to your needs:
RetryTemplate retryTemplate = new RetryTemplate();
retryTemplate.setBackOffPolicy(new FixedBackOffPolicy());
retryTemplate.setRetryPolicy(new SimpleRetryPolicy(
10, Collections.singletonMap(AssertionError.class, true)));
retryTemplate.execute(retryContext -> {
List<MyRecord> records = jdbcTemplate.query("select ...");
Assert.assertEquals(1, records.size());
return null;
});
My solution is to use org.awaitility lib and replace asserts with return statement:
await().atMost(30, TimeUnit.SECONDS).until(
() -> {
//
// assertTrue(condition);
return condition == true;
}
I have one resque job which is run at some event which finally publishes the message to RabbitMQ's exchange, how can I check in bunny(Rabbit MQ ruby client) that whether the message has been successfully published?
Using Acknowledgment or any way?
Thanks in advance!
When you execute the publish you are not sure that the message is published on the queue.
If you want to be sure you have to use you have to use publish confirm or tx transaction.
Read this post http://www.rabbitmq.com/blog/2011/02/10/introducing-publisher-confirms/
Note: By default the clients don't have any HA policy, you have to implement it. See the section Streaming Lightweight Publisher Confirms:
private volatile SortedSet<Long> unconfirmedSet =
Collections.synchronizedSortedSet(new TreeSet());
...
ch.setConfirmListener(new ConfirmListener() {
public void handleAck(long seqNo, boolean multiple) {
if (multiple) {
unconfirmedSet.headSet(seqNo+1).clear();
} else {
unconfirmedSet.remove(seqNo);
}
}
public void handleNack(long seqNo, boolean multiple) {
// handle the lost messages somehow
}
});
Note2: the message is never "put" inside an exchange, but always inside a queue.
Once the publish method returns, the message has published to the queue. There is no deferred action to publishing a message.
The controller layer can get the IP using request.getRemoteAddr() and/or request.getHeader("Client-IP") etc.
However, down in the bowels of the service layer, we might want to log some detected or suspected fraudulent activity by the user, along with the IP address of the user. However, the IP is not available to the service layer, nor is the request.
Obviously, every call from every controller method to every single service method could also pass in the IP or the request, but as we have thousands of these calls and lots of chains of them, it is not really practical.
Can anyone think of a better way?
As we are not in charge of instantiation of the services (these just get magically injected), we can't even pass the IP in when each service is created for the current HTTP call.
UPDATE 1
As suggested, tried the MDC route. Unfortunately, this does not seem to work.
in filter:
import org.apache.log4j.MDC
class IpFilters {
def filters = {
all() {
before = {
MDC.put "IP", "1.1.1.1"
println "MDC.put:" + MDC.get("IP")
}
afterView = { Exception e ->
println "MDC.remove:" + MDC.get("IP")
MDC.remove 'IP'
}
}
in service:
import org.apache.log4j.MDC
:
def someMethod() {
String ip = MDC.get("IP")
println("someMethod: IP = $ip")
}
The result is always:
MDC.put:1.1.1.1
MDC.remove:1.1.1.1
someMethod: IP = null
So the service cant access MDC variables put on the thread in the filter, which is a real shame. Possibly the problem is that "someMethod" is actually called by springSecuirty.
Well, it is highly recommended that we should keep the business logic aware of the controller logic. But keeping your situation in mind, you have to do that and absolutely available. In your service method, write this to log the IP address of the current request:
import org.springframework.web.context.request.RequestContextHolder
// ... your code and class
def request = RequestContextHolder.currentRequestAttributes().getRequest()
println request.getRemoteAddr()
Just make sure, you handle the whatever exception thrown from that line when the same service method is invoked from outside a Grails request context like from a Job.
my two pence worth
basically been using above and it works perfectly fine when a request is directed through standard grails practices.
In this scenario, user triggers websockets connection this then is injected into websockets listener using Holders.applicationContext.
The issue arises around are your outside of the web request.
the fix was painful but may come in handy for anyone else in this situation:
private static String userIp
String getIp() {
String i
new Thread({
//to bypass :
// Are you referring to request attributes outside of an actual web request, or processing a
// request outside of the originally receiving thread? If you are actually operating within a web request
// and still receive this message, your code is probably running outside of DispatcherServlet/DispatcherPortlet:
// In this case, use RequestContextListener or RequestContextFilter to expose the current request.
def webRequest = RequestContextHolder.getRequestAttributes()
if(!webRequest) {
def servletContext = ServletContextHolder.getServletContext()
def applicationContext = WebApplicationContextUtils.getRequiredWebApplicationContext(servletContext)
webRequest = grails.util.GrailsWebMockUtil.bindMockWebRequest(applicationContext)
}
//def request = RequestContextHolder.currentRequestAttributes().request
def request = WebUtils.retrieveGrailsWebRequest().currentRequest
i=request.getRemoteAddr()
if (!i ||i == '127.0.0.1') {
i=request.getHeader("X-Forwarded-For")
}
if (!i ||i == '127.0.0.1') {
i=request.getHeader("Client-IP")
}
if (!i) { i="127.0.0.1"}
this.userIp=i
} as Runnable ).start()
return i
}
Now when calling this some sleep time is required due to it running in as a runnable :
def aa = getIp()
sleep(300)
println "$aa is aa"
println "---- ip ${userIp}"
Also provided alternative way of calling request def request = WebUtils.retrieveGrailsWebRequest().currentRequest in grails 3 the commented out line .request comes up unrecognised in ide (even though it works)
the new Thread({ was still needed since even though it returned ip after getting ip it was attempting to save to a db and some other bizarre issue appeared around
java.lang.RuntimeException: org.springframework.mock.web.MockHttpServletRequest.getServletContext()Ljavax/servlet/ServletContext;
at org.apache.tomcat.websocket.pojo.PojoMessageHandlerBase.handlePojoMethodException(PojoMessageHandlerBase.java:119)
at org.apache.tomcat.websocket.pojo.PojoMessageHandlerWholeBase.onMessage(PojoMessageHandlerWholeBase.java:82)
so the fix to getting hold of request attribute in this scenario is above
for the mock libraries you will require this in build.gradle:
compile 'org.springframework:spring-test:2.5'
So the saga continued - the above did not actually appear to work in my case since basically the request originated by user but when sent to websockets - the session attempting to retrieve Request (ip/session) was not actual real user.
This in the end had to be done a very different way so really steeply off the topic but when this method of attempting ip does not work the only way left is through SessionListeners:
in src/main/groovy/{packageName}
class SessionListener implements HttpSessionListener {
private static List activeUsers = Collections.synchronizedList(new ArrayList())
static Map sessions = [:].asSynchronized()
void sessionCreated (HttpSessionEvent se) {
sessions.put(se.session.id, se.session)
}
void sessionDestroyed (HttpSessionEvent se) {
sessions.remove(se.session.id)
}
}
in grails-app/init/Application.groovy
Closure doWithSpring() {
{ ->
websocketConfig WebSocketConfig
}
}
// this already exists
static void main(String[] args) {
GrailsApp.run(Application, args)
}
in that same init folder:
class WebSocketConfig {
#Bean
public ServletContextInitializer myInitializer() {
return new ServletContextInitializer() {
#Override
public void onStartup(ServletContext servletContext) throws ServletException {
servletContext.addListener(SessionListener)
}
}
}
}
Now to get userIP, when the socket initially connects it sends the user's session to sockets. the socket registers that user's session within the websockets usersession details.
when attempting to get the user ip (i have registered the users ip to session.ip on the controller/page hitting the page opening sockets)
def aa = SessionListener.sessions.find{it.key==sessionId}?.value
println "aa $aa.ip"
I am working with a legacy Windows Service that reads messages from a private MSMQ queue processes them (does some database work, sends some emails) and then waits for the next message (PeekCompleted)
The service is problematic - whenever Windows Update requires a server reboot (so like almost always) the Service comes back up in a "Started" condition but has to be REstarted manually or the messages just pile up in the queue.
My first inclination is to think that there is something in the OnStart handler that isn't getting hit when the server comes back up and I am attempting to sort out the Logs (another story) but Windows Services and threading are not my normal domain so I am hoping someone can point me in the right direction....
Below are the OnStart Handler and message handling function, stripped inconsequential stuff.
Question: in OnStart the MessageRecieved function is attached to the PeekCompleted event.
I assume OnStart fires when the server comes back up so the handler must get attached, but I am not clear whether message that were (a) already in the queue at re-boot or (b) arrive during re-boot will actually trigger the event ?
If it should is there something else I should be looking for?
Any suggestions welcome!
protected override void OnStart(string[] args)
{
try
{
_inProcess = false;
_queueMessage = null;
_stopping = false;
_queue = ReadyQueue(_queueName);
if (_queue == null)
{
throw new Exception(string.Format("'ReadyFormQueue({0})' returned null", _queueName));
}
_queue.PeekCompleted += new PeekCompletedEventHandler(MessageReceived);
_queue.Formatter = new BinaryMessageFormatter();
_queue.BeginPeek();
}
catch (Exception exception)
{
//do cleanup and other recovery stuff
}
}
private void MessageReceived(object sender, PeekCompletedEventArgs e)
{
_currentMessage = null;
_inProcess = false;
try
{
_queueMessage = _queue.EndPeek(e.AsyncResult);
_queueMessage.Formatter = new BinaryMessageFormatter();
_currentMessage = (MyMessageType)_queueMessage.Body;
_queue.ReceiveById(_queueMessage.Id);
_inProcess = true;
_helper = new MessageHelper();
_currentMessage = _helper.Process(_currentMessage); //sets global _inProcess flag
if (_inProcess)
{
Thread.Sleep((int)(_retryWaitTime * 0x3e8));
SendFormMessageToQueue(FailedQueueName, _currentMessage);
}
else
{
_queue.BeginPeek();
}
}
catch (Exception exception)
{
_inProcess = false;
//do other recovery stuff
if (_currentMessage != null)
{
ReadyFormQueue(_poisonQueueName);
SendFormMessageToQueue(_poisonQueueName, _currentMessage);
}
}
}
This legacy windows service could be started before the queueing infrastructure is up and fully operational, must fail in the initial connection and therefore isn't processing messages.
The first thing that I would check (unless the windows service has proper logging) is if there is a windows service dependency that is properly set up - you don't want your legacy service to fully start until the MSMQ service has itself completely started.
I don't think there is a problem in the legacy service per say since once you restart it, it seems to work fine, I think you have a resource-available-race type of problem where the consumer starts before the resource and it wasn't completely designed to recover from that.
I would: create a service dependency (can be done in the SCM) and then reboot the server and see if you have any more MSMQ messages pilling up, my guess the answer will be no.
Hope this helps
I have a Struts2 Action class that places a JMS Fetch request for a list of Trade in a JMS Queue. This JMS Fetch message is processed by an external process and can take either a few seconds or even few minutes depending on the number of Trade files to be processed by the external task processing app.
I want to know how to handle this HTTP Request with an appropriate response. Does the client wait till the list of Trades is returned? (client (UI) has to action on it and has nothing else to do meanwhile).
The way I approached it is
HTTP Request -->
Struts2 Action -->
Invokes a Runnable to run in a separate Thread (separate from Action class)
UI waits
Action class thread sleeps till runnable does it's job
When Task completed, return list of Trades to UI
Flow is as follows:
Place JMS Fetch Request on Queue1
ExecutorService for Runnable
CClass cclass = new CClass();
final ExecutorService execSvc = Executors.newFixedThreadPool(1);
execSvc.execute(cclass);
Where CClass implements runnable returning a list of Trades:
List<Trade> tradesList = new ArrayList<Trade>();
#Override
public void run() {
while (true) {
try {
Message message = msgConsumer.receive(); // SYNCHRONOUS / NO MDB
if (message == null){
break;
}
if (message instanceof TextMessage) {
TextMessage txtMessage = (TextMessage) message;
Trade trade = TradeBuilder.buildTradeFromInputXML(txtMessage);
if (trade != null) {
tradesList.add(trade); // tradeList is a CClass class variable
}
}
} catch (JMSException e) {
logger.error("JMSException occurred ", e);
}
}
closeConnection();
}
And while this runnableis executing, I do a Thread.sleep in Action class (to let the Runnable execute in the separate Thread)
// In Action class
try {
Thread.sleep(5000); // some time till when the runnable will get executed
} catch (InterruptedException e) {
e.printStackTrace();
}
execSvc.shutdown();
Problem is If I use Callable with a FutureTask and do a get() , that will be blocking till any result is returned. If I do a Runnable, I have to put Action class Thread to sleep till runnable has executed and tradeList is available.
Using Runnable approach, I am able to get couple of hundred records back to UI giving a 5 second Thread.sleep() in main Action class, but only partially constructed tradeList when thousands of records are to be fetched and shown in UI.
This is clearly Not a fail-proof approach.
Any better approach to suggest ? Please elucidate steps for processing in one complete request - response flow.
Yes there is a much better approach when making a standard HTTP request (with ajax you can do other things).
You want to look at the Struts2 Execute and Wait Interceptor Which has most of the functionality you've already implemented. Also look at the token interceptor... which could be useful (it prevents duplicate requests, but doesn't provide a happy wait screen like exec and wait does).