I am working with a legacy Windows Service that reads messages from a private MSMQ queue processes them (does some database work, sends some emails) and then waits for the next message (PeekCompleted)
The service is problematic - whenever Windows Update requires a server reboot (so like almost always) the Service comes back up in a "Started" condition but has to be REstarted manually or the messages just pile up in the queue.
My first inclination is to think that there is something in the OnStart handler that isn't getting hit when the server comes back up and I am attempting to sort out the Logs (another story) but Windows Services and threading are not my normal domain so I am hoping someone can point me in the right direction....
Below are the OnStart Handler and message handling function, stripped inconsequential stuff.
Question: in OnStart the MessageRecieved function is attached to the PeekCompleted event.
I assume OnStart fires when the server comes back up so the handler must get attached, but I am not clear whether message that were (a) already in the queue at re-boot or (b) arrive during re-boot will actually trigger the event ?
If it should is there something else I should be looking for?
Any suggestions welcome!
protected override void OnStart(string[] args)
{
try
{
_inProcess = false;
_queueMessage = null;
_stopping = false;
_queue = ReadyQueue(_queueName);
if (_queue == null)
{
throw new Exception(string.Format("'ReadyFormQueue({0})' returned null", _queueName));
}
_queue.PeekCompleted += new PeekCompletedEventHandler(MessageReceived);
_queue.Formatter = new BinaryMessageFormatter();
_queue.BeginPeek();
}
catch (Exception exception)
{
//do cleanup and other recovery stuff
}
}
private void MessageReceived(object sender, PeekCompletedEventArgs e)
{
_currentMessage = null;
_inProcess = false;
try
{
_queueMessage = _queue.EndPeek(e.AsyncResult);
_queueMessage.Formatter = new BinaryMessageFormatter();
_currentMessage = (MyMessageType)_queueMessage.Body;
_queue.ReceiveById(_queueMessage.Id);
_inProcess = true;
_helper = new MessageHelper();
_currentMessage = _helper.Process(_currentMessage); //sets global _inProcess flag
if (_inProcess)
{
Thread.Sleep((int)(_retryWaitTime * 0x3e8));
SendFormMessageToQueue(FailedQueueName, _currentMessage);
}
else
{
_queue.BeginPeek();
}
}
catch (Exception exception)
{
_inProcess = false;
//do other recovery stuff
if (_currentMessage != null)
{
ReadyFormQueue(_poisonQueueName);
SendFormMessageToQueue(_poisonQueueName, _currentMessage);
}
}
}
This legacy windows service could be started before the queueing infrastructure is up and fully operational, must fail in the initial connection and therefore isn't processing messages.
The first thing that I would check (unless the windows service has proper logging) is if there is a windows service dependency that is properly set up - you don't want your legacy service to fully start until the MSMQ service has itself completely started.
I don't think there is a problem in the legacy service per say since once you restart it, it seems to work fine, I think you have a resource-available-race type of problem where the consumer starts before the resource and it wasn't completely designed to recover from that.
I would: create a service dependency (can be done in the SCM) and then reboot the server and see if you have any more MSMQ messages pilling up, my guess the answer will be no.
Hope this helps
Related
I have integration test with docker using test containers. On container I run jms. In test I am putting message on queue.
How I can wait in test to make it populated on jms?
On local machine it works, but on jenkins it fails, so I have to add
Thread.sleep(3000);
but this is nasty.
org.awaitility seems to be missed usage:
await().atMost(2, TimeUnit.SECONDS).until(() -> return true));
I just need to do a pause to make jms propagate (put on jms queue) and wait for listener to act, which is putting message to database.
Then I have to call get rest endpoint to see it worked.
With topic it would be easier, because I would create test listener on topic.
But it is queue, there can be on listener that will get message.
Use org.awaitility with a JMS QueueBrowser, e.g.:
#Test
public void myTest() throws Exception {
...
await().atMost(2, TimeUnit.SECONDS).until(() -> return queueIsEmpty(queueName)));
...
}
private boolean queueIsEmpty(String queueName) {
ConnectionFactory cf = new MyBrokersConnectionFactory();
Connection connection = cf.createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
QueueBrowser browser = session.createBrowser(session.createQueue(queueName));
Enumeration enumeration = senderBrowser.getEnumeration();
while (enumeration.hasMoreElements()) {
return false;
}
return true;
}
A QueueBrowser is read only so there is no danger that it will actually consume the message.
Another potential option would be to create a consumer with a transacted session and then try to receive the message. If you actually did receive a message you could rollback the transaction and close the consumer.
Use retries (e.g. Spring RetryTemplate or Failsafe Retry Policy) to improve integration test execution time:
Retry the SQL query until record is present
Retry the REST endpoint until it is successful
Here an example to wait for a DB record; tweak the policies to your needs:
RetryTemplate retryTemplate = new RetryTemplate();
retryTemplate.setBackOffPolicy(new FixedBackOffPolicy());
retryTemplate.setRetryPolicy(new SimpleRetryPolicy(
10, Collections.singletonMap(AssertionError.class, true)));
retryTemplate.execute(retryContext -> {
List<MyRecord> records = jdbcTemplate.query("select ...");
Assert.assertEquals(1, records.size());
return null;
});
My solution is to use org.awaitility lib and replace asserts with return statement:
await().atMost(30, TimeUnit.SECONDS).until(
() -> {
//
// assertTrue(condition);
return condition == true;
}
I have been struggling with WCF Proxies. What is the correct way to Dispose a WCF Proxy? The answer is not trivial.
System.ServiceModel.ClientBase violates Microsoft's own Dispose-pattern
System.ServiceModel.ClientBase<TChannel> does implement IDisposable so one must assume that it should be disposed or used in a using-block. These are best practices for anything disposable. The implementation is explicit, however, so one does have to explicitly cast ClientBase instances to IDisposable, clouding the issue.
The biggest source of confusion, however, is that calling Dispose() on ClientBase instances that faulted, even channels that faulted because they never opened in the first place, will result in an exception being thrown. This, inevitably, means that the meaningful exception explaining the fault is immediately lost when the stack unwinds, the using scope ends and Dispose() throws a meaningless exception saying that you can't dispose a faulted channel.
The above behaviour is anathema to the dispose pattern which states that objects must be tolerant of multiple explicit calls to Dispose(). (see http://msdn.microsoft.com/en-us/library/b1yfkh5e(v=vs.110).aspx, "...allow the Dispose(bool) method to be called more than once. The method might choose to do nothing after the first call.")
With the advent of inversion-of-control, this poor implementation becomes a real problem. I.O.C. containers (specifically, Ninject) detect the IDisposable interface and call Dispose() explicitly on activated instances at the end of an injection scope.
Solution: Proxy ClientBase and Intercept calls to Dispose()
My solution was to proxy ClientBase by subclassing System.Runtime.Remoting.Proxies.RealProxy and to hijack or intercept calls to Dispose(). My first replacement for Dispose() went something like this:
if (_client.State == CommunicationState.Faulted) _client.Abort();
else ((IDisposable)_client).Dispose();
(Note that _client is a typed reference to the target of the proxy.)
Problems with NetTcpBinding
I thought that this had nailed it, initially, but then I discovered a problem in production: under certain scenarios that were fiendishly difficult to reproduce, I found that channels using a NetTcpBinding were not closing properly in the unfaulted case, even though Dispose was being called on _client.
I had an ASP.NET MVC Application using my proxy implementation to connect to a WCF Service using a NetTcpBinding on the local network, hosted within a Windows NT Service on a service cluster with only one node. When I load-tested the MVC Application, certain endpoints on the WCF Service (which was using port-sharing) would stop responding after a while.
I struggled to reproduce this: the same components running across the LAN between two developer's machines worked perfectly; a console application hammering the real WCF endpoints (running on the staging service cluster) with many processes and many threads in each worked; configuring the MVC Application on the staging server to connect to the endpoints on a developer's machine worked under load; running the MVC Application on a developer's machine and connecting to the staging WCF endpoints worked. The last scenario only worked under IIS Express, however, and this was a breakthrough. The endpoints would sieze up when load-testing the MVC Application under full-fat IIS on a developer's machine, connecting to the staging service cluster.
Solution: Close the Channel
After failing to understand the problem and reading many, many pages of the MSDN and other sources that claimed the problem shouldn't exist at all, I tried a long-shot and changed my Dispose() work-around to...
if (_client.State == CommunicationState.Faulted) _client.Abort();
else if (_client.State == CommunicationState.Opened)
{
((IContextChannel)_client.Channel).Close();
((IDisposable)_client).Dispose();
}
else ((IDisposable)_client).Dispose();
... and the problem stopped occurring in all test setups and under load in the staging environment!
Why?
Can anyone explain what might have been happening and why explicitly closing the Channel before calling Dispose() solved it? As far as I can tell, this shouldn't be necessary.
Finally, I return to the opening question: What is the correct way to Dispose a WCF Proxy? Is my replacement for Dispose() adequate?
The issue, as far as I have been able to understand, is that calling Dispose disposes off the handle, but doesn't actually close the channel, which then holds on to the resources and then eventually times out.
This is why your service stopped responding after a while during load testing: because the initial calls held on to resources longer than you thought they would, and later calls could then not avail those resources.
I came up with the following solution. The premise of the solution is that calling Dispose should be enough to dispose off the handle as well as close the channel. The additional benefit is that if the client ends up in a faulted state, it is recreated so that subsequent calls succeed.
If ServiceClient<TService> is injected into another class via a dependency injection framework like Ninject, then all resources will properly be released.
NB: Please note that in the case of Ninject, the binding must define a scope, i.e., it must not be missing an InXyzScope or be defined with an InTransientScope. If no scope makes sense, then use InCallScope.
Here's what I came up with:
public class ServiceClient<TService> : IDisposable
{
private readonly ChannelFactory<TService> channelFactory;
private readonly Func<TService> createChannel;
private Lazy<TService> service;
public ServiceClient(ChannelFactory<TService> channelFactory)
: base()
{
this.channelFactory = channelFactory;
this.createChannel = () =>
{
var channel = ChannelFactory.CreateChannel();
return channel;
};
this.service = new Lazy<TService>(() => CreateChannel());
}
protected ChannelFactory<TService> ChannelFactory
{
get
{
return this.channelFactory;
}
}
protected Func<TService, bool> IsChannelFaulted
{
get
{
return (service) =>
{
var channel = service as ICommunicationObject;
if (channel == null)
{
return false;
}
return channel.State == CommunicationState.Faulted;
};
}
}
protected Func<TService> CreateChannel
{
get
{
return this.createChannel;
}
}
protected Action<TService> DisposeChannel
{
get
{
return (service) =>
{
var channel = service as ICommunicationObject;
if (channel != null)
{
switch (channel.State)
{
case CommunicationState.Faulted:
channel.Abort();
break;
case CommunicationState.Closed:
break;
default:
try
{
channel.Close();
}
catch (CommunicationException)
{
}
catch (TimeoutException)
{
}
finally
{
if (channel.State != CommunicationState.Closed)
{
channel.Abort();
}
}
break;
}
}
};
}
}
protected Action<ChannelFactory<TService>> DisposeChannelFactory
{
get
{
return (channelFactory) =>
{
var disposable = channelFactory as IDisposable;
if (disposable != null)
{
disposable.Dispose();
}
};
}
}
public void Dispose()
{
DisposeChannel(this.service.Value);
DisposeChannelFactory(this.channelFactory);
}
public TService Service
{
get
{
if (this.service.IsValueCreated && IsChannelFaulted(this.service.Value))
{
DisposeChannel(this.service.Value);
this.service = new Lazy<TService>(() => CreateChannel());
}
return this.service.Value;
}
}
}
In my mvc webapplication, I am using webapi to connect to my database through odata.
Both MVC WebApp and Odata WebApi are on different ports of Azure cloud service webrole endpoints.
MVC WebApp - 80
Odata WebApi - 23900
When I do a odataproxy updateobject and call savechanges like
odataProxy.UpdateObject(xxx);
odataProxy.SaveChanges(System.Data.Services.Client.SaveChangesOptions.PatchOnUpdate);
I am getting a weird exception on savechanges method call - unable to connect to remote server.
When I tried to look into inner exceptions, It says that - No connection could be made as the target machine actively refused it 127.0.0.1:23901
So if you observe the port number in the exception, it shows as 23901 and obviously this error should come as the request is supposed to hit 23900.
I am facing this exception only when running on azure cloud solution. Whenever I do an update request, it fails by hitting a wrong port (added by 1).
Another thing is, apart from this updateobject -> savechanges, rest all works like fetching data and adding data.
FWIW, I've just run across this same thing. Darn near annoying and I really hope it doesn't happen in production. I'm surprised no other people have come across this though.
The idea of creating a new context, attaching the object(s) and calling SaveChanges really repulsed me because not only does it practically break all forms of testing, it causes debug code and production code to be fundamentally different.
I was however able to work around this problem in another way, by intercepting the request just before it goes out and using reflection to poke at some private fields in memory to "fix" the port number.
UPDATE: It's actually easier than this. We can intercept the request generation process with the BuildingRequest event. It goes something like this:
var context = new Context(baseUri);
context.BuildingRequest += (o, e) =>
{
FixPort(e);
};
Then the FixPort method just needs to test the port number and build a new Uri, attaching it back to the event args.
[Conditional("DEBUG")]
private static void FixPort(BuildingRequestEventArgs eventArgs)
{
int localPort = int.Parse(LOCAL_PORT);
if (eventArgs.RequestUri.Port != localPort)
{
var builder = new UriBuilder(eventArgs.RequestUri);
builder.Port = localPort;
eventArgs.RequestUri = builder.Uri;
}
}
Here's the original method using reflection and SendingRequest2, in case anyone is still interested.
First we create a context and attach a handler to the SendingRequest2 event:
var context = new Context(baseUri);
context.SendingRequest2 += (o, e) =>
{
FixPort(e.RequestMessage);
};
The FixPort method then handles rewriting the URL of the internal request, where LOCAL_PORT is the port you expect, in your case 23900:
[Conditional("DEBUG")]
private static void FixPort(IODataRequestMessage requestMessage)
{
var httpWebRequestMessage = requestMessage as HttpWebRequestMessage;
if (httpWebRequestMessage == null) return;
int localPort = int.Parse(LOCAL_PORT);
if (httpWebRequestMessage.HttpWebRequest.RequestUri.Port != localPort)
{
var builder = new UriBuilder(requestMessage.Url);
builder.Port = localPort;
var uriField = typeof (HttpWebRequest).GetField("_Uri",
BindingFlags.Instance | BindingFlags.NonPublic);
uriField.SetValue(httpWebRequestMessage.HttpWebRequest, builder.Uri);
}
}
I have found the root cause and a temporary workaround.
Cause:
When you hit WebApi through some port :23900 in Azure compute emulator and do an update or delete operation, somehow the last request is blocking the port and because of the port walking feature in Azure emulator, it is jumping to next port where there is no service available which is causing the issue.
Even this issue is found only in development emulators.
Temp Workaround:
Use a different proxy to attach to updated context object and then save from the other proxy object.
var odataProxy1 = xxx;
var obj = odataProxy1.xyz.FirstOrDefault();
obj.property1="abcd";
...//Other update assignments
var odataProxy2 = xxx;
odataProxy2.AttachTo("objEntitySet",obj);
odataProxy2.UpdateObject(obj)
odataProxy2.SaveChanges(ReplaceOrUpdate);
I am trying to make kind of a polling service towards a activemq queue using camel routes.
I am using routing and routing-jsm plugins for grails.
I have my route configuration set like this.
class QueueRoute {
def configure = {
from("activemq:daemon").routeId("daemonRoute")
.noAutoStartup()
.shutdownRunningTask(ShutdownRunningTask.CompleteCurrentTaskOnly)
.to('bean:daemonCamelService?method=receive')
.end()
}
}
and I am basically trying to do .suspendRoute("daemonRoute") and .resumeRoute("daemonRoute") with some time inbetween. Though after issuing suspendRoute the route is not stopped.
Anyone have tried this?, I have read something about needing to kill the exchange in progress or something similar.
if you are just trying to periodically process all messages in a queue, then another option (instead of starting and stopping the route) is to use a timer and a polling consumer bean to do retrieve all the messages in the queue...
from("timer://processQueueTimer?fixedRate=true&period=30000")
.to("bean:myBean?method=poll");
public class MyBean {
public void poll() {
// loop to empty queue
while (true) {
// receive the message from the queue, wait at most 3 sec
Object msg = consumer.receiveBody("activemq:queue:daemon", 3000);
if (msg == null) {
// no more messages in queue
break;
}
// send it to the next endpoint
producer.sendBody("bean:daemonCamelService?method=receive", msg);
}
}
}
See this FAQ how to stop/suspend a route from a route
http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
An alternative is to use a route policy
http://camel.apache.org/routepolicy
For example as we do with the throttling route policy that is provided out of the box, take a look at how its implemented, you can do similar for your route as well.
After complete of asynchronous call to WCF service I want set success message into session and show user the notification .
I tried use two ways for complete this operation.
1) Event Based Model.
client.GetDataCompleted += new EventHandler<GetDataCompletedEventArgs>(GetDataCompleted);
client.GetDataAsync(id, client);
private void GetDataCompleted(object obj, GetDataCompletedEventArgs e)
{
this.SetNotification(new Notification() { Message = e.Result, Type = NotificationType.Success });
}
In MyOperationCompleted event i can set notification to HttpContext.Current.Session, but I must waiting before this operation will completed and can't navigate to others pages.
2) IAsyncResult Model.
In this way I can navigate to other pages and make asynchronous calls to wcf service, but in GetDataCallback method can't set notification, becouse session = null.
client.BeginGetData(id, GetDataCallback, client);
private void GetDataCallback(IAsyncResult ar)
{
string name = ((ServiceReference1.Service1Client)ar.AsyncState).EndGetData(ar);
this.SetNotification(new Notification() { Message = name, Type = NotificationType.Success });
}
"Generate asynchronous operations" in service reference enabled.
Please help me with this trouble. Thanks.
I'm no wcf expert, but what I've found to work is wrapping your call to the Async version of your method in ThreadPool.QueueUserWorkItem. Without this, I had same blocking issue. So this seems to free up the main thread in your asp mvc to move on while another worker thread waits for the callback.
Also, I used AsyncController, although that alone was not enough without the worker thread.
See this: http://msdn.microsoft.com/en-us/library/ee728598.aspx
I used this as a guide, but still needed the ThreadPool.
Cheers