While running Spring Integration JMS deployed on Tomcat 7.0.26 (running as a Windows Service) communicating to ActiveMQ (5.8.0) , we find that periodically the Tomcat Service fails to stop.
Can anyone shed some light on how we go about tracking this issue down?
We have taken Thread and Heap dumps using VisualVM but don't know what we are looking for! Any guidance would be helpful.
To get around this issue we have to stop Tomcat using Task Manager and then restart the process.
There are no exceptions in the logs around the time of the failure, although Tomcat does appear to be attempting to stop.
The last couple of entries in the tomcat log file are shown below in case that helps:
2013-06-05 10:07:28,582 INFO pool-2-thread-2 org.springframework.beans.factory.support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory#6517a4c6: defining beans [org.springframework.integration.internalDefaultConfiguringBeanFactoryPostProcessor,debugLoggingChannel,org.springframework.integration.handler.LoggingHandler#0,debugLoggingChannel.adapter,infoLoggingChannel,org.springframework.integration.handler.LoggingHandler#1,infoLoggingChannel.adapter,org.springframework.integration.channel.interceptor.WireTap#0,incomingRadarEventsString,org.springframework.jms.listener.DefaultMessageListenerContainer#0,org.springframework.integration.jms.ChannelPublishingJmsMessageListener#0,jmsinalerts,org.springframework.integration.transformer.HeaderEnricher#0,org.springframework.integration.transformer.MessageTransformingHandler#0,processingChannelEnricher,org.springframework.integration.xml.transformer.UnmarshallingTransformer#0,org.springframework.integration.transformer.MessageTransformingHandler#1,unmarshallingAlertsChannel,convertToRadarAlertObjects,cepRadarAlertsUnMarshaller,processRadarEventsMessage,org.springframework.integration.config.ServiceActivatorFactoryBean#0,alertEventsServiceActivator,org.springframework.integration.transformer.HeaderEnricher#1,org.springframework.integration.transformer.MessageTransformingHandler#2,addMessageProcessingHeaders,messageProcessingHeadersEnricher,alertsEventsProcessor,org.springframework.integration.config.SplitterFactoryBean#0,splitAlertsChannel,org.springframework.integration.config.ConsumerEndpointFactoryBean#0,org.springframework.integration.config.FilterFactoryBean#0,singleAlertsPoleObject,org.springframework.integration.config.ConsumerEndpointFactoryBean#1,org.springframework.integration.channel.interceptor.WireTap#1,incomingPutPoleObjectsString,org.springframework.jms.listener.DefaultMessageListenerContainer#1,org.springframework.integration.jms.ChannelPublishingJmsMessageListener#1,jmsin,org.springframework.integration.transformer.HeaderEnricher#2,org.springframework.integration.transformer.MessageTransformingHandler#3,processingPutPoleObjectsChannelEnricher,failedUnmarshalMessage,org.springframework.integration.config.TransformerFactoryBean#738bf6bf,org.springframework.integration.config.ServiceActivatorFactoryBean#69f4b0dc,org.springframework.integration.handler.MessageHandlerChain#0,org.springframework.integration.config.ConsumerEndpointFactoryBean#2,failedUnmarshalMessageToJmsOut,org.springframework.integration.jms.JmsSendingMessageHandler#0,failedUnmarshalMessageDispatcher,toSplitPutPoleObjects,org.springframework.integration.xml.transformer.UnmarshallingTransformer#1,org.springframework.integration.transformer.MessageTransformingHandler#4,unmarshallPutPoleObjectsString,convertToPutPoleObjects,cepMarshaller,singlePoleObject,org.springframework.integration.config.SplitterFactoryBean#1,org.springframework.integration.config.ConsumerEndpointFactoryBean#3,org.springframework.integration.config.FilterFactoryBean#1,org.springframework.integration.config.ConsumerEndpointFactoryBean#4,discardEvent,filteredEvent,org.springframework.integration.transformer.HeaderEnricher#3,org.springframework.integration.transformer.MessageTransformingHandler#5,matchedFilterEnricher,org.springframework.integration.channel.interceptor.WireTap#2,toSubscribedChannels,org.springframework.integration.router.RecipientListRouter#0,org.springframework.integration.config.ConsumerEndpointFactoryBean#5,profileEventChannel,aggregateEventChannel,messageErrorHandler,channelExecutor,org.springframework.integration.config.ServiceActivatorFactoryBean#1,org.springframework.integration.config.ConsumerEndpointFactoryBean#6,exceptionGatewayProfile,profileEventChannelEnriched,org.springframework.integration.config.FilterFactoryBean#2,profileFilter,profileNotify,org.springframework.integration.config.ServiceActivatorFactoryBean#2,profileNotifier,profileOut,splitMatchedQueriesForMessage,org.springframework.integration.config.SplitterFactoryBean#2,org.springframework.integration.config.ConsumerEndpointFactoryBean#7,notificationChannel,org.springframework.integration.config.FilterFactoryBean#3,gemNotificationChannelFilter,org.springframework.integration.config.FilterFactoryBean#4,workflowChannelFilter,notificationExecutor,gemNotificationAcceptedChannel,org.springframework.integration.config.ServiceActivatorFactoryBean#3,profileNotifierActivator,org.springframework.integration.config.FilterFactoryBean#5,mopiChannelFilter,mopiOutputChannel,org.springframework.integration.config.ServiceActivatorFactoryBean#6dca1837,org.springframework.integration.handler.MessageHandlerChain#1,org.springframework.integration.config.ConsumerEndpointFactoryBean#8,org.springframework.integration.channel.interceptor.WireTap#3,mopiOutputChannelToJmsOut,org.springframework.integration.jms.JmsSendingMessageHandler#1,mopiOutputQueueDispatcher,workflowAcceptedChannel,org.springframework.integration.config.ServiceActivatorFactoryBean#4,workflowTriggerActivator,failedProfile,org.springframework.integration.config.TransformerFactoryBean#6128453c,org.springframework.integration.config.ServiceActivatorFactoryBean#1ad997f9,org.springframework.integration.handler.MessageHandlerChain#2,org.springframework.integration.config.ConsumerEndpointFactoryBean#9,failedProfileToJmsOut,org.springframework.integration.jms.JmsSendingMessageHandler#2,failedProfileDispatcher,incomingResolvedProfileString,org.springframework.jms.listener.DefaultMessageListenerContainer#2,org.springframework.integration.jms.ChannelPublishingJmsMessageListener#2,org.springframework.integration.jms.JmsMessageDrivenEndpoint#0,org.springframework.integration.xml.transformer.UnmarshallingTransformer#2,org.springframework.integration.transformer.MessageTransformingHandler#bd5d765,org.springframework.integration.config.ServiceActivatorFactoryBean#4302df5,org.springframework.integration.transformer.HeaderEnricher#4,org.springframework.integration.transformer.MessageTransformingHandler#771931f8,org.springframework.integration.handler.MessageHandlerChain#3,org.springframework.integration.config.ConsumerEndpointFactoryBean#10,discardProfile,org.springframework.integration.config.ServiceActivatorFactoryBean#5,org.springframework.integration.config.ConsumerEndpointFactoryBean#11,exceptionGatewayAggregate,aggregateEventChannelEnriched,org.springframework.integration.config.ServiceActivatorFactoryBean#6,receiveAggregate,aggregateEventChannelStarted,org.springframework.integration.config.FilterFactoryBean#6,aggregateFilter,toFieldRetreiver,org.springframework.integration.config.ServiceActivatorFactoryBean#7,fieldCapture,toEvalFunction,org.springframework.integration.config.ServiceActivatorFactoryBean#8,evalFunction,toAggregateCompleted,org.springframework.integration.config.ServiceActivatorFactoryBean#9,aggregateCompleted,splitAggregate,org.springframework.integration.config.SplitterFactoryBean#3,aggregateSplitter,aggregateOut,failedAggregate,org.springframework.integration.config.TransformerFactoryBean#2904b5ae,org.springframework.integration.config.ServiceActivatorFactoryBean#7433c78b,org.springframework.integration.handler.MessageHandlerChain#4,org.springframework.integration.config.ConsumerEndpointFactoryBean#12,failedAggregateToJmsOut,org.springframework.integration.jms.JmsSendingMessageHandler#3,failedAggregateDispatcher,incomingResolvedAggregateString,org.springframework.jms.listener.DefaultMessageListenerContainer#3,org.springframework.integration.jms.ChannelPublishingJmsMessageListener#3,org.springframework.integration.jms.JmsMessageDrivenEndpoint#1,org.springframework.integration.xml.transformer.UnmarshallingTransformer#3,org.springframework.integration.transformer.MessageTransformingHandler#7371b246,org.springframework.integration.config.ServiceActivatorFactoryBean#2d6f4ce0,org.springframework.integration.transformer.HeaderEnricher#5,org.springframework.integration.transformer.MessageTransformingHandler#61735602,org.springframework.integration.handler.MessageHandlerChain#5,org.springframework.integration.config.ConsumerEndpointFactoryBean#13,org.springframework.integration.config.ServiceActivatorFactoryBean#10,toAggregateDiscard,aggregateDiscarder,discardAggregate,org.springframework.integration.endpoint.MethodInvokingMessageSource#0,org.springframework.scheduling.support.CronTrigger#0,org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0,org.springframework.integration.config.SplitterFactoryBean#4,jobProcessingChannel,org.springframework.integration.config.ConsumerEndpointFactoryBean#14,singleJobChannel,org.springframework.integration.config.ServiceActivatorFactoryBean#11,singleJobSA,cepNotification-marshaller,cepNotification-messageSender,cepNotification-webServiceTemplate,cepNotificationClient,cepNotificationUddiDestinationProvider,cepNotificationPropertiesDestinationProvider,cepNotificationService,securityServiceCacheHelper,cepPoleUddiDestinationProvider,cepPolePropertiesDestinationProvider,cepPoleUddiDestinationProviderWorkflowService,cepWorkflowUddiDestinationProvider,cepWorkflowPropertiesDestinationProvider,cepPoleService,cepPoleFlowService,queryService,cepUddiClient,securityService,cepSecurityMarshaller,cepSecurityMessageSender,cepSecurityWebServiceTemplate,cepSecurityUddiDestinationProvider,cepSecurityPropertiesDestinationProvider,cepSecurityServiceClient,connectionFactory,localConnectionFactory,jmsTransactionManager,radarEventsAlertQueue,requestQueue,persistedProfileEvents,persistedAggregateEvents,errorUnmarshalMessageQueue,errorProfileQueue,mopiOutputQueue,resolvedProfileQueue,errorAggregateQueue,resolvedAggregateQueue,placeholderConfig,baseQueryFilter,profileQueryFilter,aggregateQueryFilter,eventTypeFilter,aggregateReceiver,aggregateComplete,aggregateDiscarded,notificationActionsUtility,replyBasedNotificationsUtility,notificationsFactory,profileQueryNotifier,poleQueryLoader,queryResolver,queryCache,fieldRetriever,poleFieldCapture,groupByFieldsFilter,evaluateFunction,notificationDispatcher,convertJmsPayloadToCommonServicesXml,errorUnwrapper,errorHandler,resolvedPoleHandler,gemNotificationFilterBean,workflowFilterBean,mopiFilterBean,profileQueryNotifierBean,aggregateSplitterBean,workflowTriggerBean,org.springframework.aop.config.internalAutoProxyCreator,com.googlecode.ehcache.annotations.key.CachingReflectionHelper,com.googlecode.ehcache.annotations.key.ListCacheKeyGenerator.DEFAULT_BEAN_NAME,com.googlecode.ehcache.annotations.key.HashCodeCacheKeyGenerator.DEFAULT_BEAN_NAME,com.googlecode.ehcache.annotations.key.MessageDigestCacheKeyGenerator.DEFAULT_BEAN_NAME,com.googlecode.ehcache.annotations.key.ReflectionHashCodeCacheKeyGenerator.DEFAULT_BEAN_NAME,com.googlecode.ehcache.annotations.key.StringCacheKeyGenerator.DEFAULT_BEAN_NAME,com.googlecode.ehcache.annotations.impl.CacheAttributeSourceImpl#0,com.googlecode.ehcache.annotations.impl.CacheStaticMethodMatcherPointcut#0,com.googlecode.ehcache.annotations.interceptor.EhCacheInterceptor#0,com.googlecode.ehcache.annotations.config.internalEhCacheCachingAdvisor,cepEhCacheManager,ehcacheWrapper,splitterHelper,matchedProfileQueriesSplitter,scheduledJobTrigger,jobBuilder,validQueriesTaskExecutor,delQueryDataTaskExecutor,jobProcessor,ObjectBeanInfoBuilder,cepConnectionManager,cepResilienceConnectionManager,cepRepository,dataSource,cepResilienceRepository,nullChannel,errorChannel,_org.springframework.integration.errorLogger,taskScheduler]; root of factory hierarchy
2013-06-05 10:07:29,456 INFO pool-2-thread-2 org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Shutting down ExecutorService
2013-06-05 10:07:29,456 INFO pool-2-thread-2 org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Shutting down ExecutorService 'taskScheduler'
2013-06-05 10:07:29,456 INFO pool-2-thread-2 org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Shutting down ExecutorService
Thanks,
Mark.