Today my Bitbucket instance stopped working for some reason. Stop, start and reboot only shows message
Migrating home directory
Without any progress. It just sits there with the progress bar without doing anything. I don't understand why it is "migrating" the home directory. I'm not upgrading, changing or anything my instance.
When I stop and then start the log file shows a few lines of messages and sits at the line that states
c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /opt/bitbucket-5.15.0/home
Is it possible to recover from this error in any way, shape or form? Why is Bitbucket migrating my home directory? What is it waiting for?
==> /opt/bitbucket/home/log/atlassian-bitbucket.log <==
2019-02-22 13:40:56,195 INFO [main] o.s.boot.SpringApplication
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.5.15.RELEASE)
2019-02-22 13:40:56,396 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on jupiter with PID 12057 (/opt/bitbucket-5.15.0/app/app/WEB-INF/classes started by bitbucket in /opt/bitbucket-5.15.0/home)
2019-02-22 13:40:56,397 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
2019-02-22 13:40:57,590 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 5.15.0 (bd5e9b4 built on Wed Oct 17 03:20:25 CEST 2018)
2019-02-22 13:40:57,590 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12
2019-02-22 13:40:59,132 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 3.971 seconds (JVM running for 5.519)
2019-02-22 13:41:01,964 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /opt/bitbucket-5.15.0/home
When I stop the service Bitbucket barfs up some error messages. Which suggest that Bitbucket is no longer able to successfully manage its various threads.
==> /opt/bitbucket/home/log/launcher.log <==
2019-02-22 13:54:30
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.192-b12 mixed mode):
"localhost-startStop-2" #44 daemon prio=5 os_prio=0 tid=0x00007f6570001000 nid=0x485d in Object.wait() [0x00007f65b1759000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c19661c0> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000c19661c0> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1326)
at com.atlassian.johnson.spring.lifecycle.LifecycleDispatcherServlet.destroy(LifecycleDispatcherServlet.java:55)
at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1301)
- locked <0x00000000c18bf930> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapper.stopInternal(StandardWrapper.java:1635)
- locked <0x00000000c18bf930> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c18bf930> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5466)
- locked <0x00000000c0872398> (a org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedContext)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c0872398> (a org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedContext)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1436)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1425)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Tomcat-startStop-2" #43 daemon prio=5 os_prio=0 tid=0x00007f6520424000 nid=0x485c waiting on condition [0x00007f65b3041000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ccc91900> (a java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:998)
- locked <0x00000000c0872748> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c0872748> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1436)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1425)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Thread-4" #34 prio=5 os_prio=0 tid=0x00007f6578348800 nid=0x485b waiting on condition [0x00007f6536b98000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000cca8cbb0> (a java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:998)
- locked <0x00000000c0872eb0> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c0872eb0> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:486)
- locked <0x00000000c0872eb0> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c08758d8> (a org.apache.catalina.core.StandardService)
at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:814)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c07b26c8> (a org.apache.catalina.core.StandardServer)
at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:376)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stopTomcat(TomcatEmbeddedServletContainer.java:256)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stop(TomcatEmbeddedServletContainer.java:310)
- locked <0x00000000c088ae18> (a java.lang.Object)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.stopAndReleaseEmbeddedServletContainer(EmbeddedWebApplicationContext.java:306)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onClose(EmbeddedWebApplicationContext.java:155)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1014)
at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929)
- locked <0x00000000c073b9a8> (a java.lang.Object)
"SIGTERM handler" #42 daemon prio=9 os_prio=0 tid=0x00007f659c001000 nid=0x4859 in Object.wait() [0x00007f65b1c5a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c1965eb0> (a org.springframework.context.support.AbstractApplicationContext$2)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000c1965eb0> (a org.springframework.context.support.AbstractApplicationContext$2)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000000c02068b8> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:748)
"PostgreSQL-JDBC-SharedTimer-2" #41 daemon prio=5 os_prio=0 tid=0x00007f6528f78000 nid=0x2fe9 in Object.wait() [0x00007f6536596000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ccb8d168> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000ccb8d168> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
I fixed this issue with following database statement
UPDATE DATABASECHANGELOGLOCK SET LOCKED=false, LOCKGRANTED=null, LOCKEDBY=null where ID=1;
Notes from Atlassian Community:
https://community.atlassian.com/t5/Bitbucket-questions/starting-bitbucket-hangs-on-quot-migrating-home-directory-quot/qaq-p/785834
More information https://confluence.atlassian.com/bitbucketserverkb/bitbucket-server-does-not-start-could-not-acquire-change-log-lock-779171307.html
Related
My client application is a Spring-shell application, running SpringBoot 2.1.1.
We are creating rabbitmq connection on demand CachingConnectionFacotry
CachingConnectionFactory connectionFactory = new CachingConnectionFactory(uri);
connectionFactory.setPublisherReturns(true);
When exiting the application, calling cachingConnectionFactory.destroy(); on #PreDestroy and/or onContextClosed events is not shutting down cachingConnectionFactory .
It is taking a minute for the application to exit.
Thread dump below indicates there is an executor inside cachingConnectionFactory that hasnt shutdown yet.
"spring-rabbit-deferred-pool-12" #29 prio=5 os_prio=31 tid=0x00007ff61245e000 nid=0x7503 waiting on condition [0x000070000fa57000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076c928300> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"spring-rabbit-deferred-pool-11" #28 prio=5 os_prio=31 tid=0x00007ff616461000 nid=0x7303 waiting on condition [0x000070000f954000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076c928300> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"process reaper" #14 daemon prio=10 os_prio=31 tid=0x00007ff614cd0800 nid=0x590b waiting on condition [0x000070000ec2d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c078c1e0> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Anything I can do to exit the application immediately?
I don't think it's related to that thread pool...
public final void destroy() {
super.destroy();
resetConnection();
if (getContextStopped()) {
this.stopped = true;
if (this.channelsExecutor != null) {
this.channelsExecutor.shutdownNow();
}
}
}
shutDownNow() is called after resetting the connection and, clearly, all of the threads are interruptible.
Best guess is something (perhaps network related) is delaying the resetConnection().
Post the complete thread dump while the delay is enountered, perhaps in a GitHub gist, or on pastebin, or similar.
my spring boot app with amqp rabbitMQ hangs without errors in one of the environments. The same config works ok in another.
The thread dump shows,
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <e62a7fa> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doStart(SimpleMessageListenerContainer.java:815)
- locked <5d300728> (a java.lang.Object)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:550)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:346)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:149)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:112)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:874)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544)
- locked <1c4259b2> (a java.lang.Object)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
and then all the AMQP consumer threads are in BLOCKED state,
"AMQPConsumerThread_20" - Thread t#100
java.lang.Thread.State: BLOCKED
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.isActive(SimpleMessageListenerContainer.java:870)
- waiting to lock <5d300728> (a java.lang.Object) owned by "main" t#32
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1000(SimpleMessageListenerContainer.java:95)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1310)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:268)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
The SpringBoot app then waits with the last line on console,
2017-02-08T17:48:26.91-0500 [APP/0] OUT 2017-02-08 17:48:26 [AMQPConsumerThread_1] INFO o.s.a.r.c.CachingConnectionFactory -
Created new connection: SimpleConnection#64f05682
[delegate=amqp://b489be6b-b6e5-41b5-8f31-3be3acac4518#x.y.z.w:5672/daad99cf-fa24-40fe-b0d1-f9312fb583be, localPort= 57509]
EDIT
when I drop the consumer count down to 1, the BLOCKED state goes away and the main thread state is as follows,
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <79908042> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doStart(SimpleMessageListenerContainer.java:815)
- locked <4f3dcc9> (a java.lang.Object)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:550)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:346)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:149)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:112)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:874)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544)
- locked <1a60fd96> (a java.lang.Object)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
But the app still does not start.
the rabbitmq webconsole shows the consumer connected with the last line in logs,
2017-02-08T18:36:09.99-0500 [APP/0] OUT 2017-02-08 18:36:09 [AMQPConsumerThread_1] INFO o.s.a.r.c.CachingConnectionFactory - Created new connection: SimpleConnection#39271079
[delegate=amqp://b489be6b-b6e5-41b5-8f31-3be3acac4518#10.146.54.74:5672/daad99cf-fa24-40fe-b0d1-f9312fb583be, localPort= 32880]
what am I missing?
thanks
This...
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <79908042> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
Indicates the SimpleAsyncTaskExecutor is at its thread limit (setConcurrencyLimit). This check is only performed if you have set a limit.
This won't proceed until some task terminates. Since the consumer threads are long-lived, that will likely not happen.
I have a VPS with Centos running with the following details:
[root#XXXXXXX~]# uname -a
Linux xxxxxxxx2.6.32-042stab055.10 #1 SMP Thu May 10 15:38:32 MSD 2012 i686 i686 i386 GNU/Linux
I am trying to run the ecommerce system shopizer from the tomcat installed on it.
I had tried to build it on the VPS but did not succeed so I built it else where and copied the war files on the VPS's tomcat.
The issue I am facing now and during the build too was that the build was hanging when i ran the built ant scripts. now when i am launching the tomcat server whose details are as follows :
Server version: Apache Tomcat/6.0.35
Server built: Nov 28 2011 11:20:06
Server number: 6.0.35.0
OS Name: Linux
OS Version: 2.6.32-042stab055.10
Architecture: i386
JVM Version: 1.6.0_24-b24
JVM Vendor: Sun Microsystems Inc.
the server hangs up at start and when I take thread dump (both in this case and the build case) I get the following :
"GC Daemon" daemon prio=10 tid=0xa09fd000 nid=0x36ef in Object.wait() [0xa08f7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa6c35a88> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:117)
- locked <0xa6c35a88> (a sun.misc.GC$LatencyLock)
"Low Memory Detector" daemon prio=10 tid=0xb7686000 nid=0x36ed runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread0" daemon prio=10 tid=0xb7684000 nid=0x36ec runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0xb7682800 nid=0x36eb waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0xb7673000 nid=0x36ea in Object.wait() [0xa0ffe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa6ad0b58> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0xa6ad0b58> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0xb7671800 nid=0x36e9 in Object.wait() [0xa1198000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0xb7605c00 nid=0x36e7 runnable [0xb775d000]
java.lang.Thread.State: RUNNABLE
at java.lang.Byte$ByteCache.<clinit>(Byte.java:79)
at java.lang.Byte.valueOf(Byte.java:102)
- waiting on <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0xb7605c00 nid=0x36e7 runnable [0xb775d000]
java.lang.Thread.State: RUNNABLE
at java.lang.Byte$ByteCache.<clinit>(Byte.java:79)
at java.lang.Byte.valueOf(Byte.java:102)
at com.opensymphony.xwork2.conversion.impl.DefaultTypeConverter.<init>(DefaultTypeConverter.java:59)
at com.opensymphony.xwork2.conversion.impl.XWorkConverter.<init>(XWorkConverter.java:186)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:419)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerImpl$ParameterInjector.inject(ContainerImpl.java:462)
at com.opensymphony.xwork2.inject.ContainerImpl.getParameters(ContainerImpl.java:477)
at com.opensymphony.xwork2.inject.ContainerImpl.access$000(ContainerImpl.java:34)
at com.opensymphony.xwork2.inject.ContainerImpl$MethodInjector.inject(ContainerImpl.java:293)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:431)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerImpl$ParameterInjector.inject(ContainerImpl.java:462)
at com.opensymphony.xwork2.inject.ContainerImpl.getParameters(ContainerImpl.java:477)
at com.opensymphony.xwork2.inject.ContainerImpl.access$000(ContainerImpl.java:34)
at com.opensymphony.xwork2.inject.ContainerImpl$MethodInjector.inject(ContainerImpl.java:293)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:431)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerImpl$ParameterInjector.inject(ContainerImpl.java:462)
at com.opensymphony.xwork2.inject.ContainerImpl.getParameters(ContainerImpl.java:477)
at com.opensymphony.xwork2.inject.ContainerImpl.access$000(ContainerImpl.java:34)
at com.opensymphony.xwork2.inject.ContainerImpl$MethodInjector.inject(ContainerImpl.java:293)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:431)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerBuilder$3.create(ContainerBuilder.java:93)
at com.opensymphony.xwork2.inject.ContainerBuilder$7.call(ContainerBuilder.java:487)
at com.opensymphony.xwork2.inject.ContainerBuilder$7.call(ContainerBuilder.java:484)
at com.opensymphony.xwork2.inject.ContainerImpl.callInContext(ContainerImpl.java:574)
at com.opensymphony.xwork2.inject.ContainerBuilder.create(ContainerBuilder.java:484)
at com.opensymphony.xwork2.config.impl.DefaultConfiguration.createBootstrapContainer(DefaultConfiguration.java:252)
at com.opensymphony.xwork2.config.impl.DefaultConfiguration.reloadContainer(DefaultConfiguration.java:193)
- locked <0xa37c05b0> (a com.opensymphony.xwork2.config.impl.DefaultConfiguration)
....
....
..
"VM Thread" prio=10 tid=0xb766d800 nid=0x36e8 runnable
"VM Periodic Task Thread" prio=10 tid=0xb7688400 nid=0x36ee waiting on condition
JNI global references: 911
Heap
def new generation total 39424K, used 5740K [0xa1580000, 0xa4040000, 0xa6ad0000)
eden space 35072K, 12% used [0xa1580000, 0xa19a8e88, 0xa37c0000)
from space 4352K, 34% used [0xa37c0000, 0xa3932570, 0xa3c00000)
to space 4352K, 0% used [0xa3c00000, 0xa3c00000, 0xa4040000)
tenured generation total 87424K, used 1432K [0xa6ad0000, 0xac030000, 0xb1580000)
the space 87424K, 1% used [0xa6ad0000, 0xa6c36038, 0xa6c36200, 0xac030000)
compacting perm gen total 12288K, used 11825K [0xb1580000, 0xb2180000, 0xb5580000)
the space 12288K, 96% used [0xb1580000, 0xb210c588, 0xb210c600, 0xb2180000)
No shared spaces configured.
What can I do to resolve the issue or is it a bug/issue with java running on VPS??
B) the jvm sometimes crashes on the same VPS with the following error :
[root#xxxxxx ~]# java
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
[root#xxxxxx ~]# free
total used free shared buffers cached
Mem: 1155072 561320 593752 0 0 317124
-/+ buffers/cache: 244196 910876
Swap: 0 0 0
Very strange to me, can anyone explain me this behaviour.
I suggest that you try to update your operating system and your JVM.
I think this crash is due to a bug in your actual configuration.
We have an intermittent problem with slaves hanging AFTER the job itself is finished. In the post processing step (?) what we see is that the console log has this line:
Description set: vap_current_iter-2012_03_29_19_01_03
And then nothing. Usually, it will look like this:
Description set: prod_pull-2012_03_28_19_01_03
Notifying upstream build armada_Launch_prod_pull #13 of job completion
Project armada_Launch_prod_pull still waiting for 1 builds to complete
Notifying upstream projects of job completion
Notifying upstream of completion: armada_Launch_prod_pull #13
Finished: SUCCESS
I setup a logger for hudson.model.Run, and it currently has this :
at java.lang.Thread.run(Thread.java:619)
Mar 30, 2012 12:44:00 PM hudson.model.Run run
INFO: galleon_allUnit #1134 main build action completed: SUCCESS
Mar 30, 2012 12:44:00 PM hudson.model.Run setResult
FINE: galleon_allUnit #1134 : result is set to SUCCESS
java.lang.Exception
at hudson.model.Run.setResult(Run.java:352)
at hudson.model.Run.run(Run.java:1410)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:238)
Repeated for every hung slave.
The main hudson log doesn't have any additional information.
Disconnecting the slave has no effect.
Trying to do an orderly shutdown of Jenkins has no effect (jenkins actually appears to hang on shutdown).
The only way we have found to recover is to kill -9 the tomcat process.
The tread dump for one of the slaves (they are all the same) is:
Thread Dump
Channel reader thread: channel
"Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:199)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked java.io.BufferedInputStream#1ae615a
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1030)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Channel#e1d5ea
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Channel#e1d5ea
at java.lang.Object.wait(Object.java:485)
at hudson.remoting.Channel.join(Channel.java:766)
at hudson.remoting.Launcher.main(Launcher.java:420)
at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:366)
at hudson.remoting.Launcher.run(Launcher.java:206)
at hudson.remoting.Launcher.main(Launcher.java:168)
Ping thread for channel hudson.remoting.Channel#e1d5ea:channel
"Ping thread for channel hudson.remoting.Channel#e1d5ea:channel" Id=10 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
Pipe writer thread: channel
"Pipe writer thread: channel" Id=12 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#14263ed
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#14263ed
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
pool-1-thread-267
"pool-1-thread-267" Id=285 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374)
at hudson.Functions.getThreadInfos(Functions.java:872)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:93)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:89)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Number of locked synchronizers = 1
- java.util.concurrent.locks.ReentrantLock$NonfairSync#1186f88
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock#1798fdd
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock#1798fdd
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock#1d40442
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock#1d40442
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Signal Dispatcher
"Signal Dispatcher" Id=4 Group=system RUNNABLE
Any ideas on how to better recover or prevent this would be greatly appreciated.
We honestly just wrote a script that restarts jenkins every night at 4pm. We found that our breakages were happening at 3am give or take a half hour or so. Since restarting the server at this time, we haven't seen any further hangs. This is a way to prevent it as you asked, though it doesn't "Fix" the problem obviously!
When i run a
mvn test
Maven runs all the tests successfully and then goes on a wait infinite loop
I see a java process in the task manager and I have to kill it before maven returns with this error message
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:33.949s
[INFO] Finished at: Fri Jun 17 14:50:01 EDT 2011
[INFO] Final Memory: 5M/247M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.
4:test (default-test) on project EDIFileHandling: There are test failures.
[ERROR]
[ERROR] Please refer to c:\code\target\surefire-reports for the individual test
results.
Any help appreciated.
Thanks
R
C:\Users\Raghuveer.Rao>jstack 7256
2011-06-17 15:24:55
Full thread dump Java HotSpot(TM) Client VM (20.0-b11 mixed mode, sharing):
"Thread-2" prio=6 tid=0x03ed3000 nid=0x14d0 waiting for monitor entry [0x04ddf00
0]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.hsqldb.Session.close(Unknown Source)
- waiting to lock <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- locked <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23b8efe8> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23b90f30> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2951f1c0> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
"Thread-1" prio=6 tid=0x03ed2c00 nid=0x1fe0 waiting for monitor entry [0x04d8f00
0]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- waiting to lock <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23ba9568> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2910d510> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
"Low Memory Detector" daemon prio=6 tid=0x01bdac00 nid=0x1eb0 runnable [0x000000
00]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread0" daemon prio=10 tid=0x01bc8400 nid=0x18e8 waiting on conditi
on [0x00000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" daemon prio=10 tid=0x01bc5400 nid=0x1ac0 waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x01bc2400 nid=0x1fbc runnable [0x0000000
0]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=0x01bbb800 nid=0x1c00 in Object.wait() [0x03cdf000
]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x290702b0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x290702b0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x01bba000 nid=0x810 in Object.wait() [0x
03c8f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x29070340> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x29070340> (a java.lang.ref.Reference$Lock)
"main" prio=6 tid=0x00e29800 nid=0x1e20 in Object.wait() [0x001bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x294c4300> (a org.springframework.context.support.Abstrac
tApplicationContext$3)
at java.lang.Thread.join(Thread.java:1186)
- locked <0x294c4300> (a org.springframework.context.support.AbstractApp
licationContext$3)
at java.lang.Thread.join(Thread.java:1239)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.
java:79)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.jav
a:24)
at java.lang.Shutdown.runHooks(Shutdown.java:79)
at java.lang.Shutdown.sequence(Shutdown.java:123)
at java.lang.Shutdown.exit(Shutdown.java:168)
- locked <0x38b53ef8> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Runtime.exit(Runtime.java:90)
at java.lang.System.exit(System.java:904)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.j
ava:986)
"VM Thread" prio=10 tid=0x01b7e400 nid=0x1e10 runnable
"VM Periodic Task Thread" prio=10 tid=0x01bec800 nid=0x1bac waiting on condition
JNI global references: 1776
Found one Java-level deadlock:
"Thread-2":
waiting to lock monitor 0x0467c1cc (object 0x23ba76f0, a org.hsqldb.Session),
which is held by "Thread-1"
"Thread-1":
waiting to lock monitor 0x01bbf56c (object 0x292ce828, a org.hsqldb.SessionMan
ager),
which is held by "Thread-2"
Java stack information for the threads listed above:
"Thread-2":
at org.hsqldb.Session.close(Unknown Source)
- waiting to lock <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- locked <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23b8efe8> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23b90f30> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2951f1c0> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
"Thread-1":
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- waiting to lock <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23ba9568> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2910d510> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
Found 1 deadlock.
C:\Users\Raghuveer.Rao>
Maven surefire plugin by default spawns a new process to run the tests. This is probably the process you are killing. When maven notices its child process was interrupted, it probably assummes that the tests failed.
Have a look at /target/surefire-reports for confirmation. What you can do to help us diagnose the problem is to find the surefire process id (the one you are killing) and run:
$ jstack ID
Stack dump will tell you what is causing the test to hung.