I have a java process loading a lot of data from a bunch of .csv files into a Neo4j database using the BatchInserter. I was using:
OpenJDK 7
Ubuntu 12.04
Neo4j 2.0 M3
After loading the first 164 GB (according to ls -lh) the folder size stopped increasing but the process kept running, no memory was released, and CPU was still at 100% (all according to htop).
The loading process is single threaded, only the JVM is using more than 1 thread - I guess by the ParallelGC.
I'm not sure how to diagnose this type of problem but was instructed to try jstack, so have included its output below.
Anyone has an idea what went wrong or has suggestions on how I can go about diagnosing this?
Full thread dump OpenJDK 64-Bit Server VM (22.0-b10 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00007fc3a4001000 nid=0x5636 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" daemon prio=10 tid=0x00007fcf58123000 nid=0x4545 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007fcf58120800 nid=0x4544 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007fcf5811d800 nid=0x4543 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007fcf5811b800 nid=0x4542 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007fcf580c4800 nid=0x4541 in Object.wait() [0x00007fc3f3cfb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fc4165fc708> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00007fc4165fc708> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0x00007fcf580c2000 nid=0x4540 in Object.wait() [0x00007fc3f3dfc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fc4165ffe08> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00007fc4165ffe08> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007fcf58007800 nid=0x452c runnable [0x00007fcf606b7000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.substring(String.java:1960)
at java.lang.String.subSequence(String.java:1993)
at java.util.regex.Pattern.split(Pattern.java:1202)
at com.ldbc.socialnet.neo4j.CsvFileReader.parseLine(CsvFileReader.java:73)
at com.ldbc.socialnet.neo4j.CsvFileReader.nextLine(CsvFileReader.java:61)
at com.ldbc.socialnet.neo4j.CsvFileReader.hasNext(CsvFileReader.java:33)
at com.ldbc.socialnet.neo4j.CsvFileInserter.bufferLines(CsvFileInserter.java:62)
at com.ldbc.socialnet.neo4j.CsvFileInserter.insertAllBuffered(CsvFileInserter.java:93)
at com.ldbc.socialnet.neo4j.LdbcSocialNeworkNeo4jImporter.load(LdbcSocialNeworkNeo4jImporter.java:79)
at com.ldbc.socialnet.neo4j.LdbcSocialNeworkNeo4jImporter.main(LdbcSocialNeworkNeo4jImporter.java:49)
"VM Thread" prio=10 tid=0x00007fcf580ba000 nid=0x453f runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fcf58012800 nid=0x452d runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fcf58014800 nid=0x452e runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fcf58016000 nid=0x452f runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fcf58018000 nid=0x4530 runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007fcf5801a000 nid=0x4531 runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007fcf5801b800 nid=0x4532 runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x00007fcf5801d800 nid=0x4533 runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x00007fcf5801f800 nid=0x4534 runnable
"GC task thread#8 (ParallelGC)" prio=10 tid=0x00007fcf58021000 nid=0x4535 runnable
"GC task thread#9 (ParallelGC)" prio=10 tid=0x00007fcf58023000 nid=0x4536 runnable
"GC task thread#10 (ParallelGC)" prio=10 tid=0x00007fcf58025000 nid=0x4537 runnable
"GC task thread#11 (ParallelGC)" prio=10 tid=0x00007fcf58026800 nid=0x4538 runnable
"GC task thread#12 (ParallelGC)" prio=10 tid=0x00007fcf58028800 nid=0x4539 runnable
"GC task thread#13 (ParallelGC)" prio=10 tid=0x00007fcf5802a800 nid=0x453a runnable
"GC task thread#14 (ParallelGC)" prio=10 tid=0x00007fcf5802c800 nid=0x453b runnable
"GC task thread#15 (ParallelGC)" prio=10 tid=0x00007fcf5802e000 nid=0x453c runnable
"GC task thread#16 (ParallelGC)" prio=10 tid=0x00007fcf58030000 nid=0x453d runnable
"GC task thread#17 (ParallelGC)" prio=10 tid=0x00007fcf58032000 nid=0x453e runnable
"VM Periodic Task Thread" prio=10 tid=0x00007fcf5812d800 nid=0x4546 waiting on condition
JNI global references: 175
Heap
PSYoungGen total 11211840K, used 7045440K [0x00007fcb95000000, 0x00007fcf3d160000, 0x00007fcf55000000)
eden space 7045440K, 100% used [0x00007fcb95000000,0x00007fcd43050000,0x00007fcd43050000)
from space 4166400K, 0% used [0x00007fce39510000,0x00007fce39510000,0x00007fcf379d0000)
to space 4035328K, 0% used [0x00007fcd43050000,0x00007fcd43050000,0x00007fce39510000)
PSOldGen total 31457280K, used 31300002K [0x00007fc415000000, 0x00007fcb95000000, 0x00007fcb95000000)
object space 31457280K, 99% used [0x00007fc415000000,0x00007fcb8b668b98,0x00007fcb95000000)
PSPermGen total 21248K, used 7487K [0x00007fc40aa00000, 0x00007fc40bec0000, 0x00007fc415000000)
object space 21248K, 35% used [0x00007fc40aa00000,0x00007fc40b14ff48,0x00007fc40bec0000)
In particular, can anyone give a little insight on these heap values or suggest a good read about them?
Heap
PSYoungGen total 16348096K, used 12660905K [0x00007f833a560000, 0x00007f87639c0000, 0x00007f8765000000)
eden space 15282432K, 82% used [0x00007f833a560000,0x00007f863f18a688,0x00007f86df1a0000)
from space 1065664K, 0% used [0x00007f86df1a0000,0x00007f86df1a0000,0x00007f8720250000)
to space 1036288K, 0% used [0x00007f87245c0000,0x00007f87245c0000,0x00007f87639c0000)
ParOldGen total 34952576K, used 34903343K [0x00007f7ae5000000, 0x00007f833a560000, 0x00007f833a560000)
object space 34952576K, 99% used [0x00007f7ae5000000,0x00007f833754bd90,0x00007f833a560000)
PSPermGen total 21248K, used 7130K [0x00007f7adfe00000, 0x00007f7ae12c0000, 0x00007f7ae5000000)
object space 21248K, 33% used [0x00007f7adfe00000,0x00007f7ae04f6860,0x00007f7ae12c0000)
It looks like your process is almost out of memory:
eden space 7045440K, 100% used
object space 31457280K, 99% used
In this case GC may run continiosly, trying to free some memory, consuming alomist all CPU. So application logic thread is under heavy starvation. You may either add more memory via -Xmx JVM option or profile the application. Profiling via JVisualVM, Jprofiler or other tool will give you important runtime data:
Memory allocaton dynamics. If memory consumption is increasing all the time you're likely to have a memory leak
Heap contents to find out what occupies the momory
GC statistics
Based on this data you'll certanly be able to resolve the issue.
Related
My client application is a Spring-shell application, running SpringBoot 2.1.1.
We are creating rabbitmq connection on demand CachingConnectionFacotry
CachingConnectionFactory connectionFactory = new CachingConnectionFactory(uri);
connectionFactory.setPublisherReturns(true);
When exiting the application, calling cachingConnectionFactory.destroy(); on #PreDestroy and/or onContextClosed events is not shutting down cachingConnectionFactory .
It is taking a minute for the application to exit.
Thread dump below indicates there is an executor inside cachingConnectionFactory that hasnt shutdown yet.
"spring-rabbit-deferred-pool-12" #29 prio=5 os_prio=31 tid=0x00007ff61245e000 nid=0x7503 waiting on condition [0x000070000fa57000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076c928300> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"spring-rabbit-deferred-pool-11" #28 prio=5 os_prio=31 tid=0x00007ff616461000 nid=0x7303 waiting on condition [0x000070000f954000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076c928300> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"process reaper" #14 daemon prio=10 os_prio=31 tid=0x00007ff614cd0800 nid=0x590b waiting on condition [0x000070000ec2d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c078c1e0> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Anything I can do to exit the application immediately?
I don't think it's related to that thread pool...
public final void destroy() {
super.destroy();
resetConnection();
if (getContextStopped()) {
this.stopped = true;
if (this.channelsExecutor != null) {
this.channelsExecutor.shutdownNow();
}
}
}
shutDownNow() is called after resetting the connection and, clearly, all of the threads are interruptible.
Best guess is something (perhaps network related) is delaying the resetConnection().
Post the complete thread dump while the delay is enountered, perhaps in a GitHub gist, or on pastebin, or similar.
my spring boot app with amqp rabbitMQ hangs without errors in one of the environments. The same config works ok in another.
The thread dump shows,
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <e62a7fa> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doStart(SimpleMessageListenerContainer.java:815)
- locked <5d300728> (a java.lang.Object)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:550)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:346)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:149)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:112)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:874)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544)
- locked <1c4259b2> (a java.lang.Object)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
and then all the AMQP consumer threads are in BLOCKED state,
"AMQPConsumerThread_20" - Thread t#100
java.lang.Thread.State: BLOCKED
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.isActive(SimpleMessageListenerContainer.java:870)
- waiting to lock <5d300728> (a java.lang.Object) owned by "main" t#32
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1000(SimpleMessageListenerContainer.java:95)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1310)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:268)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
The SpringBoot app then waits with the last line on console,
2017-02-08T17:48:26.91-0500 [APP/0] OUT 2017-02-08 17:48:26 [AMQPConsumerThread_1] INFO o.s.a.r.c.CachingConnectionFactory -
Created new connection: SimpleConnection#64f05682
[delegate=amqp://b489be6b-b6e5-41b5-8f31-3be3acac4518#x.y.z.w:5672/daad99cf-fa24-40fe-b0d1-f9312fb583be, localPort= 57509]
EDIT
when I drop the consumer count down to 1, the BLOCKED state goes away and the main thread state is as follows,
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <79908042> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doStart(SimpleMessageListenerContainer.java:815)
- locked <4f3dcc9> (a java.lang.Object)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:550)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:346)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:149)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:112)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:874)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544)
- locked <1a60fd96> (a java.lang.Object)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
But the app still does not start.
the rabbitmq webconsole shows the consumer connected with the last line in logs,
2017-02-08T18:36:09.99-0500 [APP/0] OUT 2017-02-08 18:36:09 [AMQPConsumerThread_1] INFO o.s.a.r.c.CachingConnectionFactory - Created new connection: SimpleConnection#39271079
[delegate=amqp://b489be6b-b6e5-41b5-8f31-3be3acac4518#10.146.54.74:5672/daad99cf-fa24-40fe-b0d1-f9312fb583be, localPort= 32880]
what am I missing?
thanks
This...
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <79908042> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
Indicates the SimpleAsyncTaskExecutor is at its thread limit (setConcurrencyLimit). This check is only performed if you have set a limit.
This won't proceed until some task terminates. Since the consumer threads are long-lived, that will likely not happen.
I have a VPS with Centos running with the following details:
[root#XXXXXXX~]# uname -a
Linux xxxxxxxx2.6.32-042stab055.10 #1 SMP Thu May 10 15:38:32 MSD 2012 i686 i686 i386 GNU/Linux
I am trying to run the ecommerce system shopizer from the tomcat installed on it.
I had tried to build it on the VPS but did not succeed so I built it else where and copied the war files on the VPS's tomcat.
The issue I am facing now and during the build too was that the build was hanging when i ran the built ant scripts. now when i am launching the tomcat server whose details are as follows :
Server version: Apache Tomcat/6.0.35
Server built: Nov 28 2011 11:20:06
Server number: 6.0.35.0
OS Name: Linux
OS Version: 2.6.32-042stab055.10
Architecture: i386
JVM Version: 1.6.0_24-b24
JVM Vendor: Sun Microsystems Inc.
the server hangs up at start and when I take thread dump (both in this case and the build case) I get the following :
"GC Daemon" daemon prio=10 tid=0xa09fd000 nid=0x36ef in Object.wait() [0xa08f7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa6c35a88> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:117)
- locked <0xa6c35a88> (a sun.misc.GC$LatencyLock)
"Low Memory Detector" daemon prio=10 tid=0xb7686000 nid=0x36ed runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread0" daemon prio=10 tid=0xb7684000 nid=0x36ec runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0xb7682800 nid=0x36eb waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0xb7673000 nid=0x36ea in Object.wait() [0xa0ffe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa6ad0b58> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0xa6ad0b58> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0xb7671800 nid=0x36e9 in Object.wait() [0xa1198000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0xb7605c00 nid=0x36e7 runnable [0xb775d000]
java.lang.Thread.State: RUNNABLE
at java.lang.Byte$ByteCache.<clinit>(Byte.java:79)
at java.lang.Byte.valueOf(Byte.java:102)
- waiting on <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0xb7605c00 nid=0x36e7 runnable [0xb775d000]
java.lang.Thread.State: RUNNABLE
at java.lang.Byte$ByteCache.<clinit>(Byte.java:79)
at java.lang.Byte.valueOf(Byte.java:102)
at com.opensymphony.xwork2.conversion.impl.DefaultTypeConverter.<init>(DefaultTypeConverter.java:59)
at com.opensymphony.xwork2.conversion.impl.XWorkConverter.<init>(XWorkConverter.java:186)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:419)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerImpl$ParameterInjector.inject(ContainerImpl.java:462)
at com.opensymphony.xwork2.inject.ContainerImpl.getParameters(ContainerImpl.java:477)
at com.opensymphony.xwork2.inject.ContainerImpl.access$000(ContainerImpl.java:34)
at com.opensymphony.xwork2.inject.ContainerImpl$MethodInjector.inject(ContainerImpl.java:293)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:431)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerImpl$ParameterInjector.inject(ContainerImpl.java:462)
at com.opensymphony.xwork2.inject.ContainerImpl.getParameters(ContainerImpl.java:477)
at com.opensymphony.xwork2.inject.ContainerImpl.access$000(ContainerImpl.java:34)
at com.opensymphony.xwork2.inject.ContainerImpl$MethodInjector.inject(ContainerImpl.java:293)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:431)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerImpl$ParameterInjector.inject(ContainerImpl.java:462)
at com.opensymphony.xwork2.inject.ContainerImpl.getParameters(ContainerImpl.java:477)
at com.opensymphony.xwork2.inject.ContainerImpl.access$000(ContainerImpl.java:34)
at com.opensymphony.xwork2.inject.ContainerImpl$MethodInjector.inject(ContainerImpl.java:293)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:431)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerBuilder$3.create(ContainerBuilder.java:93)
at com.opensymphony.xwork2.inject.ContainerBuilder$7.call(ContainerBuilder.java:487)
at com.opensymphony.xwork2.inject.ContainerBuilder$7.call(ContainerBuilder.java:484)
at com.opensymphony.xwork2.inject.ContainerImpl.callInContext(ContainerImpl.java:574)
at com.opensymphony.xwork2.inject.ContainerBuilder.create(ContainerBuilder.java:484)
at com.opensymphony.xwork2.config.impl.DefaultConfiguration.createBootstrapContainer(DefaultConfiguration.java:252)
at com.opensymphony.xwork2.config.impl.DefaultConfiguration.reloadContainer(DefaultConfiguration.java:193)
- locked <0xa37c05b0> (a com.opensymphony.xwork2.config.impl.DefaultConfiguration)
....
....
..
"VM Thread" prio=10 tid=0xb766d800 nid=0x36e8 runnable
"VM Periodic Task Thread" prio=10 tid=0xb7688400 nid=0x36ee waiting on condition
JNI global references: 911
Heap
def new generation total 39424K, used 5740K [0xa1580000, 0xa4040000, 0xa6ad0000)
eden space 35072K, 12% used [0xa1580000, 0xa19a8e88, 0xa37c0000)
from space 4352K, 34% used [0xa37c0000, 0xa3932570, 0xa3c00000)
to space 4352K, 0% used [0xa3c00000, 0xa3c00000, 0xa4040000)
tenured generation total 87424K, used 1432K [0xa6ad0000, 0xac030000, 0xb1580000)
the space 87424K, 1% used [0xa6ad0000, 0xa6c36038, 0xa6c36200, 0xac030000)
compacting perm gen total 12288K, used 11825K [0xb1580000, 0xb2180000, 0xb5580000)
the space 12288K, 96% used [0xb1580000, 0xb210c588, 0xb210c600, 0xb2180000)
No shared spaces configured.
What can I do to resolve the issue or is it a bug/issue with java running on VPS??
B) the jvm sometimes crashes on the same VPS with the following error :
[root#xxxxxx ~]# java
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
[root#xxxxxx ~]# free
total used free shared buffers cached
Mem: 1155072 561320 593752 0 0 317124
-/+ buffers/cache: 244196 910876
Swap: 0 0 0
Very strange to me, can anyone explain me this behaviour.
I suggest that you try to update your operating system and your JVM.
I think this crash is due to a bug in your actual configuration.
We have an intermittent problem with slaves hanging AFTER the job itself is finished. In the post processing step (?) what we see is that the console log has this line:
Description set: vap_current_iter-2012_03_29_19_01_03
And then nothing. Usually, it will look like this:
Description set: prod_pull-2012_03_28_19_01_03
Notifying upstream build armada_Launch_prod_pull #13 of job completion
Project armada_Launch_prod_pull still waiting for 1 builds to complete
Notifying upstream projects of job completion
Notifying upstream of completion: armada_Launch_prod_pull #13
Finished: SUCCESS
I setup a logger for hudson.model.Run, and it currently has this :
at java.lang.Thread.run(Thread.java:619)
Mar 30, 2012 12:44:00 PM hudson.model.Run run
INFO: galleon_allUnit #1134 main build action completed: SUCCESS
Mar 30, 2012 12:44:00 PM hudson.model.Run setResult
FINE: galleon_allUnit #1134 : result is set to SUCCESS
java.lang.Exception
at hudson.model.Run.setResult(Run.java:352)
at hudson.model.Run.run(Run.java:1410)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:238)
Repeated for every hung slave.
The main hudson log doesn't have any additional information.
Disconnecting the slave has no effect.
Trying to do an orderly shutdown of Jenkins has no effect (jenkins actually appears to hang on shutdown).
The only way we have found to recover is to kill -9 the tomcat process.
The tread dump for one of the slaves (they are all the same) is:
Thread Dump
Channel reader thread: channel
"Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:199)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked java.io.BufferedInputStream#1ae615a
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1030)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Channel#e1d5ea
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Channel#e1d5ea
at java.lang.Object.wait(Object.java:485)
at hudson.remoting.Channel.join(Channel.java:766)
at hudson.remoting.Launcher.main(Launcher.java:420)
at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:366)
at hudson.remoting.Launcher.run(Launcher.java:206)
at hudson.remoting.Launcher.main(Launcher.java:168)
Ping thread for channel hudson.remoting.Channel#e1d5ea:channel
"Ping thread for channel hudson.remoting.Channel#e1d5ea:channel" Id=10 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
Pipe writer thread: channel
"Pipe writer thread: channel" Id=12 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#14263ed
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#14263ed
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
pool-1-thread-267
"pool-1-thread-267" Id=285 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374)
at hudson.Functions.getThreadInfos(Functions.java:872)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:93)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:89)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Number of locked synchronizers = 1
- java.util.concurrent.locks.ReentrantLock$NonfairSync#1186f88
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock#1798fdd
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock#1798fdd
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock#1d40442
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock#1d40442
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Signal Dispatcher
"Signal Dispatcher" Id=4 Group=system RUNNABLE
Any ideas on how to better recover or prevent this would be greatly appreciated.
We honestly just wrote a script that restarts jenkins every night at 4pm. We found that our breakages were happening at 3am give or take a half hour or so. Since restarting the server at this time, we haven't seen any further hangs. This is a way to prevent it as you asked, though it doesn't "Fix" the problem obviously!
When i run a
mvn test
Maven runs all the tests successfully and then goes on a wait infinite loop
I see a java process in the task manager and I have to kill it before maven returns with this error message
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:33.949s
[INFO] Finished at: Fri Jun 17 14:50:01 EDT 2011
[INFO] Final Memory: 5M/247M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.
4:test (default-test) on project EDIFileHandling: There are test failures.
[ERROR]
[ERROR] Please refer to c:\code\target\surefire-reports for the individual test
results.
Any help appreciated.
Thanks
R
C:\Users\Raghuveer.Rao>jstack 7256
2011-06-17 15:24:55
Full thread dump Java HotSpot(TM) Client VM (20.0-b11 mixed mode, sharing):
"Thread-2" prio=6 tid=0x03ed3000 nid=0x14d0 waiting for monitor entry [0x04ddf00
0]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.hsqldb.Session.close(Unknown Source)
- waiting to lock <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- locked <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23b8efe8> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23b90f30> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2951f1c0> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
"Thread-1" prio=6 tid=0x03ed2c00 nid=0x1fe0 waiting for monitor entry [0x04d8f00
0]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- waiting to lock <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23ba9568> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2910d510> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
"Low Memory Detector" daemon prio=6 tid=0x01bdac00 nid=0x1eb0 runnable [0x000000
00]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread0" daemon prio=10 tid=0x01bc8400 nid=0x18e8 waiting on conditi
on [0x00000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" daemon prio=10 tid=0x01bc5400 nid=0x1ac0 waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x01bc2400 nid=0x1fbc runnable [0x0000000
0]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=0x01bbb800 nid=0x1c00 in Object.wait() [0x03cdf000
]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x290702b0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x290702b0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x01bba000 nid=0x810 in Object.wait() [0x
03c8f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x29070340> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x29070340> (a java.lang.ref.Reference$Lock)
"main" prio=6 tid=0x00e29800 nid=0x1e20 in Object.wait() [0x001bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x294c4300> (a org.springframework.context.support.Abstrac
tApplicationContext$3)
at java.lang.Thread.join(Thread.java:1186)
- locked <0x294c4300> (a org.springframework.context.support.AbstractApp
licationContext$3)
at java.lang.Thread.join(Thread.java:1239)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.
java:79)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.jav
a:24)
at java.lang.Shutdown.runHooks(Shutdown.java:79)
at java.lang.Shutdown.sequence(Shutdown.java:123)
at java.lang.Shutdown.exit(Shutdown.java:168)
- locked <0x38b53ef8> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Runtime.exit(Runtime.java:90)
at java.lang.System.exit(System.java:904)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.j
ava:986)
"VM Thread" prio=10 tid=0x01b7e400 nid=0x1e10 runnable
"VM Periodic Task Thread" prio=10 tid=0x01bec800 nid=0x1bac waiting on condition
JNI global references: 1776
Found one Java-level deadlock:
"Thread-2":
waiting to lock monitor 0x0467c1cc (object 0x23ba76f0, a org.hsqldb.Session),
which is held by "Thread-1"
"Thread-1":
waiting to lock monitor 0x01bbf56c (object 0x292ce828, a org.hsqldb.SessionMan
ager),
which is held by "Thread-2"
Java stack information for the threads listed above:
"Thread-2":
at org.hsqldb.Session.close(Unknown Source)
- waiting to lock <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- locked <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23b8efe8> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23b90f30> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2951f1c0> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
"Thread-1":
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- waiting to lock <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23ba9568> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2910d510> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
Found 1 deadlock.
C:\Users\Raghuveer.Rao>
Maven surefire plugin by default spawns a new process to run the tests. This is probably the process you are killing. When maven notices its child process was interrupted, it probably assummes that the tests failed.
Have a look at /target/surefire-reports for confirmation. What you can do to help us diagnose the problem is to find the surefire process id (the one you are killing) and run:
$ jstack ID
Stack dump will tell you what is causing the test to hung.