Jenkins slaves hanging/Jenkins wedged - jenkins

We have an intermittent problem with slaves hanging AFTER the job itself is finished. In the post processing step (?) what we see is that the console log has this line:
Description set: vap_current_iter-2012_03_29_19_01_03
And then nothing. Usually, it will look like this:
Description set: prod_pull-2012_03_28_19_01_03
Notifying upstream build armada_Launch_prod_pull #13 of job completion
Project armada_Launch_prod_pull still waiting for 1 builds to complete
Notifying upstream projects of job completion
Notifying upstream of completion: armada_Launch_prod_pull #13
Finished: SUCCESS
I setup a logger for hudson.model.Run, and it currently has this :
at java.lang.Thread.run(Thread.java:619)
Mar 30, 2012 12:44:00 PM hudson.model.Run run
INFO: galleon_allUnit #1134 main build action completed: SUCCESS
Mar 30, 2012 12:44:00 PM hudson.model.Run setResult
FINE: galleon_allUnit #1134 : result is set to SUCCESS
java.lang.Exception
at hudson.model.Run.setResult(Run.java:352)
at hudson.model.Run.run(Run.java:1410)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:238)
Repeated for every hung slave.
The main hudson log doesn't have any additional information.
Disconnecting the slave has no effect.
Trying to do an orderly shutdown of Jenkins has no effect (jenkins actually appears to hang on shutdown).
The only way we have found to recover is to kill -9 the tomcat process.
The tread dump for one of the slaves (they are all the same) is:
Thread Dump
Channel reader thread: channel
"Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:199)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked java.io.BufferedInputStream#1ae615a
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1030)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Channel#e1d5ea
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Channel#e1d5ea
at java.lang.Object.wait(Object.java:485)
at hudson.remoting.Channel.join(Channel.java:766)
at hudson.remoting.Launcher.main(Launcher.java:420)
at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:366)
at hudson.remoting.Launcher.run(Launcher.java:206)
at hudson.remoting.Launcher.main(Launcher.java:168)
Ping thread for channel hudson.remoting.Channel#e1d5ea:channel
"Ping thread for channel hudson.remoting.Channel#e1d5ea:channel" Id=10 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
Pipe writer thread: channel
"Pipe writer thread: channel" Id=12 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#14263ed
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#14263ed
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
pool-1-thread-267
"pool-1-thread-267" Id=285 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374)
at hudson.Functions.getThreadInfos(Functions.java:872)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:93)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:89)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Number of locked synchronizers = 1
- java.util.concurrent.locks.ReentrantLock$NonfairSync#1186f88
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock#1798fdd
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock#1798fdd
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock#1d40442
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock#1d40442
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Signal Dispatcher
"Signal Dispatcher" Id=4 Group=system RUNNABLE
Any ideas on how to better recover or prevent this would be greatly appreciated.

We honestly just wrote a script that restarts jenkins every night at 4pm. We found that our breakages were happening at 3am give or take a half hour or so. Since restarting the server at this time, we haven't seen any further hangs. This is a way to prevent it as you asked, though it doesn't "Fix" the problem obviously!

Related

Spring amqp cachingConnectionFactory not shutting down for a minute

My client application is a Spring-shell application, running SpringBoot 2.1.1.
We are creating rabbitmq connection on demand CachingConnectionFacotry
CachingConnectionFactory connectionFactory = new CachingConnectionFactory(uri);
connectionFactory.setPublisherReturns(true);
When exiting the application, calling cachingConnectionFactory.destroy(); on #PreDestroy and/or onContextClosed events is not shutting down cachingConnectionFactory .
It is taking a minute for the application to exit.
Thread dump below indicates there is an executor inside cachingConnectionFactory that hasnt shutdown yet.
"spring-rabbit-deferred-pool-12" #29 prio=5 os_prio=31 tid=0x00007ff61245e000 nid=0x7503 waiting on condition [0x000070000fa57000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076c928300> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"spring-rabbit-deferred-pool-11" #28 prio=5 os_prio=31 tid=0x00007ff616461000 nid=0x7303 waiting on condition [0x000070000f954000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076c928300> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"process reaper" #14 daemon prio=10 os_prio=31 tid=0x00007ff614cd0800 nid=0x590b waiting on condition [0x000070000ec2d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c078c1e0> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Anything I can do to exit the application immediately?
I don't think it's related to that thread pool...
public final void destroy() {
super.destroy();
resetConnection();
if (getContextStopped()) {
this.stopped = true;
if (this.channelsExecutor != null) {
this.channelsExecutor.shutdownNow();
}
}
}
shutDownNow() is called after resetting the connection and, clearly, all of the threads are interruptible.
Best guess is something (perhaps network related) is delaying the resetConnection().
Post the complete thread dump while the delay is enountered, perhaps in a GitHub gist, or on pastebin, or similar.

spring boot app with amqp stuck in blocked mode

my spring boot app with amqp rabbitMQ hangs without errors in one of the environments. The same config works ok in another.
The thread dump shows,
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <e62a7fa> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doStart(SimpleMessageListenerContainer.java:815)
- locked <5d300728> (a java.lang.Object)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:550)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:346)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:149)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:112)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:874)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544)
- locked <1c4259b2> (a java.lang.Object)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
and then all the AMQP consumer threads are in BLOCKED state,
"AMQPConsumerThread_20" - Thread t#100
java.lang.Thread.State: BLOCKED
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.isActive(SimpleMessageListenerContainer.java:870)
- waiting to lock <5d300728> (a java.lang.Object) owned by "main" t#32
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1000(SimpleMessageListenerContainer.java:95)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1310)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:268)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
The SpringBoot app then waits with the last line on console,
2017-02-08T17:48:26.91-0500 [APP/0] OUT 2017-02-08 17:48:26 [AMQPConsumerThread_1] INFO o.s.a.r.c.CachingConnectionFactory -
Created new connection: SimpleConnection#64f05682
[delegate=amqp://b489be6b-b6e5-41b5-8f31-3be3acac4518#x.y.z.w:5672/daad99cf-fa24-40fe-b0d1-f9312fb583be, localPort= 57509]
EDIT
when I drop the consumer count down to 1, the BLOCKED state goes away and the main thread state is as follows,
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <79908042> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doStart(SimpleMessageListenerContainer.java:815)
- locked <4f3dcc9> (a java.lang.Object)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:550)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:346)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:149)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:112)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:874)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544)
- locked <1a60fd96> (a java.lang.Object)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
But the app still does not start.
the rabbitmq webconsole shows the consumer connected with the last line in logs,
2017-02-08T18:36:09.99-0500 [APP/0] OUT 2017-02-08 18:36:09 [AMQPConsumerThread_1] INFO o.s.a.r.c.CachingConnectionFactory - Created new connection: SimpleConnection#39271079
[delegate=amqp://b489be6b-b6e5-41b5-8f31-3be3acac4518#10.146.54.74:5672/daad99cf-fa24-40fe-b0d1-f9312fb583be, localPort= 32880]
what am I missing?
thanks
This...
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <79908042> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
Indicates the SimpleAsyncTaskExecutor is at its thread limit (setConcurrencyLimit). This check is only performed if you have set a limit.
This won't proceed until some task terminates. Since the consumer threads are long-lived, that will likely not happen.

How to prevent Jenkins from resuming an aborted job

I had aborted a job by clicking the [x] button next to it's name in the Build Executor Status panel of Jenkins, yet the job is still present and waiting.
Even when I restart Jenkins it will resume building. When I take a look at the console I see:
Sending interrupt signal to process Click here to forcibly terminate
running steps java.io.IOException: Aborting build at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source) at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source) at java.lang.reflect.Constructor.newInstance(Unknown Source)
at
org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
at
org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
at Script1.run(Script1.groovy:1) at
groovy.lang.GroovyShell.evaluate(GroovyShell.java:585) [SNIP]
Finished: ABORTED Resuming build at Tue Oct 25 14:56:36 CDT 2016 after
Jenkins restart
I believe this version of my Pipeline script also included an "Input" block, it's possible the job was aborted as it was waiting for Input if that makes a difference
I clean these up by going Manage Jenkins -> Script Console and then putting in code like this:
Jenkins.instance.getItemByFullName(
"tpr-ereg")
.getBuildByNumber(24).
finish(hudson.model.Result.ABORTED,
new java.io.IOException("Aborting build"));
NOTE: the above may need to be all on one line in the Script Console

Java threading issue on VPS

I have a VPS with Centos running with the following details:
[root#XXXXXXX~]# uname -a
Linux xxxxxxxx2.6.32-042stab055.10 #1 SMP Thu May 10 15:38:32 MSD 2012 i686 i686 i386 GNU/Linux
I am trying to run the ecommerce system shopizer from the tomcat installed on it.
I had tried to build it on the VPS but did not succeed so I built it else where and copied the war files on the VPS's tomcat.
The issue I am facing now and during the build too was that the build was hanging when i ran the built ant scripts. now when i am launching the tomcat server whose details are as follows :
Server version: Apache Tomcat/6.0.35
Server built: Nov 28 2011 11:20:06
Server number: 6.0.35.0
OS Name: Linux
OS Version: 2.6.32-042stab055.10
Architecture: i386
JVM Version: 1.6.0_24-b24
JVM Vendor: Sun Microsystems Inc.
the server hangs up at start and when I take thread dump (both in this case and the build case) I get the following :
"GC Daemon" daemon prio=10 tid=0xa09fd000 nid=0x36ef in Object.wait() [0xa08f7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa6c35a88> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:117)
- locked <0xa6c35a88> (a sun.misc.GC$LatencyLock)
"Low Memory Detector" daemon prio=10 tid=0xb7686000 nid=0x36ed runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread0" daemon prio=10 tid=0xb7684000 nid=0x36ec runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0xb7682800 nid=0x36eb waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0xb7673000 nid=0x36ea in Object.wait() [0xa0ffe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa6ad0b58> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0xa6ad0b58> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0xb7671800 nid=0x36e9 in Object.wait() [0xa1198000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0xb7605c00 nid=0x36e7 runnable [0xb775d000]
java.lang.Thread.State: RUNNABLE
at java.lang.Byte$ByteCache.<clinit>(Byte.java:79)
at java.lang.Byte.valueOf(Byte.java:102)
- waiting on <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0xa6ad0a58> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0xb7605c00 nid=0x36e7 runnable [0xb775d000]
java.lang.Thread.State: RUNNABLE
at java.lang.Byte$ByteCache.<clinit>(Byte.java:79)
at java.lang.Byte.valueOf(Byte.java:102)
at com.opensymphony.xwork2.conversion.impl.DefaultTypeConverter.<init>(DefaultTypeConverter.java:59)
at com.opensymphony.xwork2.conversion.impl.XWorkConverter.<init>(XWorkConverter.java:186)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:419)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerImpl$ParameterInjector.inject(ContainerImpl.java:462)
at com.opensymphony.xwork2.inject.ContainerImpl.getParameters(ContainerImpl.java:477)
at com.opensymphony.xwork2.inject.ContainerImpl.access$000(ContainerImpl.java:34)
at com.opensymphony.xwork2.inject.ContainerImpl$MethodInjector.inject(ContainerImpl.java:293)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:431)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerImpl$ParameterInjector.inject(ContainerImpl.java:462)
at com.opensymphony.xwork2.inject.ContainerImpl.getParameters(ContainerImpl.java:477)
at com.opensymphony.xwork2.inject.ContainerImpl.access$000(ContainerImpl.java:34)
at com.opensymphony.xwork2.inject.ContainerImpl$MethodInjector.inject(ContainerImpl.java:293)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:431)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerImpl$ParameterInjector.inject(ContainerImpl.java:462)
at com.opensymphony.xwork2.inject.ContainerImpl.getParameters(ContainerImpl.java:477)
at com.opensymphony.xwork2.inject.ContainerImpl.access$000(ContainerImpl.java:34)
at com.opensymphony.xwork2.inject.ContainerImpl$MethodInjector.inject(ContainerImpl.java:293)
at com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector.construct(ContainerImpl.java:431)
at com.opensymphony.xwork2.inject.ContainerBuilder$5.create(ContainerBuilder.java:207)
at com.opensymphony.xwork2.inject.Scope$2$1.create(Scope.java:51)
- locked <0xa18cf408> (a com.opensymphony.xwork2.inject.ContainerImpl)
at com.opensymphony.xwork2.inject.ContainerBuilder$3.create(ContainerBuilder.java:93)
at com.opensymphony.xwork2.inject.ContainerBuilder$7.call(ContainerBuilder.java:487)
at com.opensymphony.xwork2.inject.ContainerBuilder$7.call(ContainerBuilder.java:484)
at com.opensymphony.xwork2.inject.ContainerImpl.callInContext(ContainerImpl.java:574)
at com.opensymphony.xwork2.inject.ContainerBuilder.create(ContainerBuilder.java:484)
at com.opensymphony.xwork2.config.impl.DefaultConfiguration.createBootstrapContainer(DefaultConfiguration.java:252)
at com.opensymphony.xwork2.config.impl.DefaultConfiguration.reloadContainer(DefaultConfiguration.java:193)
- locked <0xa37c05b0> (a com.opensymphony.xwork2.config.impl.DefaultConfiguration)
....
....
..
"VM Thread" prio=10 tid=0xb766d800 nid=0x36e8 runnable
"VM Periodic Task Thread" prio=10 tid=0xb7688400 nid=0x36ee waiting on condition
JNI global references: 911
Heap
def new generation total 39424K, used 5740K [0xa1580000, 0xa4040000, 0xa6ad0000)
eden space 35072K, 12% used [0xa1580000, 0xa19a8e88, 0xa37c0000)
from space 4352K, 34% used [0xa37c0000, 0xa3932570, 0xa3c00000)
to space 4352K, 0% used [0xa3c00000, 0xa3c00000, 0xa4040000)
tenured generation total 87424K, used 1432K [0xa6ad0000, 0xac030000, 0xb1580000)
the space 87424K, 1% used [0xa6ad0000, 0xa6c36038, 0xa6c36200, 0xac030000)
compacting perm gen total 12288K, used 11825K [0xb1580000, 0xb2180000, 0xb5580000)
the space 12288K, 96% used [0xb1580000, 0xb210c588, 0xb210c600, 0xb2180000)
No shared spaces configured.
What can I do to resolve the issue or is it a bug/issue with java running on VPS??
B) the jvm sometimes crashes on the same VPS with the following error :
[root#xxxxxx ~]# java
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
[root#xxxxxx ~]# free
total used free shared buffers cached
Mem: 1155072 561320 593752 0 0 317124
-/+ buffers/cache: 244196 910876
Swap: 0 0 0
Very strange to me, can anyone explain me this behaviour.
I suggest that you try to update your operating system and your JVM.
I think this crash is due to a bug in your actual configuration.

Maven test does not complete

When i run a
mvn test
Maven runs all the tests successfully and then goes on a wait infinite loop
I see a java process in the task manager and I have to kill it before maven returns with this error message
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:33.949s
[INFO] Finished at: Fri Jun 17 14:50:01 EDT 2011
[INFO] Final Memory: 5M/247M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.
4:test (default-test) on project EDIFileHandling: There are test failures.
[ERROR]
[ERROR] Please refer to c:\code\target\surefire-reports for the individual test
results.
Any help appreciated.
Thanks
R
C:\Users\Raghuveer.Rao>jstack 7256
2011-06-17 15:24:55
Full thread dump Java HotSpot(TM) Client VM (20.0-b11 mixed mode, sharing):
"Thread-2" prio=6 tid=0x03ed3000 nid=0x14d0 waiting for monitor entry [0x04ddf00
0]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.hsqldb.Session.close(Unknown Source)
- waiting to lock <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- locked <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23b8efe8> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23b90f30> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2951f1c0> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
"Thread-1" prio=6 tid=0x03ed2c00 nid=0x1fe0 waiting for monitor entry [0x04d8f00
0]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- waiting to lock <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23ba9568> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2910d510> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
"Low Memory Detector" daemon prio=6 tid=0x01bdac00 nid=0x1eb0 runnable [0x000000
00]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread0" daemon prio=10 tid=0x01bc8400 nid=0x18e8 waiting on conditi
on [0x00000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" daemon prio=10 tid=0x01bc5400 nid=0x1ac0 waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x01bc2400 nid=0x1fbc runnable [0x0000000
0]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=0x01bbb800 nid=0x1c00 in Object.wait() [0x03cdf000
]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x290702b0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x290702b0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x01bba000 nid=0x810 in Object.wait() [0x
03c8f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x29070340> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x29070340> (a java.lang.ref.Reference$Lock)
"main" prio=6 tid=0x00e29800 nid=0x1e20 in Object.wait() [0x001bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x294c4300> (a org.springframework.context.support.Abstrac
tApplicationContext$3)
at java.lang.Thread.join(Thread.java:1186)
- locked <0x294c4300> (a org.springframework.context.support.AbstractApp
licationContext$3)
at java.lang.Thread.join(Thread.java:1239)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.
java:79)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.jav
a:24)
at java.lang.Shutdown.runHooks(Shutdown.java:79)
at java.lang.Shutdown.sequence(Shutdown.java:123)
at java.lang.Shutdown.exit(Shutdown.java:168)
- locked <0x38b53ef8> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Runtime.exit(Runtime.java:90)
at java.lang.System.exit(System.java:904)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.j
ava:986)
"VM Thread" prio=10 tid=0x01b7e400 nid=0x1e10 runnable
"VM Periodic Task Thread" prio=10 tid=0x01bec800 nid=0x1bac waiting on condition
JNI global references: 1776
Found one Java-level deadlock:
"Thread-2":
waiting to lock monitor 0x0467c1cc (object 0x23ba76f0, a org.hsqldb.Session),
which is held by "Thread-1"
"Thread-1":
waiting to lock monitor 0x01bbf56c (object 0x292ce828, a org.hsqldb.SessionMan
ager),
which is held by "Thread-2"
Java stack information for the threads listed above:
"Thread-2":
at org.hsqldb.Session.close(Unknown Source)
- waiting to lock <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- locked <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23b8efe8> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23b90f30> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2951f1c0> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
"Thread-1":
at org.hsqldb.SessionManager.closeAllSessions(Unknown Source)
- waiting to lock <0x292ce828> (a org.hsqldb.SessionManager)
at org.hsqldb.Database.close(Unknown Source)
at org.hsqldb.StatementCommand.getResult(Unknown Source)
at org.hsqldb.StatementCommand.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.executeDirectStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
- locked <0x23ba76f0> (a org.hsqldb.Session)
at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source)
- locked <0x23ba9568> (a org.hsqldb.jdbc.JDBCStatement)
at org.springframework.jdbc.datasource.embedded.AbstractEmbeddedDatabase
Configurer.shutdown(AbstractEmbeddedDatabaseConfigurer.java:42)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory.
shutdownDatabase(EmbeddedDatabaseFactory.java:184)
at org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactoryB
ean.destroy(EmbeddedDatabaseFactoryBean.java:57)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destr
oy(DisposableBeanAdapter.java:184)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x2910d510> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destro
yBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClos
e(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext$3.run(
AbstractApplicationContext.java:940)
Found 1 deadlock.
C:\Users\Raghuveer.Rao>
Maven surefire plugin by default spawns a new process to run the tests. This is probably the process you are killing. When maven notices its child process was interrupted, it probably assummes that the tests failed.
Have a look at /target/surefire-reports for confirmation. What you can do to help us diagnose the problem is to find the surefire process id (the one you are killing) and run:
$ jstack ID
Stack dump will tell you what is causing the test to hung.

Resources