Spring amqp cachingConnectionFactory not shutting down for a minute - spring-amqp

Question

My client application is a Spring-shell application, running SpringBoot 2.1.1.
We are creating rabbitmq connection on demand CachingConnectionFacotry
CachingConnectionFactory connectionFactory = new CachingConnectionFactory(uri);
connectionFactory.setPublisherReturns(true);
When exiting the application, calling cachingConnectionFactory.destroy(); on #PreDestroy and/or onContextClosed events is not shutting down cachingConnectionFactory .
It is taking a minute for the application to exit.
Thread dump below indicates there is an executor inside cachingConnectionFactory that hasnt shutdown yet.
"spring-rabbit-deferred-pool-12" #29 prio=5 os_prio=31 tid=0x00007ff61245e000 nid=0x7503 waiting on condition [0x000070000fa57000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076c928300> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"spring-rabbit-deferred-pool-11" #28 prio=5 os_prio=31 tid=0x00007ff616461000 nid=0x7303 waiting on condition [0x000070000f954000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076c928300> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"process reaper" #14 daemon prio=10 os_prio=31 tid=0x00007ff614cd0800 nid=0x590b waiting on condition [0x000070000ec2d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c078c1e0> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Anything I can do to exit the application immediately?

Answer 1

I don't think it's related to that thread pool...
public final void destroy() {
super.destroy();
resetConnection();
if (getContextStopped()) {
this.stopped = true;
if (this.channelsExecutor != null) {
this.channelsExecutor.shutdownNow();
}
}
}
shutDownNow() is called after resetting the connection and, clearly, all of the threads are interruptible.
Best guess is something (perhaps network related) is delaying the resetConnection().
Post the complete thread dump while the delay is enountered, perhaps in a GitHub gist, or on pastebin, or similar.