Jenkins jobs hang indefinitely on maven goal - docker

Question

Summary: A month or two ago we started noticing our Jenkins builds stalling indefinitely. When a build spins up, our Jenkins Shared Library creates a new docker container on a virtual machine and executes the library code to build our service, usually including a maven build. We sporadically see a build or two stall indefinitely on one of various Maven goals (and only maven goals, as seen below). Sometimes this is happens on single build on a virtual machine, but it is most egregious when we run multiple builds at the same time. Some will build, others will stall. All builds on the same virtual machine share an .m2 repository so we don't download dependencies every time.
We are using maven 3.8.1 running in an ubuntu 18.04 container. We're looking for possible leads as we've exhausted all of ours.
We tried upgrading to maven 3.8.2 with no luck, we tried looking into this similar issue but its solution did not apply to us: Jenkins docker container simply hangs and never executes steps
We've also tried having each node have its own maven repository and that didn't work, so we ruled out read/write conflicts.
Details:
Some of the places it stalls:
Mvn help:evaluate after "Picking up Java_Tools_Options...",
Mvn set:property,
On checking on updates from virtual,
On actually downloading from artifactory.
These are a couple example of where it stalls:
[2022-02-22T17:39:15.452Z] + mvn versions:set-property -Dproperty=my.version -DnewVersion=4.0.1 -DgenerateBackupPoms=false -X --debug
[2022-02-22T17:39:15.452Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400
And
[2022-03-03T13:27:01.046Z] + mvn clean install -P snapshot -Dmaven.test.skip=true -Dbuild.number=32 -U
[2022-03-03T13:27:01.046Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400
[2022-03-03T13:27:02.425Z] [INFO] Scanning for projects...
[2022-03-03T13:27:04.913Z] [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) # sample-child-java ---
[2022-03-03T13:27:05.481Z] Downloading from virtual: https://my.artifactory/org/codehaus/plexus/plexus-java/0.9.10/plexus-java-0.9.10.pom
When we turn on debug mode for maven, we only get a little extra information before it stalls:
02-22T17:39:20.880Z] [DEBUG] Reading global settings from /opt/maven/conf/settings.xml
[2022-02-22T17:39:20.880Z] [DEBUG] Reading user settings from /home/myUser/.m2/settings.xml
[2022-02-22T17:39:20.880Z] [DEBUG] Reading global toolchains from /opt/maven/conf/toolchains.xml
[2022-02-22T17:39:20.880Z] [DEBUG] Reading user toolchains from /home/myUser/.m2/toolchains.xml
[2022-02-22T17:39:20.880Z] [DEBUG] Using local repository at /home/myUser/.m2/repository
[2022-02-22T17:39:21.144Z] [DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10.0 for /home/myUser/.m2/repository
After about 6 hours of stalling, the logs will usually show one of these statements:
Cannot contact myVirtualMachine6: Remote call on myVirtualMachine6 failed. The channel is closing down or has closed down
Wrapper script does not seem to be touching the log file in /home/jenkins/workspace/myService#tmp/durable-7db8f58b
(JENKINS-48300: if on an extremely laggy filesystem, consider - HEARBEAT_CHECK_INTERVAL=86400)
We tried using the withMaven() from the Maven Integration Plugin, but even that did not solve our problem:
11:14:07 + mvn versions:set-property -Dproperty=myService.version -DnewVersion=9.9.9-RC1 -DgenerateBackupPoms=false
11:14:07 Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/jenkins/workspace/myService#tmp/withMaven368661c2/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/jenkins/workspace/myService#tmp/withMaven368661c2"
11:14:10 [INFO] [jenkins-event-spy] Generate /home/jenkins/workspace/myService#tmp/withMaven368661c2/maven-spy-20220308-161411-4457772303541115170084.log.tmp ...
When we add a tee command to our withMaven() maven commands, the output where it stalls is just:
[INFO] [jenkins-event-spy] Generated /home/jenkins/workspace/myService#tmp/withMavena1de258e/maven-spy-20220308-192941-7666252427488201541499.log
[INFO] [jenkins-event-spy] Generate /home/jenkins/workspace/myService#tmp/withMavena1de258e/maven-spy-20220308-193003-7826277450333225168004.log.tmp ...
Our Jenkinsfile is just this and calling our maven script:
agent {
docker {
label 'myVirtualMachine6'
image 'myImage'
registryUrl 'xxxx'
registryCredentialsId 'yyyy'
args '-v maven-repo:/home/myUser/.m2/repository/ -v /var/run/docker.sock:/var/run/docker.sock'
}
}