Jenkins Windows worker fails first build with ClosedChannelException - jenkins

Question

I have a self-provisioned Jenkins setup using vSphere cloud, where workers are cloned from previously built templates on demand. Everything is working smooth except for Windows workers. When I start a job which uses Windows node and there is none available, Jenkins creates one to run the job (it takes a 2 or 3 mins) but then fails on the checkout scm step, which gives following output:
> git fetch --no-tags --force --progress -- <repo> +refs/heads/master:refs/remotes/origin/master # timeout=10
12:58:50 [Pipeline] cleanWs
12:58:50 jenkins-windows-2019-worker1 was marked offline: Connection was broken: java.nio.channels.ClosedChannelException
12:58:50 at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:144)
12:58:50 at jenkins.websocket.WebSocketSession.onWebSocketSomething(WebSocketSession.java:91)
12:58:50 at com.sun.proxy.$Proxy129.onWebSocketClose(Unknown Source)
12:58:50 at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:149)
12:58:50 at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:394)
12:58:50 at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:225)
12:58:50 at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection$Flusher.onCompleteFailure(AbstractWebSocketConnection.java:100)
12:58:50 at org.eclipse.jetty.util.IteratingCallback.failed(IteratingCallback.java:402)
12:58:50 at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)
12:58:50 at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
12:58:50 at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
12:58:50 at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
12:58:50 at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
12:58:50 at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
12:58:50 at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:581)
12:58:50 at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181)
12:58:50 at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:510)
12:58:50 at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
12:58:50 at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
12:58:50 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
12:58:50 at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
12:58:50 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
12:58:50 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
12:58:50 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
12:58:50 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
12:58:50 at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
12:58:50 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
12:58:50 at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
12:58:50 at java.base/java.lang.Thread.run(Thread.java:829)
12:58:50 org.jenkinsci.plugins.workflow.support.steps.AgentOfflineException: Unable to create live FilePath for jenkins-windows-2019-worker1
12:58:50 at org.jenkinsci.plugins.workflow.support.steps.FilePathDynamicContext.get(FilePathDynamicContext.java:66)
12:58:50 at org.jenkinsci.plugins.workflow.support.steps.FilePathDynamicContext.get(FilePathDynamicContext.java:48)
12:58:50 at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:95)
12:58:50 at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:139)
12:58:50 at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:137)
12:58:50 at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:298)
12:58:50 at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:75)
12:58:50 at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:83)
12:58:50 at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:71)
12:58:50 at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
12:58:50 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
12:58:50 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
12:58:50 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
12:58:50 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
12:58:50 at java.base/java.lang.Thread.run(Thread.java:829)
Interesting thing is that I can replay the job and it will succeed on the same worker - only the first job on given Windows worker instance will fail.
Jenkins worker is started as Windows service using New-Service cmdlet. The service runs following command: java -Xmx6g -jar C:\jenkins-worker\agent.jar -jnlpUrl <jnlpUrl> -secret <secret>.
Linux workers are fine while having the same env (same network, same vCenter etc.)
I've tried updating Java versions to exactly match up to minor version - no result. Adding sleep before checkout scm solves the issue but is very dirty.