Gradle with Java11 build produces intermittent compilation errors - docker

Question

We are using gradle:6.9.1-jdk11-hotspot image to build one of our java jar components.
Gradle 6.9.1
Build time: 2021-08-20 11:15:18 UTC.
Revision: f0ddb54aaae0e44f0a7209c3c0274d506ea742a0
Kotlin: 1.4.20
Groovy: 2.5.12
Ant: Apache Ant(TM) version 1.10.9 compiled on September 27 2020
JVM: 11.0.11 (AdoptOpenJDK 11.0.11+9)
OS: Linux 5.10.47-linuxkit amd64.
The Gradle build file also use io.freefair.aspectj.compile-time-weaving:3.8.4 for compile time weaving.
We get the following compilation errors
The type javax.xml.stream.XMLEventReader is not accessible.
import javax.xml.stream.XMLEventReader;
The type javax.xml.stream.XMLInputFactory is not accessible
import javax.xml.stream.XMLInputFactory;
XMLEventReader cannot be resolved to a type
private XMLEventReader xmlEventReader;
XMLInputFactory cannot be resolved
XMLInputFactory xmlInputFactory = XMLInputFactory.newInstance();
This compilation error does not happen consistently and it happens intermittently. Retrying after a couple of times the build succeeds.
We use docker-compose of version 2.0.1 to build the image. The Gradle build is executed as part of building the docker image.
Here is the Dockerfile:
FROM gradle:6.9.1-jdk11-hotspot
ENV GRADLE_USER_HOME=/app/.gradle
USER root
WORKDIR app/libs
COPY common/*.gradle ./common/
COPY common/src ./common/src
RUN gradle --stacktrace --no-daemon :common:build