How to parse build.gradle just like Gradle 7 that supports variable references at end of interpolated strings not enclosed in curly brackets - parsing

Question

NOTE: I have replicated this issue on Windows 10 and MacOS 12.6.2. I get the same results with Groovy 4.0.6 and Groovy 4.0.7 (and after updates: 3.0.9).
In order to automate some changes to build.gradle files that are written in Groovy, I'm using the official Apache Groovy parser 4.0.7 (which uses ANTLR 4.11.1) to parse the Groovy build.gradle files. I'm finding that the Groovy parser fails to parse build.gradle files that include variable references at the end of interpolated strings that are not enclosed in curly brackets. However, Gradle 7.5.1 successfully parses the same build.gradle file just fine with the unbracketed variable references at the end of interpolated strings.
To avoid the parser failures I am having to preprocess the build.gradle files to enclose the variable names in curly brackets when located at the end of an interpolated string. However, that adds complications and risks of RegExp pattern errors considering all the various scenarios I have to deal with. To some degree, that involves creating my own mini Groovy parser via RegExp.
Maybe Gradle is preprocessing the build.gradle file? Maybe Gradle 7.5.1 uses a different parser than the official Apache Groovy parser 4.0.7?
I would like to match what Gradle is doing to parse the build.gradle files as closely as possible to avoid these kinds of issues.
The following OneCompiler clip (https://onecompiler.com/groovy/3yuaw5jdv) shows both bracketed and unbracketed vars at the end of interpolated strings working for the following source in Groovy 2.7 (note: my program uses Groovy 4.0.7):
String name = "Joe"
println "Hello, ${name}"
println "Hello, $name "
println "Hello, $name/"
println "Hello, $name" // this fails in Groovy Parser 4.0.7
Output:
#1 Hello, Joe
#2 Hello, Joe
#3 Hello, Joe/
#4 Hello, Joe
I can paste the above code into a build.gradle file and run ./gradlew.bat build -x test without errors to get the expected output:
// build.gradle
plugins {
id 'java'
}
group 'org.example'
version '1.0-SNAPSHOT'
String name = "Joe"
println "#1 Hello, ${name}"
println "#2 Hello, $name "
println "#3 Hello, $name/"
println "#4 Hello, $name" // this fails in Groovy Parser 4.0.7
repositories {
mavenCentral()
}
dependencies {
testImplementation 'org.junit.jupiter:junit-jupiter-api:5.8.1'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.8.1'
implementation 'org.apache.groovy:groovy:4.0.7'
}
test {
useJUnitPlatform()
}
Output:
PS C:\projects\groovy\groovy407> ./gradlew.bat build
> Configure project :
#1 Hello, Joe
#2 Hello, Joe
#3 Hello, Joe/
#4 Hello, Joe
BUILD SUCCESSFUL in 7s
4 actionable tasks: 4 executed
However, when I try to parse the above build.gradle file via the Groovy Parser 4.0.7, it can't parse the file and reports parser errors when a variable reference is at the end of an interpolated string and not enclosed in curly brackets.
The following Java class and Java test class demonstrate how the Groovy parser 4.0.7 handles the variable references at the end of interpolated strings not enclosed in curly brackets.
InterpolatedStringParseTester.java:
package org.example;
import groovyjarjarantlr4.v4.runtime.CharStreams;
import groovyjarjarantlr4.v4.runtime.CommonTokenStream;
import groovyjarjarantlr4.v4.runtime.InputMismatchException;
import org.apache.groovy.parser.antlr4.GroovyLexer;
import org.apache.groovy.parser.antlr4.GroovyParser;
import java.rmi.UnexpectedException;
public class InterpolatedStringParseTester {
public String upgrade(String contents) throws UnexpectedException{
try {
GroovyLexer groovyLexer = new GroovyLexer(CharStreams.fromString(contents));
CommonTokenStream tokens = new CommonTokenStream(groovyLexer);
GroovyParser groovyParser = new GroovyParser(tokens);
GroovyParser.CompilationUnitContext tree = groovyParser.compilationUnit();
if (tree.exception instanceof InputMismatchException) {
throw new UnexpectedException("failed to parse contents: ", tree.exception);
}
} catch (Exception e) {
e.printStackTrace();
throw new UnexpectedException(e.getLocalizedMessage());
}
return "";
}
}
InterpolatedStringParseTesterTest.java:
package org.example;
import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.rmi.UnexpectedException;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
class InterpolatedStringParseTesterTest {
String bracketedVarInStrInClass_OK = "class AClass { String x=\"y\" \n" +
"String a=\"${x}\" }";
String unbracketedVarInStrInClass_ParserErr = "class AClass { String x=\"y\" \n" +
"String a=\"$x\" }";
String closureWithVarInStr_OK = "{ String x-> println \"${x}\" }";
String closureWithUnbracketedVarInStrFailsParse_ThrowsException = "{ String x-> println \"$x\" }";
String helloJoe4Ways = "String name = \"Joe\"\n" +
"println \"#1 Hello, ${name}\"\n" +
"println \"#2 Hello, $name \"\n" +
"println \"#3 Hello, $name/\"\n" +
"println \"#4 Hello, $name\" // this fails in Groovy Parser 4.0.7\n";
#Test
void bracketedVarInStrInClassTest_OK() throws UnexpectedException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
assertEquals("", interpolatedStringParseTester.upgrade(bracketedVarInStrInClass_OK));
}
#Test
void unbracketedVarInStrInClassTest_NoException_ButParserErr() throws UnexpectedException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
assertEquals("", interpolatedStringParseTester.upgrade(unbracketedVarInStrInClass_ParserErr));
// does not throw an error, but outputs error: line 2:13 mismatched input ' }' expecting {GStringEnd, GStringPart}
}
#Test
void closureWithVarInStrTest_OK() throws UnexpectedException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
assertEquals("", interpolatedStringParseTester.upgrade(closureWithVarInStr_OK));
}
#Test
void closureWithUnbracketedVarInStrTest_ThrowsException() throws UnexpectedException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
assertThrows(UnexpectedException.class,
() -> {
interpolatedStringParseTester.upgrade(closureWithUnbracketedVarInStrFailsParse_ThrowsException);
/*
Throws:
line 1:21 missing RBRACE at '"$'
java.rmi.UnexpectedException: failed to parse contents: ; nested exception is:
groovyjarjarantlr4.v4.runtime.InputMismatchException
*/
});
}
#Test
void helloJoe4WaysTest_ThrowsInputMismatchException() throws UnexpectedException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
assertThrows(UnexpectedException.class,
() -> {
interpolatedStringParseTester.upgrade(helloJoe4Ways);
});
/**
* this results in:
* line 5:25 token recognition error at: ' // this fails in Groovy Parser 4.0.7\n'
* line 5:8 mismatched input '"#4 Hello, $' expecting {<EOF>, ';', NL}
* java.rmi.UnexpectedException: failed to parse contents: ; nested exception is:
*/
}
#Test
void buildGradleParse_ThrowsInputMismatchException() throws IOException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
Path path = Paths.get("build.gradle");
Stream<String> lines = Files.lines(path);
String buildGradleContents = lines.collect(Collectors.joining("\n"));
lines.close();
assertThrows(UnexpectedException.class,
() -> {
interpolatedStringParseTester.upgrade(buildGradleContents);
});
}
}
UPDATE:
Below is the stderr output along with the stacktrace of the exception that is returned (i.e., returned in the tree.exception field as opposed to being thrown) by GroovyParser.CompilationUnitContext tree = groovyParser.compilationUnit(); in the unit test named helloJoe4WaysTest_ThrowsInputMismatchException:
line 5:25 token recognition error at: ' // this fails in Groovy Parser 4.0.7\n'
line 5:8 mismatched input '"#4 Hello, $' expecting {<EOF>, ';', NL}
groovyjarjarantlr4.v4.runtime.InputMismatchException
at groovyjarjarantlr4.v4.runtime.DefaultErrorStrategy.recoverInline(DefaultErrorStrategy.java:492)
at groovyjarjarantlr4.v4.runtime.Parser.match(Parser.java:213)
at org.apache.groovy.parser.antlr4.GroovyParser.compilationUnit(GroovyParser.java:368)
at org.example.InterpolatedStringParseTester.upgrade(InterpolatedStringParseTester.java:18)
at org.example.InterpolatedStringParseTesterTest.lambda$helloJoe4WaysTest_ThrowsInputMismatchException$1(InterpolatedStringParseTesterTest.java:71)
at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:55)
at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:37)
at org.junit.jupiter.api.Assertions.assertThrows(Assertions.java:3082)
at org.example.InterpolatedStringParseTesterTest.helloJoe4WaysTest_ThrowsInputMismatchException(InterpolatedStringParseTesterTest.java:69)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.stop(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
UPDATE:
On Windows 10, Gradle is using:
Gradle version: 7.4
Groovy version: 3.0.9
On MacOS 12.6.1 Gradle is using:
Gradle version: 7.4
Groovy version: 3.0.9
When I have more time I will try to re-run these tests with the parser that comes with Groovy 3.0.9 and 3.0.10 to match what Gradle is using.
UPDATE:
Since learning that Gradle 7.4 uses Groovy 3.0.9, I have re-run the same tests with Groovy 3.0.9 that result in the same exception when the variable reference is not wrapped in curl brackets and is at the end of a double-quoted interpolated string.
The following is the dependency in build.gradle to make use of the Groovy parse v3.0.9:
implementation 'org.codehaus.groovy:groovy-all:3.0.9'
I created a simple Java project to test Groovy 3.0.9.
Below is the code and output that demonstrates the difference between "with curl brackets" and "without curly brackets" around variable references that appear at the end of interpolated strings:
Main.java:
package org.example;
import groovyjarjarantlr4.v4.runtime.CharStreams;
import groovyjarjarantlr4.v4.runtime.CommonTokenStream;
import org.apache.groovy.parser.antlr4.GroovyLexer;
import org.apache.groovy.parser.antlr4.GroovyParser;
public class Main {
public static void main(String[] args) {
Main main = new Main();
boolean useBrackets = args.length> 0 && "y".equalsIgnoreCase(args[0]);
String brackets = "buildscript { def v=\"1.2.3\"; dependencies { classpath { \"grp:pkg:${v}\" } } }";
String noBrackets = "buildscript { def v=\"1.2.3\"; dependencies { classpath { \"grp:pkg:$v\" } } }";
String groovyCode = useBrackets ? brackets : noBrackets;
System.out.println("Groovy code:" + groovyCode);
main.runVisitor(groovyCode);
}
void runVisitor(String buildGradleContents) {
GroovyLexer groovyLexer = new GroovyLexer(CharStreams.fromString(buildGradleContents));
CommonTokenStream tokens = new CommonTokenStream(groovyLexer);
GroovyParser groovyParser = new GroovyParser(tokens);
GroovyParser.CompilationUnitContext tree = groovyParser.compilationUnit();
if (tree.exception != null) {
tree.exception.printStackTrace();
throw tree.exception;
}
TestVisitor testVisitor = new TestVisitor();
String results = testVisitor.visit(tree);
System.out.println("results: "+ results);
}
}
TestVisitor.java:
package org.example;
import groovyjarjarantlr4.v4.runtime.Token;
import org.apache.groovy.parser.antlr4.GroovyParser;
import org.apache.groovy.parser.antlr4.GroovyParserBaseVisitor;
public class TestVisitor extends GroovyParserBaseVisitor<String>{
public String visitIdentifier(GroovyParser.IdentifierContext ctx) {
Token startToken = ctx.getStart();
if (startToken != null) {
String startTxt = startToken.getText();
}
return visitChildren(ctx);
}
}
OUTPUT:
# WITH curly brackets
PS C:\projects\research\gradle-tooling> java -jar .\build\libs\gradle-tooling-1.0-SNAPSHOT.jar y
Groovy code:buildscript { def v="1.2.3"; dependencies { classpath { "grp:pkg:${v}" } } }
results: null
# WITHOUT curly brackets
PS C:\projects\research\gradle-tooling> java -jar .\build\libs\gradle-tooling-1.0-SNAPSHOT.jar n
Groovy code:buildscript { def v="1.2.3"; dependencies { classpath { "grp:pkg:$v" } } }
line 1:12 mismatched input '{' expecting {<EOF>, StringLiteral, GStringBegin, 'as', 'in', 'trait', 'var', IntegerLiteral, FloatingPointLiteral, BooleanLiteral, 'null', ';', Capitalized
Identifier, Identifier, NL}
groovyjarjarantlr4.v4.runtime.InputMismatchException
at groovyjarjarantlr4.v4.runtime.DefaultErrorStrategy.recoverInline(DefaultErrorStrategy.java:492)
at groovyjarjarantlr4.v4.runtime.Parser.match(Parser.java:213)
at org.apache.groovy.parser.antlr4.GroovyParser.compilationUnit(GroovyParser.java:362)
at org.example.Main.runVisitor(Main.java:28)
at org.example.Main.main(Main.java:21)
Exception in thread "main" groovyjarjarantlr4.v4.runtime.InputMismatchException
at groovyjarjarantlr4.v4.runtime.DefaultErrorStrategy.recoverInline(DefaultErrorStrategy.java:492)
at groovyjarjarantlr4.v4.runtime.Parser.match(Parser.java:213)
at org.apache.groovy.parser.antlr4.GroovyParser.compilationUnit(GroovyParser.java:362)
at org.example.Main.runVisitor(Main.java:28)
at org.example.Main.main(Main.java:21)