Recursive listing of all files in Groovy with eachFileRecurse return FileNotFoundException - jenkins

My new File work and contains any xml files but my but eachFileRecurse return java.io.FileNotFoundException:
/home/jenkins/workspace/myjob/abcd/TestFlux
Caught: java.io.FileNotFoundException: /home/jenkins/workspace/myjob/abcd/TestFlux
java.io.FileNotFoundException: /home/jenkins/workspace/myjob/abcd/TestFlux
at tests_flux.foo(tests_flux.groovy:48)
at tests_flux$traiter.callCurrent(Unknown Source)
at tests_flux.run(tests_flux.groovy:114)
This is my code run on Jenkins:
import static groovy.io.FileType.FILES
def foo() {
File f = new File("abcd/TestFlux")
String currentDir = f.getAbsolutePath()
println currentDir
f.eachFileRecurse(FILES) {
println it
}
}

new File() does not imply the file exists or anything. It's an holder, that be asked to look into the fileystem for you. E.g.
groovy:000> new File("/hehe/lol")
===> /hehe/lol
groovy:000> new File("/hehe/lol").exists()
===> false
groovy:000> new File("/hehe/lol").isDirectory()
===> false

Related

How to parse build.gradle just like Gradle 7 that supports variable references at end of interpolated strings not enclosed in curly brackets

NOTE: I have replicated this issue on Windows 10 and MacOS 12.6.2. I get the same results with Groovy 4.0.6 and Groovy 4.0.7 (and after updates: 3.0.9).
In order to automate some changes to build.gradle files that are written in Groovy, I'm using the official Apache Groovy parser 4.0.7 (which uses ANTLR 4.11.1) to parse the Groovy build.gradle files. I'm finding that the Groovy parser fails to parse build.gradle files that include variable references at the end of interpolated strings that are not enclosed in curly brackets. However, Gradle 7.5.1 successfully parses the same build.gradle file just fine with the unbracketed variable references at the end of interpolated strings.
To avoid the parser failures I am having to preprocess the build.gradle files to enclose the variable names in curly brackets when located at the end of an interpolated string. However, that adds complications and risks of RegExp pattern errors considering all the various scenarios I have to deal with. To some degree, that involves creating my own mini Groovy parser via RegExp.
Maybe Gradle is preprocessing the build.gradle file? Maybe Gradle 7.5.1 uses a different parser than the official Apache Groovy parser 4.0.7?
I would like to match what Gradle is doing to parse the build.gradle files as closely as possible to avoid these kinds of issues.
The following OneCompiler clip (https://onecompiler.com/groovy/3yuaw5jdv) shows both bracketed and unbracketed vars at the end of interpolated strings working for the following source in Groovy 2.7 (note: my program uses Groovy 4.0.7):
String name = "Joe"
println "Hello, ${name}"
println "Hello, $name "
println "Hello, $name/"
println "Hello, $name" // this fails in Groovy Parser 4.0.7
Output:
#1 Hello, Joe
#2 Hello, Joe
#3 Hello, Joe/
#4 Hello, Joe
I can paste the above code into a build.gradle file and run ./gradlew.bat build -x test without errors to get the expected output:
// build.gradle
plugins {
id 'java'
}
group 'org.example'
version '1.0-SNAPSHOT'
String name = "Joe"
println "#1 Hello, ${name}"
println "#2 Hello, $name "
println "#3 Hello, $name/"
println "#4 Hello, $name" // this fails in Groovy Parser 4.0.7
repositories {
mavenCentral()
}
dependencies {
testImplementation 'org.junit.jupiter:junit-jupiter-api:5.8.1'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.8.1'
implementation 'org.apache.groovy:groovy:4.0.7'
}
test {
useJUnitPlatform()
}
Output:
PS C:\projects\groovy\groovy407> ./gradlew.bat build
> Configure project :
#1 Hello, Joe
#2 Hello, Joe
#3 Hello, Joe/
#4 Hello, Joe
BUILD SUCCESSFUL in 7s
4 actionable tasks: 4 executed
However, when I try to parse the above build.gradle file via the Groovy Parser 4.0.7, it can't parse the file and reports parser errors when a variable reference is at the end of an interpolated string and not enclosed in curly brackets.
The following Java class and Java test class demonstrate how the Groovy parser 4.0.7 handles the variable references at the end of interpolated strings not enclosed in curly brackets.
InterpolatedStringParseTester.java:
package org.example;
import groovyjarjarantlr4.v4.runtime.CharStreams;
import groovyjarjarantlr4.v4.runtime.CommonTokenStream;
import groovyjarjarantlr4.v4.runtime.InputMismatchException;
import org.apache.groovy.parser.antlr4.GroovyLexer;
import org.apache.groovy.parser.antlr4.GroovyParser;
import java.rmi.UnexpectedException;
public class InterpolatedStringParseTester {
public String upgrade(String contents) throws UnexpectedException{
try {
GroovyLexer groovyLexer = new GroovyLexer(CharStreams.fromString(contents));
CommonTokenStream tokens = new CommonTokenStream(groovyLexer);
GroovyParser groovyParser = new GroovyParser(tokens);
GroovyParser.CompilationUnitContext tree = groovyParser.compilationUnit();
if (tree.exception instanceof InputMismatchException) {
throw new UnexpectedException("failed to parse contents: ", tree.exception);
}
} catch (Exception e) {
e.printStackTrace();
throw new UnexpectedException(e.getLocalizedMessage());
}
return "";
}
}
InterpolatedStringParseTesterTest.java:
package org.example;
import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.rmi.UnexpectedException;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
class InterpolatedStringParseTesterTest {
String bracketedVarInStrInClass_OK = "class AClass { String x=\"y\" \n" +
"String a=\"${x}\" }";
String unbracketedVarInStrInClass_ParserErr = "class AClass { String x=\"y\" \n" +
"String a=\"$x\" }";
String closureWithVarInStr_OK = "{ String x-> println \"${x}\" }";
String closureWithUnbracketedVarInStrFailsParse_ThrowsException = "{ String x-> println \"$x\" }";
String helloJoe4Ways = "String name = \"Joe\"\n" +
"println \"#1 Hello, ${name}\"\n" +
"println \"#2 Hello, $name \"\n" +
"println \"#3 Hello, $name/\"\n" +
"println \"#4 Hello, $name\" // this fails in Groovy Parser 4.0.7\n";
#Test
void bracketedVarInStrInClassTest_OK() throws UnexpectedException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
assertEquals("", interpolatedStringParseTester.upgrade(bracketedVarInStrInClass_OK));
}
#Test
void unbracketedVarInStrInClassTest_NoException_ButParserErr() throws UnexpectedException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
assertEquals("", interpolatedStringParseTester.upgrade(unbracketedVarInStrInClass_ParserErr));
// does not throw an error, but outputs error: line 2:13 mismatched input ' }' expecting {GStringEnd, GStringPart}
}
#Test
void closureWithVarInStrTest_OK() throws UnexpectedException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
assertEquals("", interpolatedStringParseTester.upgrade(closureWithVarInStr_OK));
}
#Test
void closureWithUnbracketedVarInStrTest_ThrowsException() throws UnexpectedException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
assertThrows(UnexpectedException.class,
() -> {
interpolatedStringParseTester.upgrade(closureWithUnbracketedVarInStrFailsParse_ThrowsException);
/*
Throws:
line 1:21 missing RBRACE at '"$'
java.rmi.UnexpectedException: failed to parse contents: ; nested exception is:
groovyjarjarantlr4.v4.runtime.InputMismatchException
*/
});
}
#Test
void helloJoe4WaysTest_ThrowsInputMismatchException() throws UnexpectedException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
assertThrows(UnexpectedException.class,
() -> {
interpolatedStringParseTester.upgrade(helloJoe4Ways);
});
/**
* this results in:
* line 5:25 token recognition error at: ' // this fails in Groovy Parser 4.0.7\n'
* line 5:8 mismatched input '"#4 Hello, $' expecting {<EOF>, ';', NL}
* java.rmi.UnexpectedException: failed to parse contents: ; nested exception is:
*/
}
#Test
void buildGradleParse_ThrowsInputMismatchException() throws IOException {
InterpolatedStringParseTester interpolatedStringParseTester = new InterpolatedStringParseTester();
Path path = Paths.get("build.gradle");
Stream<String> lines = Files.lines(path);
String buildGradleContents = lines.collect(Collectors.joining("\n"));
lines.close();
assertThrows(UnexpectedException.class,
() -> {
interpolatedStringParseTester.upgrade(buildGradleContents);
});
}
}
UPDATE:
Below is the stderr output along with the stacktrace of the exception that is returned (i.e., returned in the tree.exception field as opposed to being thrown) by GroovyParser.CompilationUnitContext tree = groovyParser.compilationUnit(); in the unit test named helloJoe4WaysTest_ThrowsInputMismatchException:
line 5:25 token recognition error at: ' // this fails in Groovy Parser 4.0.7\n'
line 5:8 mismatched input '"#4 Hello, $' expecting {<EOF>, ';', NL}
groovyjarjarantlr4.v4.runtime.InputMismatchException
at groovyjarjarantlr4.v4.runtime.DefaultErrorStrategy.recoverInline(DefaultErrorStrategy.java:492)
at groovyjarjarantlr4.v4.runtime.Parser.match(Parser.java:213)
at org.apache.groovy.parser.antlr4.GroovyParser.compilationUnit(GroovyParser.java:368)
at org.example.InterpolatedStringParseTester.upgrade(InterpolatedStringParseTester.java:18)
at org.example.InterpolatedStringParseTesterTest.lambda$helloJoe4WaysTest_ThrowsInputMismatchException$1(InterpolatedStringParseTesterTest.java:71)
at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:55)
at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:37)
at org.junit.jupiter.api.Assertions.assertThrows(Assertions.java:3082)
at org.example.InterpolatedStringParseTesterTest.helloJoe4WaysTest_ThrowsInputMismatchException(InterpolatedStringParseTesterTest.java:69)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.stop(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
UPDATE:
On Windows 10, Gradle is using:
Gradle version: 7.4
Groovy version: 3.0.9
On MacOS 12.6.1 Gradle is using:
Gradle version: 7.4
Groovy version: 3.0.9
When I have more time I will try to re-run these tests with the parser that comes with Groovy 3.0.9 and 3.0.10 to match what Gradle is using.
UPDATE:
Since learning that Gradle 7.4 uses Groovy 3.0.9, I have re-run the same tests with Groovy 3.0.9 that result in the same exception when the variable reference is not wrapped in curl brackets and is at the end of a double-quoted interpolated string.
The following is the dependency in build.gradle to make use of the Groovy parse v3.0.9:
implementation 'org.codehaus.groovy:groovy-all:3.0.9'
I created a simple Java project to test Groovy 3.0.9.
Below is the code and output that demonstrates the difference between "with curl brackets" and "without curly brackets" around variable references that appear at the end of interpolated strings:
Main.java:
package org.example;
import groovyjarjarantlr4.v4.runtime.CharStreams;
import groovyjarjarantlr4.v4.runtime.CommonTokenStream;
import org.apache.groovy.parser.antlr4.GroovyLexer;
import org.apache.groovy.parser.antlr4.GroovyParser;
public class Main {
public static void main(String[] args) {
Main main = new Main();
boolean useBrackets = args.length> 0 && "y".equalsIgnoreCase(args[0]);
String brackets = "buildscript { def v=\"1.2.3\"; dependencies { classpath { \"grp:pkg:${v}\" } } }";
String noBrackets = "buildscript { def v=\"1.2.3\"; dependencies { classpath { \"grp:pkg:$v\" } } }";
String groovyCode = useBrackets ? brackets : noBrackets;
System.out.println("Groovy code:" + groovyCode);
main.runVisitor(groovyCode);
}
void runVisitor(String buildGradleContents) {
GroovyLexer groovyLexer = new GroovyLexer(CharStreams.fromString(buildGradleContents));
CommonTokenStream tokens = new CommonTokenStream(groovyLexer);
GroovyParser groovyParser = new GroovyParser(tokens);
GroovyParser.CompilationUnitContext tree = groovyParser.compilationUnit();
if (tree.exception != null) {
tree.exception.printStackTrace();
throw tree.exception;
}
TestVisitor testVisitor = new TestVisitor();
String results = testVisitor.visit(tree);
System.out.println("results: "+ results);
}
}
TestVisitor.java:
package org.example;
import groovyjarjarantlr4.v4.runtime.Token;
import org.apache.groovy.parser.antlr4.GroovyParser;
import org.apache.groovy.parser.antlr4.GroovyParserBaseVisitor;
public class TestVisitor extends GroovyParserBaseVisitor<String>{
public String visitIdentifier(GroovyParser.IdentifierContext ctx) {
Token startToken = ctx.getStart();
if (startToken != null) {
String startTxt = startToken.getText();
}
return visitChildren(ctx);
}
}
OUTPUT:
# WITH curly brackets
PS C:\projects\research\gradle-tooling> java -jar .\build\libs\gradle-tooling-1.0-SNAPSHOT.jar y
Groovy code:buildscript { def v="1.2.3"; dependencies { classpath { "grp:pkg:${v}" } } }
results: null
# WITHOUT curly brackets
PS C:\projects\research\gradle-tooling> java -jar .\build\libs\gradle-tooling-1.0-SNAPSHOT.jar n
Groovy code:buildscript { def v="1.2.3"; dependencies { classpath { "grp:pkg:$v" } } }
line 1:12 mismatched input '{' expecting {<EOF>, StringLiteral, GStringBegin, 'as', 'in', 'trait', 'var', IntegerLiteral, FloatingPointLiteral, BooleanLiteral, 'null', ';', Capitalized
Identifier, Identifier, NL}
groovyjarjarantlr4.v4.runtime.InputMismatchException
at groovyjarjarantlr4.v4.runtime.DefaultErrorStrategy.recoverInline(DefaultErrorStrategy.java:492)
at groovyjarjarantlr4.v4.runtime.Parser.match(Parser.java:213)
at org.apache.groovy.parser.antlr4.GroovyParser.compilationUnit(GroovyParser.java:362)
at org.example.Main.runVisitor(Main.java:28)
at org.example.Main.main(Main.java:21)
Exception in thread "main" groovyjarjarantlr4.v4.runtime.InputMismatchException
at groovyjarjarantlr4.v4.runtime.DefaultErrorStrategy.recoverInline(DefaultErrorStrategy.java:492)
at groovyjarjarantlr4.v4.runtime.Parser.match(Parser.java:213)
at org.apache.groovy.parser.antlr4.GroovyParser.compilationUnit(GroovyParser.java:362)
at org.example.Main.runVisitor(Main.java:28)
at org.example.Main.main(Main.java:21)

How to list a windows share folder by active choices parameter or extened choice parameter in jenkins

I tryed to write a groovy script with extened choice parameter:
import jcifs.smb.*
try {
def source = "smb://192.168.1.xx/build/"
NtlmPasswordAuthentication auth = new NtlmPasswordAuthentication("mydc", "jenkins", "*****");
def dir = new SmbFile(source, auth)
// def dir = new SmbFile(source)
def files = dir.listFiles()
return ["xx"]
def list = []
for (SmbFile f : files)
{
list << f.getName()
}
list << "xx"
return list
} catch (Exception e1) {
return [e1.getMessage()]
}
but then I get the exception:
Failed to connect: 0.0.0.0<00>/192.168.1.xx .
I have open the server's smb1 support.
How can I list the smb folder or file with jenkins parameter choice?
I find the solution my self:
import groovy.io.FileType
try {
def source = "\\\\192.168.xx.xx\\xx\\xx"
def dir = new File(source)
def list = []
dir.eachFile(FileType.DIRECTORIES) {
list << it.name
}
return list
} catch (Exception e1) {
return [e1.getMessage()]
}
maybe I have saw the answer some place, but my java is so poor, I didn't understand. I tried many times to get the solution.

Apache Beam IllegalArgumentException on Google Dataflow with message `Not expecting a splittable ParDoSingle: should have been overridden`

I am trying to write a pipeline which periodically checks a Google Storage bucket for new .gz files which are actually compressed .csv files. Then it writes those records to a BigQuery table. The following code was working in batch mode before I added the .watchForNewFiles(...) and .withMethod(STREAMING_INSERTS) parts. I am expecting it to run in streaming mode with those changes. However I am getting an exception that I can't find anything related on the web. Here is my code:
public static void main(String[] args) {
DataflowDfpOptions options = PipelineOptionsFactory.fromArgs(args)
//.withValidation()
.as(DataflowDfpOptions.class);
Pipeline pipeline = Pipeline.create(options);
Stopwatch sw = Stopwatch.createStarted();
log.info("DFP data transfer from GS to BQ has started.");
pipeline.apply("ReadFromStorage", TextIO.read()
.from("gs://my-bucket/my-folder/*.gz")
.withCompression(Compression.GZIP)
.watchForNewFiles(
// Check for new files every 30 seconds
Duration.standardSeconds(30),
// Never stop checking for new files
Watch.Growth.never()
)
)
.apply("TransformToTableRow", ParDo.of(new TableRowConverterFn()))
.apply("WriteToBigQuery", BigQueryIO.writeTableRows()
.to(options.getTableId())
.withMethod(STREAMING_INSERTS)
.withCreateDisposition(CREATE_NEVER)
.withWriteDisposition(WRITE_APPEND)
.withSchema(TableSchema)); //todo: use withJsonScheme(String json) method instead
pipeline.run().waitUntilFinish();
log.info("DFP data transfer from GS to BQ is finished in {} seconds.", sw.elapsed(TimeUnit.SECONDS));
}
/**
* Creates a TableRow from a CSV line
*/
private static class TableRowConverterFn extends DoFn<String, TableRow> {
#ProcessElement
public void processElement(ProcessContext c) throws Exception {
String[] split = c.element().split(",");
//Ignore the header line
//Since this is going to be run in parallel, we can't guarantee that the first line passed to this method will be the header
if (split[0].equals("Time")) {
log.info("Skipped header");
return;
}
TableRow row = new TableRow();
for (int i = 0; i < split.length; i++) {
TableFieldSchema col = TableSchema.getFields().get(i);
//String is the most common type, putting it in the first if clause for a little bit optimization.
if (col.getType().equals("STRING")) {
row.set(col.getName(), split[i]);
} else if (col.getType().equals("INTEGER")) {
row.set(col.getName(), Long.valueOf(split[i]));
} else if (col.getType().equals("BOOLEAN")) {
row.set(col.getName(), Boolean.valueOf(split[i]));
} else if (col.getType().equals("FLOAT")) {
row.set(col.getName(), Float.valueOf(split[i]));
} else {
//Simply try to write it as a String if
//todo: Consider other BQ data types.
row.set(col.getName(), split[i]);
}
}
c.output(row);
}
}
And the stack trace:
java.lang.IllegalArgumentException: Not expecting a splittable ParDoSingle: should have been overridden
at org.apache.beam.repackaged.beam_runners_google_cloud_dataflow_java.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
at org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$PayloadTranslator.payloadForParDoSingle(PrimitiveParDoSingleFactory.java:167)
at org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$PayloadTranslator.translate(PrimitiveParDoSingleFactory.java:145)
at org.apache.beam.runners.core.construction.PTransformTranslation.toProto(PTransformTranslation.java:206)
at org.apache.beam.runners.core.construction.SdkComponents.registerPTransform(SdkComponents.java:86)
at org.apache.beam.runners.core.construction.PipelineTranslation$1.visitPrimitiveTransform(PipelineTranslation.java:87)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:668)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:660)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:660)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:660)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:660)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:660)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
at org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:59)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:165)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:684)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:173)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
at com.diply.data.App.main(App.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
at java.lang.Thread.run(Thread.java:748)
Here is my command to publish the job on Dataflow:
clean compile exec:java -Dexec.mainClass=com.my.project.App "-Dexec.args=--runner=DataflowRunner --tempLocation=gs://my-bucket/tmp --tableId=Temp.TestTable --project=my-project --jobName=dataflow-dfp-streaming" -Pdataflow-runner
I use apache beam version 2.5.0. Here is the relevant section from my pom.xml.
<properties>
<beam.version>2.5.0</beam.version>
<bigquery.version>v2-rev374-1.23.0</bigquery.version>
<google-clients.version>1.23.0</google-clients.version>
...
</properties>
Running the code with Dataflow 2.4.0 gives a more explicit error: java.lang.UnsupportedOperationException: DataflowRunner does not currently support splittable DoFn
However, this answer suggests that this is supported since 2.2.0. This is indeed the case, and following this remark you need to add the --streaming option in your Dexec.args to force it into streaming mode.
I tested it with the code I supplied in the comments with both your pom and mine and both 1. produce your error without --streaming 2. run fine with --streaming
You might want to open a github beam issue since this behavior is not documented anywhere offically as far as I know.

dart recipe file.rename() renames the file but does not update the path in the object

The dart cookbook recipe 'Renaming a file, directory, or symlink' at https://www.dartlang.org/dart-vm/dart-by-example#renaming-a-file-directory-or-symlink doesn't seem to work as expected:
import 'dart:io';
main() async {
// Get the system temp directory.
var systemTempDir = Directory.systemTemp;
// Create a file.
var file = await new File('${systemTempDir.path}\\foo.txt').create();
// Prints path ending with `foo.txt`.
print('The path is ${file.path}');
// Rename the file.
await file.rename('${systemTempDir.path}\\bar.txt');
// Prints path ending with `bar.txt`.
print('The path is ${file.path}');
}
The output shows that the internal path field of the file object has not been changed (although the rename is successful):
[Running] dart "d:\src\dart\renameAsOnWeb.dart"
The path is C:\Users\guivh\AppData\Local\Temp\foo.txt
The path is C:\Users\guivh\AppData\Local\Temp\foo.txt
[Done] exited with code=0 in 0.327 seconds
I have extended / reworked the cookbook example to further investigate this:
import 'dart:io';
main() async {
var systemTempDir = Directory.systemTemp;
var file = await new File('${systemTempDir.path}\\foo.txt').create();
print('The file is located at ${file.path}');
File toDelete;
var newName = '${systemTempDir.path}\\fubar.toodeloo';
await file.rename(newName);
if (await new File(newName).exists() == false) {
print('The rename failed: there is no ${newName} file');
} else {
var newFile = new File(newName);
print('The rename was succesful');
var nameChangedInObject = file.path == newName;
if (nameChangedInObject) {
print('The path of the file object has changed correctly');
toDelete = newFile;
} else {
print(
'The path in the file object still is: ${file.path}');
toDelete = newFile;
}
await toDelete.delete();
print(
'And now, ${toDelete.path} is gone: ${await toDelete.exists() == false}');
}
}
And this output confirms the fact that the internal path field is not updated with the new name:
[Running] dart "d:\src\dart\renamingExample.dart"
The file is located at C:\Users\guivh\AppData\Local\Temp\foo.txt
The rename was succesful
The path in the file object still is: C:\Users\guivh\AppData\Local\Temp\foo.txt
And now, C:\Users\guivh\AppData\Local\Temp\fubar.toodeloo is gone: true
[Done] exited with code=0 in 0.369 seconds
I am runing the dev version on a windows box:
PS D:\src\dart> dart --version
Dart VM version: 2.0.0-dev.39.0 (Fri Mar 16 00:17:07 2018 +0100) on "windows_x64"
PS D:\src\dart>
Can somebody please explain what is going on here?
The example is wrong. The File object is immutable, so it definitely won't be changed by the rename operation. The example is wrong in expecting so.

Jenkins Pipeline Execute Multiple FreeStyleProjects in Parallel [duplicate]

The script is not iterating through all the values of the 'modules' array.
class Module {
public String name = '';
public Boolean isCustom = false;
public Module(String name, Boolean custom){
this.name = name;
this.isCustom = custom;
}
}
//creates array from the ext_module env var
modules = [];
EXT_MODULE.split(',').each {
modules.add(new Module(it, false));
}
println modules;
modules.each {
println "MODULE NAME ::::: ${it.name}"
if(it.isCustom)
{
println "install custom";
} else {
println "install non custom";
}
};
This is the result of the run. The array shows 4 elements, but the code inside the .each black only executes once.
Running: Print Message
[Module#71f09325, Module#e1ddb41, Module#7069a674, Module#1f68f952]
Running: Print Message
MODULE NAME ::::: puppetlabs-ntp
Running: Print Message
install non custom
Running: End of Workflow
Finished: SUCCESS
The messages "Running: Print Message" and "Running: End of Workflow" indicate that you are using the new workflow plugin: https://wiki.jenkins-ci.org/display/JENKINS/Workflow+Plugin. This plugin currently has a bug causing at least some Groovy iterations involving a closure to be aborted after one iteration: https://issues.jenkins-ci.org/browse/JENKINS-26481
The workaround is to simply use an old school for loop (code below).
Also, NonCPS is another workaround.
There is an open issue for this matter. See here: https://issues.jenkins-ci.org/browse/JENKINS-26481
Update, Oct 24th, 2016
/**
* Dumps environment varibles to the log, using an old school for loop.
*/
import com.cloudbees.groovy.cps.NonCPS
def version = '1.0'
#NonCPS
def dumpEnvVars() {
def str = "Dumping build environment variables...\n"
for (Map.Entry<String, String> entry : currentBuild.build().environment) {
str += " ${entry.key} = ${entry.value}\n"
}
echo str
}
return this;
As of yesterday, the new Pipeline plugin was delivered in version 2.0 and correct this problem.
.each closures now work, but .collect still only iterate once.

Resources