I've been using the option Restrict matrix execution to a subset from the Parameterized Trigger Plugin to pass on a combination filter to a rather large Matrix Project where all test execution is made. As the number of tests grow, so does the combination filter (which is dynamically built up) and I seemed to hit the cap. The following job gets this error message:
FATAL: Invalid method Code length 69871 in class file Script1
java.lang.ClassFormatError: Invalid method Code length 69871 in class file Script1
After reading about this problem, it seems to be a JVM constraint after reading the JVM documentation
The value of the code_length item must be less than 65536.
I get the impression that this is not something I can (or even should) tinker with in Jenkins.
My second idea was to go around this problem was to create the combination filter and then pass it as String parameter to the following Matrix Project, then use the Combination Filter option and expand the variable to achieve the same result.
Unfortunately I get this exception when trying to save my Matrix Project with a String parameter as combination filter
javax.servlet.ServletException: groovy.lang.MissingPropertyException: No such property: $COMBINATION_FILTER for class: groovy.lang.Binding
I guess this is because the variable needs to be available in the configuration when saving but I want to inject it when starting the Matrix Project.
I am running out of ideas to solve this problem. Any ideas?
You could try the Matrix Groovy Execution Strategy which is like a super combination filter
If I can quote myself
A plugin to decide the execution order and valid combinations of
matrix projects.
This uses a user defined groovy script to arrange the order which will
then be executed
Disclaimer: I built this plugin
Related
I am trying to configure Jenkins Build Parameter "users" ,to be passed as input to JMETER (v5.1) --> No.Of Threads using the function:${__javaScript(Math.round(${XX}))}
While executing test i am getting following error
error : caused by jdk.nashorn.internal.runtime.ParserException::1:12 Expected but found { Math.round(${__jexl())}
Use a variable to store Math.round(${. Test it with println. Then Invoke ${__javaScript(myVariable)}. Nested ${} is a bad idea, groovy interprets them.
as M Navneet Krishna said, your question is a bit succinct for us to provide better answer. But I gave you a method that should help you debug your stuff
I am building a dataflow pipeline with Apache beam. Below is the pseudo code:
PCollection<GenericRecord> rows = pipeline.apply("Read Json from PubSub", <some reader>)
.apply("Convert Json to pojo", ParDo.of(new JsonToPojo()))
.apply("Convert pojo to GenericRecord", ParDo.of(new PojoToGenericRecord()))
.setCoder(AvroCoder.of(GenericRecord.class, schema));
I am trying to get rid of setting the coder in the pipeline as schema won't be known at pipeline creation time (it will be present in the message).
I commented out the line that sets the coder and got an Exception saying that default coder is not configured. I used one argument version of of method and got the following Exception:
Not a Specific class: interface org.apache.avro.generic.GenericRecord
at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:285)
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:594)
at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:218)
at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:215)
at avro.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at avro.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at avro.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at avro.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
... 9 more
Is there any way for us to supply the coder at runtime, without knowing the schema beforehand?
This is possible. I recommend the following approach:
Do not use an intermediate collection of type GenericRecord. Keep it as a collection of your POJOs.
Write some transform that extracts the schema of your data and makes it available as a PCollectionView<however you want to represent the schema>.
When writing to BigQuery, write your PCollection<YourPojo> via write().to(DynamicDestinations), and when writing to Avro, use FileIO.write() or writeDynamic() in combination with AvroIO.sinkViaGenericRecords(). Both of these can take a dynamically computed schema from a side input (that you computed above).
I have an Apache Beam job running on Google Cloud Dataflow, and as part of its initialization it needs to run some basic sanity/availability checks on services, pub/sub subscriptions, GCS blobs, etc. It's a streaming pipeline intended to run ad infinitum that processes hundreds of thousands of pub/sub messages.
Currently it needs a whole heap of required, variable parameters: which Google Cloud project it needs to run in, which bucket and directory prefix it's going to be storing files in, which pub/sub subscriptions it needs to read from, and so on. It does some work with these parameters before pipeline.run is called - validation, string splitting, and the like. In its current form in order to start a job we've been passing these parameters to to a PipelineOptionsFactory and issuing a new compile every single time, but it seems like there should be a better way. I've set up the parameters to be ValueProvider objects, but because they're being called outside of pipeline.run, Maven complains at compile time that ValueProvider.get() is being called outside of a runtime context (which, yes, it is.)
I've tried using NestedValueProviders as in the Google "Creating Templates" document, but my IDE complains if I try to use NestedValueProvider.of to return a string as shown in the document. The only way I've been able to get NestedValueProviders to compile is as follows:
NestedValueProvider<String, String> pid = NestedValueProvider.of(
pipelineOptions.getDataflowProjectId(),
(SerializableFunction<String, String>) s -> s
);
(String pid = NestedValueProvider.of(...) results in the following error: "incompatible types: no instance(s) of type variable(s) T,X exist so that org.apache.beam.sdk.options.ValueProvider.NestedValueProvider conforms to java.lang.String")
I have the following in my pipelineOptions:
ValueProvider<String> getDataflowProjectId();
void setDataflowProjectId(ValueProvider<String> value);
Because of the volume of messages we're going to be processing, adding these checks at the front of the pipeline for every message that comes through isn't really practical; we'll hit daily account administrative limits on some of these calls pretty quickly.
Are templates the right approach for what I want to do? How do I go about actually productionizing this? Should (can?) I compile with maven into a jar, then just run the jar on a local dev/qa/prod box with my parameters and just not bother with ValueProviders at all? Or is it possible to provide a default to a ValueProvider and override it as part of the options passed to the template?
Any advice on how to proceed would be most appreciated. Thanks!
The way templates are currently implemented there is no point to perform "post-template creation" but "pre-pipeline start" initialization/validation.
All of the existing validation executes during template creation. If the validation detects that there the values aren't available (due to being a ValueProvider) the validation is skipped.
In some cases it is possible to approximate validation by adding runtime checks either as part of initial splitting of a custom source or part of the #Setup method of a DoFn. In the latter case, the #Setup method will run once for each instance of the DoFn that is created. If the pipeline is Batch, after 4 failures for a specific instance it will fail the pipeline.
Another option for productionizing pipelines is to build the JAR that runs the pipeline, and have a production process that runs that JAR to initiate the pipeline.
Regarding the compile error you received -- the NestedValueProvider returns a ValueProvider -- it isn't possible to get a String out of that. You could, however, put the validation code into the SerializableFunction that is run within the NestedValueProvider.
Although I believe this will currently re-run the validation everytime the value is accessed, it wouldn't be unreasonable to have the NestedValueProvider cache the translated value.
Cobertura plugin in Jenkins has a support of ratcheting by ticking these boxes:
Health auto update
Stability auto update
When ticking this box, the coverage metric targets (in Jenkins configuration page) will be updated on every successful build:
These values will be overridden by job-dsl-plugin when seed job is triggered. How can I retain these values when my seed job is triggered?
Seems like I can't find a pretty way to do this right now, but here is my solution.
Solution
1. Execute a Groovy script and store all of the current job cobertura configuration in a JSON file.
Cobertura configuration can be retrieved like:
def coberturaPublisher = project.getPublishersList().get(CoberturaPublisher)
coberturaPublisher.**healthyTarget**.getTarget(**CoverageMetric.METHOD**)
2. job-dsl-plugin to configure cobertura by using the JSON file if it's available
job-dsl's CoberturaContext normal method can't be called here because the data represented in the first step is different with the method parameter:
80% is stored as 8000000 in the JSON file
80% must be passed in as 80 instead of 8000000 to CoberturaContext methods.
As of today, I can't simply divide the value by 100000 because the method is accepting Integer instead of double. To retain the precision of the ratcheted configuration, I have to bypass the validation by manipulating the target directly:
coberturaContext.targets = [
'METHOD': new CoberturaContext.CoberturaTarget(
targetType: CoberturaContext.TargetType.METHOD,
healthyTarget: 8000000,
unhealthyTarget: previousConfig ? previousConfig.cobertura.method.unhealthy : 0,
failingTarget: previousConfig ? previousConfig.cobertura.method.failing : 0
),
Why bother creating the JSON file while you can call Jenkins API directly?
My seed job is configured with this example here, hence I have an additional classpath used in the job configuration. When I tried to hit Jenkins API directly, I'm getting class loading issue for Cobertura plugin classes.
What do these warnings mean when running our pipeline?
1497 [main] WARN com.google.cloud.dataflow.sdk.Pipeline - Transform
AsIterable2 does not have a stable unique name. In the future, this
will prevent reloading streaming pipelines
The warning in question indicates that the specified transform -- AsIterable2 -- isn't uniquely named. A likely cause of this is that there are two applications of an AsIterable transform at the top-level.
You can get rid of the warning by using the PTransform#setName method on the transform in question.
We attempt to infer names from the class names being applied. The only times it should be necessary to set an explicit name are:
When using an anonymous PTransform or DoFn.
When the same named transform is used multiple times within the same PTransform.
Specifically, requirement #2 means that if you use a named PTransform multiple times within an outer PTransform you need to make sure that each application has a different name. For instance:
input.apply(someTransform)
.apply(View.<TableRow>asIterable().withName("iterable1"));
input.apply(someOtherTransform)
.apply(View.<TableRow>asIterable().withName("iterable2"));
instead of:
View.AsIterable<TableRow> iterable = View.TableRow>asIterable().setName("aName");
input.apply(someTransform).apply(iterable);
input.apply(someOtherTransform).apply(iterable);