I am using Dataflow flex templates and am trying to launch as a job with GPU. I am following docs here to build my template from base nvidia image: https://cloud.google.com/dataflow/docs/guides/using-gpus
I want to run the template with a GPU attached. This requires experiments:
"worker_accelerator=type:nvidia-tesla-t4l;count:1;install-nvidia-driver
"use_runner_v2"
I believe these need to be specified separately rather than as a list. Or at least I haven't found a way to do that and the docs specify two --experiment arguments. From looking at docs I also believe that I need to specify the experiments as part of the --parameters argument as there is no --experiments argument for running flex templates.
I have tried the following:
In gcloud command line:
Specified experiments argument twice under --parameters. In this case it only assigns the second specified experiment.
gcloud dataflow flex-template run "test-flex" --template-file-gcs-location=<LOCATION> --parameters=source="bigquery",experiments="worker_accelerator=type:nvidia-tesla-t4;count:1;install-nvidia-driver",experiments="use_runner_v2" --max-workers=1 --region=us-east4 --worker-zone=us-east4-b
Assigned experiments in --parameters to specify the GPU and set --additional-experiments to use_runner_v2.
gcloud dataflow flex-template run "test-flex" --template-file-gcs-location=<LOCATION> --parameters=source="bigquery",experiments="worker_accelerator=type:nvidia-tesla-t4;count:1;install-nvidia-driver" --max-workers=1 --region=us-east4 --worker-zone=us-east4-b --additional-experiments='use_runner_v2'
This caused an error:
ERROR: (gcloud.dataflow.flex-template.run) INVALID_ARGUMENT: The template parameters are invalid. Details:
experiments: Runtime parameter experiments should not be specified in both parameters field and environment field.
I can get each experiment to work separately but cannot get them to both work. Is there a simple fix for this? I haven't been able to find anything in the documentation nor figured it out myself.
Please let me know any additional information you need me to provide.
You can pass both experiments in additional-experiments:
--additional-experiments=worker_accelerator=type:nvidia-tesla-t4;count:1;install-nvidia-driver,use_runner_v2
--additional-experiments=worker_accelerator=type:nvidia-tesla-t4;count:1;install-nvidia-driver \
--additional-experiments=use_runner_v2
should both work.
Related
I am trying to configure Jenkins Build Parameter "users" ,to be passed as input to JMETER (v5.1) --> No.Of Threads using the function:${__javaScript(Math.round(${XX}))}
While executing test i am getting following error
error : caused by jdk.nashorn.internal.runtime.ParserException::1:12 Expected but found { Math.round(${__jexl())}
Use a variable to store Math.round(${. Test it with println. Then Invoke ${__javaScript(myVariable)}. Nested ${} is a bad idea, groovy interprets them.
as M Navneet Krishna said, your question is a bit succinct for us to provide better answer. But I gave you a method that should help you debug your stuff
I have an Apache Beam job running on Google Cloud Dataflow, and as part of its initialization it needs to run some basic sanity/availability checks on services, pub/sub subscriptions, GCS blobs, etc. It's a streaming pipeline intended to run ad infinitum that processes hundreds of thousands of pub/sub messages.
Currently it needs a whole heap of required, variable parameters: which Google Cloud project it needs to run in, which bucket and directory prefix it's going to be storing files in, which pub/sub subscriptions it needs to read from, and so on. It does some work with these parameters before pipeline.run is called - validation, string splitting, and the like. In its current form in order to start a job we've been passing these parameters to to a PipelineOptionsFactory and issuing a new compile every single time, but it seems like there should be a better way. I've set up the parameters to be ValueProvider objects, but because they're being called outside of pipeline.run, Maven complains at compile time that ValueProvider.get() is being called outside of a runtime context (which, yes, it is.)
I've tried using NestedValueProviders as in the Google "Creating Templates" document, but my IDE complains if I try to use NestedValueProvider.of to return a string as shown in the document. The only way I've been able to get NestedValueProviders to compile is as follows:
NestedValueProvider<String, String> pid = NestedValueProvider.of(
pipelineOptions.getDataflowProjectId(),
(SerializableFunction<String, String>) s -> s
);
(String pid = NestedValueProvider.of(...) results in the following error: "incompatible types: no instance(s) of type variable(s) T,X exist so that org.apache.beam.sdk.options.ValueProvider.NestedValueProvider conforms to java.lang.String")
I have the following in my pipelineOptions:
ValueProvider<String> getDataflowProjectId();
void setDataflowProjectId(ValueProvider<String> value);
Because of the volume of messages we're going to be processing, adding these checks at the front of the pipeline for every message that comes through isn't really practical; we'll hit daily account administrative limits on some of these calls pretty quickly.
Are templates the right approach for what I want to do? How do I go about actually productionizing this? Should (can?) I compile with maven into a jar, then just run the jar on a local dev/qa/prod box with my parameters and just not bother with ValueProviders at all? Or is it possible to provide a default to a ValueProvider and override it as part of the options passed to the template?
Any advice on how to proceed would be most appreciated. Thanks!
The way templates are currently implemented there is no point to perform "post-template creation" but "pre-pipeline start" initialization/validation.
All of the existing validation executes during template creation. If the validation detects that there the values aren't available (due to being a ValueProvider) the validation is skipped.
In some cases it is possible to approximate validation by adding runtime checks either as part of initial splitting of a custom source or part of the #Setup method of a DoFn. In the latter case, the #Setup method will run once for each instance of the DoFn that is created. If the pipeline is Batch, after 4 failures for a specific instance it will fail the pipeline.
Another option for productionizing pipelines is to build the JAR that runs the pipeline, and have a production process that runs that JAR to initiate the pipeline.
Regarding the compile error you received -- the NestedValueProvider returns a ValueProvider -- it isn't possible to get a String out of that. You could, however, put the validation code into the SerializableFunction that is run within the NestedValueProvider.
Although I believe this will currently re-run the validation everytime the value is accessed, it wouldn't be unreasonable to have the NestedValueProvider cache the translated value.
I try to test my first Dataflow app by running it in Eclipse.
When I try to pass 4 values for the arguments on "Run Configuration" on "Arguments" tab as following:
projects/poc/subscriptions/poc-TestApp1 poc myDataSet my_logs
I get the error:
Argument 'projects/poc/subscriptions/poc-TestApp1' does not begin
with '--'
adding -- to all arguments produced a different error.
Based on your question, it seems that you have custom argument parsing code in your program (I suppose you're extracting your arguments as args[0], args[1] etc. in your main() function?), but still use PipelineOptionsFactory.fromArgs(args) to configure the options for Dataflow itself.
Dataflow does not support this mixed way of specifying command-line arguments - you need to define your own PipelineOptions to represent your configuration parameters, and specify them prefixed with --.
Please see here for details, in particular here for creating custom options.
The maven release plugin in Jenkins prompts you for input when doing a release:prepare. It'll give you a suggested value, but you also have the choice to override that suggested value to put your own value. How can you do this same thing when writing a Jenkins pipeline in groovy? The only thing that I've found online is -> https://gist.github.com/cyrille-leclerc/552e3103139557e0196a. This method is strictly command and just gives batch mode with no user interaction.
The parameter definitions passed to the input step can specify default values. Try it in Snippet Generator.
I've been using the option Restrict matrix execution to a subset from the Parameterized Trigger Plugin to pass on a combination filter to a rather large Matrix Project where all test execution is made. As the number of tests grow, so does the combination filter (which is dynamically built up) and I seemed to hit the cap. The following job gets this error message:
FATAL: Invalid method Code length 69871 in class file Script1
java.lang.ClassFormatError: Invalid method Code length 69871 in class file Script1
After reading about this problem, it seems to be a JVM constraint after reading the JVM documentation
The value of the code_length item must be less than 65536.
I get the impression that this is not something I can (or even should) tinker with in Jenkins.
My second idea was to go around this problem was to create the combination filter and then pass it as String parameter to the following Matrix Project, then use the Combination Filter option and expand the variable to achieve the same result.
Unfortunately I get this exception when trying to save my Matrix Project with a String parameter as combination filter
javax.servlet.ServletException: groovy.lang.MissingPropertyException: No such property: $COMBINATION_FILTER for class: groovy.lang.Binding
I guess this is because the variable needs to be available in the configuration when saving but I want to inject it when starting the Matrix Project.
I am running out of ideas to solve this problem. Any ideas?
You could try the Matrix Groovy Execution Strategy which is like a super combination filter
If I can quote myself
A plugin to decide the execution order and valid combinations of
matrix projects.
This uses a user defined groovy script to arrange the order which will
then be executed
Disclaimer: I built this plugin