I have python script that creates dataflow template in the specified GCS path. I have tested the script using my GCP Free Trial and it works perfect.
My question is using same code in production environment I want to generate a template but I can not use Cloud-Shell as there are restrictions also can not directly run the Python script that is using the SA keys.
Also I can not create VM and using that generate a template in GCS.
Considering above restrictions is there any option to generate the dataflow template.
Using dataflow flex templates should obviate the need to automatically generate templates--instead you could create a single template that can be parameterized arbitrarily.
Using Composer, I have triggered the Dataflow DAGs which created Jobs in dataflow. Also managed to generate Dataflow template. Using Dataflow console & template executed the job
Related
Intellij's SwaggerHub Smartbear plugin has functionality 'Export self-contained specification(YAML)' which will resolve all the remote $ref(s) and generate local
definitions of domain models in the generated output document.
Is there a way to do it programmatically for making it as part of build pipeline? Any java program or scripts can we write for the same?
I use Bazel to build my Beam pipeline. The pipeline works well using the DirectRunner, however, I have some trouble managing dependencies when I use DataflowRunner, Python can not find local dependencies (e.g. generated by py_library) in DataflowRunner. Is there any way to hint Dataflow to use the python binary (py_binray zip file) in the worker container to resolve the issue?
Thanks,
Please see here for more details on setting up dependencies for Python SDK on Dataflow. If you are using a local dependency, you should probably look into developing a Python package and using the extra_package option or developing a custom container.
I am trying to create a dataflow template and run it via the DataFlow Cloud UI, and after executing the pipeline via command-line with the dataflow runner, it works correctly (i.e. the right data appears in the right places) but there are no "pre-compiled" template/staging files appearing in the Google Cloud Storage Bucket.
I did see this, but the post never mentions a resolution and I did include the parameter mentioned therein.
My command to run is python apache_beam_test.py --runner DataflowRunner --project prototypes-project --staging_location gs://dataflow_templates/staging --temp_location gs://dataflow_templates/temp --template_location gs://dataflow_templates/
I do get a warning regarding the options however:
C:\Python38\lib\site-packages\apache_beam\io\gcp\bigquery.py:1677: BeamDeprecationWarning: options is deprecated since First stable release. References to .options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
C:\Python38\lib\site-packages\apache_beam\io\gcp\bigquery_file_loads.py:900: BeamDeprecationWarning: options
is deprecated since First stable release. References to .options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
Does that mean my command-line arguments will not be interpreted and if so, how do I get the DataFlow/Beam templates into my GCS so I can reference them from the DataFlow UI and run them again later on?
Help much appreciated!
The problem was indeed that the cli flags needed to be explicitly passed into the pipeline options.
As I did not have any custom flags added to my project, I wrongly assumed Beam would handle the standard flags automatically, but this was not the case.
Basically, you have to follow this even if you have no new parameters to add.
I assumed that step is optional (which it technically is if you only want to execute a pipeline without any runtime parameters once), but in order to reuse and monitor the pipelines in Dataflow UI you have to stage them first. That in turn requires passing a staging location into the pipeline.
Also, as far as I understand, execution of the pipeline requires a service account, while uploading the staging files requires Google Cloud SDK authentication.
In Jenkins, I have configured multiple JMeter scripts using maven-jmeter-plugin in single job. After execution, each JMeter script is generating its own report file(.jtl) in results folder.
I have also configured the Performance plugin of Jenkins to display the report after JMeter script execution.
My problem is, the Performance plugin is showing different graph for each .jtl file.
Current Graph
I need to report to be shown in single graph.
Expected Graph
Is there any plugin available to achieve the same?
if no, how can I update the current plugin to achieve merged graph?
There is Merge Results plugin which can be used for combining/comparing results of up to 4 different JMeter tests.
You can install the plugin using JMeter Plugins Manager
Merge Results tool can also be used in command-line
Amend merge-results.properties (by default it's located in "bin" folder of your JMeter installation), make sure it contains these lines:
inputJtl1=TestResult_1.jtl
inputJtl2=TestResult_2.jtl
Execute Merge Results using JMeter Plugins Command Line Tool like:
JMeterPluginsCMD --tool Reporter --generate-csv CombinedResult.jtl --input-jtl merge-results.properties --plugin-type MergeResults
Configure Jenkins Performance Plugin to read CombinedResult.jtl file instead of your 2 files.
I use a shell script to create/run doxygen doxyfile to document my code base
which works absolutely fine(Schedule runs and recursive scan code base also
works fine).
Now my requirement is to do the same job using Jenkins CI.
I added doxygen plug which generates documentation output and stores the result in Jenkin workspace.
My question, is there any another ways to run script and generate doxyfile in
Jenkins environment and also
How to create url link to display doxygen HTML output.
Have you seen the Jenkins DocLink plugin? This plugin makes it easy to put a link on your project page to documentation generated in a build.