I'm new to Jenkins. I have a task where I need to create a Jenkins job to automate builds of certain projects. The build job parameters are going to be stored in the SQL database. So, the job would keep querying the DB and it has to load data from the DB and perform the build operation.
Examples would be greatly appreciated.
How can this be done?
You have to transform the data from available source to the format expecting by the destination.
Here your source data available in DB and you want to use this data in Jenkins.
There might be numerous ways but the efficient way of reading data is using EnvyInJect Plugin.
If you were able to provide the whole data as Properties file format and type to EnvyInject plugin, the whole data is available as environment variables you can use these variable in the Jobs configuration.
EnvyInject Plugin can read this properties file from the Jenkins Job Workspace. And you can provide that file path in Properties File Path input.
To read the data from source and make available as properties file.
Either you can write a executable or if your application provides api to download the properties data.
Both ways to be executed before the SCM step, for this you have to use Pre-SCM-Step
Get the data and inject the data in pre-scm-step only, so that the data available as environment variables.
This is one thought to give gist for you to start. while implementing you may get lot of thoughts to implement according to your requirement.
Related
I am working with the Jenkins API JSON.
I understand the format to retrieve API data in JSON
<Jenkins_URL>/job/<job_name>/api/json
Within the job/<job_name>/configure UI we can configure/add Build triggers, build env, and build data.
I want to be able to view the Build, Build Env, and Build Triggers data in a JSON API.
Is it even possible to get said data? What are alternative ways to get all available data that is found in the configure page of a job?
I think the most straightforward way is to access <Jenkins_URL>/job/<job_name>/config.xml.
Yes, it's not JSON, but you can be sure that this contains everything that was configured on the configuration page.
The XML file is the "native" serialized version of the Job configuration. The JSON API will always require some additional glue that may exist or not exist.
I'm new on Apache Beam and I'm developing a pipeline to get rows from JDBCIO and send them to BigQueryIO. I'm converting the rows to avro files with withAvroFormatFunction but it is creating a new file for each row returned by JDBCIO. The same for withFormatFunction with json files.
It is so slow to run locally with DirectRunner because it uploads a lot of files to Google Storage. Is this approach good for scaling on Google Dataflow? Is there a better way to deal with it?
Thanks
In BigqueryIO there is an option to specify withNumFileShards which controls the number of files that get generated while using Bigquery Load Jobs.
From the documentation
Control how many file shards are written when using BigQuery load jobs. Applicable only when also setting withTriggeringFrequency(org.joda.time.Duration).
You can set test your process by setting the value to 1 to see if only 1 large file gets created.
BigQueryIO will commit results to BigQuery for each bundle. The DirectRunner is known to be a bit inefficient about bundling. It never combines bundles. So whatever bundling is provided by a source is propagated to the sink. You can try using other runners such as Flink, Spark, or Dataflow. The in-process open source runners are about as easy to use as the direct runner. Just change --runner=DirectRunner to --runner=FlinkRunner and the default settings will run in local embedded mode.
When I create a jenkins job via the API, I use a previous job's config.xml, make my modifications and then make the POST call to create the job.
My questions is, is there a way to generate this programmatically? I.e. is there a structure of a config.xml, what XML entities it should have, what values, etc so I can write a small module to generate one and send it to the jenkins API call?
I don't think there's any mandatory XML entities. Submitting an empty structure should result in a job that has default values for all settings.
What you want to do is exactly what's done by the Jenkins Job Builder. It provides a YAML-based framework for creating Job configuration XML files and submitting them to Jenkins. It's a common alternative to the Job DSL plugin. I wouldn't recommend to re-implement such a solution yourself -- handling all the plugin-specific XML configuration parts will be a nightmare.
We create our jobs using Job DSL plugin. You can try the playground http://job-dsl.herokuapp.com/.
At first, it seems that it is hard to learn, but after the first seed job, it is much better.
When we started writing our scripts we were afraid that there will not be suitable API methods for our needs. It turned out that we had one such case, which was solved using the configure block.
Get started guide here.
Right now I'm using EnvInject plugin to insert my environment variables through a properties file into my Jenkins job.
However, now I have a second job which needs the same environment variables as the first job and than some more additional variables which I would like to load via another properties file.
I know, there is a possibility to insert the values via Properties Content Edit field of the EnvInject-plugin, but I would like to keep it in a file, so it can be shared between jobs. But there seems to be no possibility to add a second properties file to EnvInject-plugin.
Is there any way to inject more than one properties file into a job or any other plugin, that could handle my scenario?
There is a simple way to get around the limitation you have.
You should load each file in the Build section, as a build step.
Use the Inject environment variables build step, and load each file you want. You can add multiple files by setting up multiple build steps of this type.
This works well for me on a similar need.
You can use Config File Provider Plugin to config some shell scripts.
You can add multiple files and then execute them.
My question may be silly but I've been trying several ways and I still can't do what I want, i.e.:
use the scp target of Ant to target a remote machine and execute
a script there
this script creates a dynamic list of files
get this list of files (only their names) back in Hudson to use it in the next build step (another scp from Ant)
I tried to use environment variables but they are interpreted by Hudson so I'm stuck here...
Globally my question would be: how to get a result from an Ant build step ?
Thanks for your ideas,
Emmanuel
You may find File parameter useful. This allows you to create an input file, pass it to build. You may need to write script/ant script to process the file though.
In the long term you may evaluate a Hudson farm. This will allow to create tasks that span multiple machines , pass results around. (https://wiki.jenkins-ci.org/display/JENKINS/Plugins)
You can get the ID(s) of the job that triggered your job via the API and fetch their status.