I posted "How to set chef attributes via Jenkins?" which was answered correctly, that is, using the "-j" option. However, what if I want to set the load version in the data bag in Jenkins so ALL cookbooks can use it? That is I don't want to use the "-j" option and instead search the value in the data bag? How do I do I set chef data bag values via Jenkins
Best way: use knife commands to upload predfined or generated databags.
knife data bag from file BAG json_file_for_item
the file must have a defined format, extended documentation is HERE
There's requirements on the file system hierarchy and on the file format, copying the doc here sounds a bad idea.
In addition to knife, you can also write your own scripts using the Chef REST API. There are clients for Ruby (Chef-API), Python (PyChef), JavaScript (chef-js), and many others.
Related
I'm new on Apache Beam and I'm developing a pipeline to get rows from JDBCIO and send them to BigQueryIO. I'm converting the rows to avro files with withAvroFormatFunction but it is creating a new file for each row returned by JDBCIO. The same for withFormatFunction with json files.
It is so slow to run locally with DirectRunner because it uploads a lot of files to Google Storage. Is this approach good for scaling on Google Dataflow? Is there a better way to deal with it?
Thanks
In BigqueryIO there is an option to specify withNumFileShards which controls the number of files that get generated while using Bigquery Load Jobs.
From the documentation
Control how many file shards are written when using BigQuery load jobs. Applicable only when also setting withTriggeringFrequency(org.joda.time.Duration).
You can set test your process by setting the value to 1 to see if only 1 large file gets created.
BigQueryIO will commit results to BigQuery for each bundle. The DirectRunner is known to be a bit inefficient about bundling. It never combines bundles. So whatever bundling is provided by a source is propagated to the sink. You can try using other runners such as Flink, Spark, or Dataflow. The in-process open source runners are about as easy to use as the direct runner. Just change --runner=DirectRunner to --runner=FlinkRunner and the default settings will run in local embedded mode.
I would like to update a chef role attribute from Jenkins.
The knife command:
Knife role edit role-name
Appears to open a text file from a defined editor where changes can be made and then saved. This doesn’t seem like a feasible solution from Jenkins. Are there any options for achieving this?
You can manage your roles using "role cookbooks" and then manage the role attributes using the recipe therein from jenkins. Refer to this link for "role cookbooks"
i strongly advise you not to let the build server (jenkins in your case) to push any changes to the repository. the repository is a source of truth, don't let any automation break\modify it.
to my understanding, it feels like there is some other pretty solution.
note that, chef-client has support for in-line attributes (--json-attributes), which you can utilize to override an attribute and make decisions in your recipe based on the value of that attribute.
Thanks for your responses. The solution I went with was to call
knife role show role-name
persist the output and update the attributes I wanted to modify using jq. Then, update the role using the updated json using
knife role from file FILE
I hope this help someone else in the future.
I am trying to figure out if there is a difference between the two pre-defined variables in TFS Online 2017: $(Build.Repository.LocalPath) and $(Build.SourcesDirectory). I have a build that uses these two variables and didn't know if I could use them interchangeably or not.
Looking at Microsoft's documentation the descriptions are as follows:
$(Build.SourcesDirectory): The local path on the agent where your source code files are downloaded. For example: c:\agent_work\1\s
By default, new build definitions update only the changed files. You can modify how files are downloaded on the Repository tab.
$(Build.Repository.LocalPath): The local path on the agent where your source code files are downloaded. For example: c:\agent_work\1\s
By default, new build definitions update only the changed files. You can modify how files are downloaded on the Repository tab.
Are these representing the same thing or am I missing something?
They're synonyms. Most standard templates and tasks use the $(Build.SourcesDirectory), so that is what I tend to use.
They often result in the same but not necessarily. As described in the docs:
If you check out multiple repositories, the behavior is as follows (and might differ from the value of the Build.SourcesDirectory variable):
The description for Build.SourcesDirectory on the same page contains a similar note.
Basically if you want to define a custom path for the self checkout and still not need to specify the extra dir, you specifically need Build.Repository.LocalPath.
For clarity, you can still use Build.SourcesDirectory to resolve to the full path if you have the usual
- checkout: self
path: s
and I'd recommend using it whenever possible if so. If you have something like
- checkout: self
path: main_project
then you'd need $(Agent.BuildDirectory)/main_project to reach the same.
I'm new to Jenkins. I have a task where I need to create a Jenkins job to automate builds of certain projects. The build job parameters are going to be stored in the SQL database. So, the job would keep querying the DB and it has to load data from the DB and perform the build operation.
Examples would be greatly appreciated.
How can this be done?
You have to transform the data from available source to the format expecting by the destination.
Here your source data available in DB and you want to use this data in Jenkins.
There might be numerous ways but the efficient way of reading data is using EnvyInJect Plugin.
If you were able to provide the whole data as Properties file format and type to EnvyInject plugin, the whole data is available as environment variables you can use these variable in the Jobs configuration.
EnvyInject Plugin can read this properties file from the Jenkins Job Workspace. And you can provide that file path in Properties File Path input.
To read the data from source and make available as properties file.
Either you can write a executable or if your application provides api to download the properties data.
Both ways to be executed before the SCM step, for this you have to use Pre-SCM-Step
Get the data and inject the data in pre-scm-step only, so that the data available as environment variables.
This is one thought to give gist for you to start. while implementing you may get lot of thoughts to implement according to your requirement.
I am trying to manage a jenkins build where a user may select to build either from master, or a specific branch. There are several variables that need to be set depending on what the user chooses. I've spent all day trying to find any reasonable way of doing this, and the best thing I've come up with is offering the user two dropdowns populated with low level parameter values.
I think you can just set up two jobs by using different branch, that will be better, since different branch might will has different datamodel, setup a different schema for each of them will make more sense.
You could use the environment script plugin and execute a shell script to define and populate your build variables.
Have a look at the Extensible Choice Parameter Plugin. The global choice parameter might be exactly what you want.
You make a single choice parameter and than add "Inject environment variables" step specifying MyBuildConfig\${ChosenParameter}.prop in Properties File Path. Create such a file for each of the choice options. Finally, specify additional variables you want to set together with particular chosen parameter in each of the created files.