I am configuring my Jenkins instance using jenkins-helm chart (https://github.com/jenkinsci/helm-charts/blob/main/charts/jenkins/VALUES_SUMMARY.md#jenkins-configuration-as-code-jcasc)
Currently Jenkins config is provided in values.yaml as:
jenkins:
controller:
JCasC:
configScripts:
key1:|-
<a-very-big-yaml-value>
Is there a way to import this 'big-yaml-value' from separate yaml file, as it will enhance maintainability of code for us.
As I don't use the helm-charts,I can't answer authoritatively, but it is supported in the abstract. According to the JCasC Getting Started documentation:
First, start a Jenkins instance with the Configuration as Code plugin installed.
Those running Jenkins as a Docker container (and maybe also pre-installing plugins), do include Configuration as Code plugin.
Second, the plugin looks for the CASC_JENKINS_CONFIG environment variable. The variable points to a comma-separated list of any of the following:
Path to a folder containing a set of config files. For example, /var/jenkins_home/init.CasC.
A full path to a single file. For example, /var/jenkins_home/init.CasC/jenkins.yaml.
A URL pointing to a file served on the web. For example, https://acme.org/jenkins.yaml.
If an element of CASC_JENKINS_CONFIG points to a folder, the plugin will recursively traverse the folder to find file(s) with .yml,.yaml,.YAML,.YML suffix. It will exclude hidden files or files that contain a hidden folder in any part of the full path. It follows symbolic links for both files and directories.
So, yes, you can have multiple yml files. I have over 20 (for 120 plugins). They are broken down by capability (eg: global, agents, tools, credentials , including 2 for RBAC (1 for roles, for users, etc.)), plus some plugin specific yml files. Some are also reused across instances while others are specific.
You should be aware of Merge Strategy in the event of conflicts:
ErrorOnConflictMergeStrategy (default)
The strategy name is errorOnConflict.
Throws an exception if there's a conflict in multiple YAML files.
OverrideMergeStrategy
The strategy name is override
Override the config files according to the loading order.
Also be aware when updating an existing instance, certain plugin configurations may replace configurations, while others may augment an existing configuration, regardless of one yaml or many. And of course, not 100% of options are JCasC-able yet, so some init.groovy is also required. YMMV.
You may also wish to review: JCasC Handling Secrets.
The setup below worked for me. Will put the relevant parts.
Directory layout for the helm chart:
jenkins/
├── conf/
│ ├── shared-library.yaml
│ └── big-yaml.yaml
├── templates/
│ └── jenkins-custom-casc-config.yaml
├── values.yaml
└── Chart.yaml
In the values.yaml, we override the CASC_JENKINS_CONFIG so it takes into account an additional path for config files on top of the default one.
controller:
containerEnv:
- name: CASC_JENKINS_CONFIG
value: "/var/jenkins_home/casc_configs,/var/jenkins_home/custom-casc_configs"
persistence:
volumes:
- name: jenkins-custom-casc-config
configMap:
name: jenkins-custom-casc-config
mounts:
- mountPath: /var/jenkins_home/custom-casc_configs
name: jenkins-custom-casc-config
ConfigMap jenkins-custom-casc-config.yaml that loads all files present in the conf/ folder
apiVersion: v1
kind: ConfigMap
metadata:
name: jenkins-custom-casc-config
data:
{{- (.Files.Glob "conf/*").AsConfig | nindent 2 }}
I got it working in when doing subfolder to /var/jenkins_home/casc_configs where I inject all config files. Otherwise HiroCereal´s answer works.
I did it following the idea of HiroCereal, but it keeps showing in the UI
Configuration loaded from :
* /var/jenkins_home/casc_configs
and the folder casc_configs has nothing.
I'm missing something? i'm using helm charts.
Related
With Bazel I'm building an external library using http_archive together with
some patches which bring additional features:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name="some-lib",
build_file="#my-project//some-lib:BUILD.some-lib",
url="https://example/download/some-lib.tar.gz",
sha256="8d9405baf113a9f25e4fb961d56f9f231da02e3ada0f41dbb0fa4654534f717b",
patches=[
"//some-lib/patches:01-add-additional-features.dif",
],
patch_args=["-p1"],
patch_tool="patch",
)
The file structure looks like this:
some-lib
├── BUILD
├── BUILD.some-lib
├── include/additional_features.h
├── some-lib.bzl
└── patches
├── 01-add-additional-features.dif
└── BUILD
This basically works but I still struggle with adding include/additional_features.h
into the extracted source folder.
I first tried to just list the file in the filegroup I use to later run
configure_make like this:
filegroup(
name="all_srcs",
srcs=glob(["**"]) + ["#my-project//some-lib:include/additional_features.h"],
)
then I'm getting
no such target '//some-lib:includes/additional_features.h': target 'includes/additional_features.h' not declared in package 'some-lib'; however, a source file of this name exists.
My next idea was to use the tools http_archive provides to make the file part of the source folder.
While you can use patches to modify the extracted folder you'd need a dedicated
dif file just to create the extra header file, which then you would have to
create in advance (i.E. create a Bazel rule) and declare it a dependency etc..
which I'd like to avoid to keep things simple.
There is also patch_cmds which next to patches can be used to modify the
extracted source folder by running arbitrary bash commands so I tried something like this:
patch_cmds=[
"cp #my-project//some-lib:include/additional_features.h include/",
],
but this does not work for me, I'm getting
Error in fail: Error applying patch command cp #my-project//some-lib:include/additional_features.h include/:
cp: cannot stat '#my-project//some-lib:include/additional_features.h': No such file or directory
So it looks like the syntax for specifying a path like I do with build_file does
not work with patch_cmds or that file can't be accessed at that specific stage.
Does one of the approaches I tried actual work and I just didn't use the right
syntax?
What's the Bazel-way to add (a bunch of) readily available files (i.e. in the same
repository as the Bazel-rules I provide) to a http_archive based source directory?
Try putting exports_files(["include/additional_features.h"], visibility=["//visibility:public"]) in some-lib/BUILD, so that Bazel will let you reference source files from the //some-lib package in the external repository (and elsewhere).
I even thought the "no such target" error message suggested exports_files?
Here is my issue: I want to include/exclude specific file when building a service called django. But when playing with Dockerfile.dockerignore file, I do not succeed what I want and I can't figure out which syntax to adopt in order to do so.
Repo structure is the following:
.
└── root_dir
├── django
│ ├── Dockerfile
│ ├── Dockerfile.ignore
│ └── stuff
├── documentation
│ └── stuff
├── useless dir
└── another useless dir
My docker-compose.yml used to make the build is the following:
version: '3.5'
services:
[...]
django:
build:
context: $ROOT_DIR
dockerfile: $ROOT_DIR/django/Dockerfile
[...]
As you can see, context is root_dir and not django (because I need to copy a few things from documentation dir at build time).
What I need when building my image:
django rep
documentation rep
What I want to ignore when building my image:
everything but django and documentation dirs
Dockerfile and Dockerfile.dockerignore
and a few other things (virtual env, *.pyc etc, but we'll stick with Dockerfile and Dockerfile.dockerignore for that question!)
What my Dockerfile.dockerignore looks like in order to do so:
# Ignore everything
*
# Allows django and documentation rep (context is root of project)
!/django
!/documentation
# Ignoring some files
**/Dockerfile
**/Dockerfile.dockerignore
but my issue is that the files I want to ignore are still present in my image! And this is not good for me. I think I tried any syntax possible (**/, */*/......) but I can't find one that suits my need.
Many thanks if you can help me with that!
EDIT: for those wondering about that Dockerfile.dockerignore file, you can have a look here: https://docs.docker.com/engine/reference/commandline/build/#use-a-dockerignore-file
Docker ignore file name should be .dockerignore
You can check the documentation here https://docs.docker.com/engine/reference/builder/#dockerignore-file
Running this command before the 'docker build' fixed it for me:
export DOCKER_BUILDKIT=1
FYI: I did not need to install 'buildkit' first.
I have a complex config search path consisting of multiple locations where each location looks similar to this:
├── conf
│ └── foo
│ ├── foo.yaml
│ └── bar.yaml
└── files
├── foo.txt
└── bar.txt
with foo.yaml:
# #package _group_
path: "../../files/foo.txt"
and bar.yaml:
# #package _group_
path: "../../files/bar.txt"
Now the problem is: how do I find the correct location of the files specified in the configurations? I am aware of the to_absolute_path() method provided by hydra, but it interprets the path relative to the directory in which the application was started. However, I would like to interpret that path relative to the position of the configuration file. I cannot do this manually in my code, because I don't know how hydra resolved the configuration file and where exactly it is used to.
Is there some mechanism to determine the location of a config file from hydra? I really want to refrain from putting hard coded absolute paths in my configurations.
You can't get the path of a config file. In fact, it may not be a file at all (such as the case for Structured Configs), or it can be inside a python wheel (even in a zipped wheel).
You can do something like
path = os.path.join(os.path.dirname(__file__), "relative_path_from_config")
You can use also APIs designed for loading resources files from Python modules.
Here is a good answer in the topic.
I am trying to generate service worker for the Polymer 3 with workbox 4.3.1.
I have some specific files inside bower and node_modules I want to cache.
I tried adding "en-in/node_modules/**" to globIgnores and include specific files like - en-in/node_modules/#webcomponents/webcomponentsjs/custom-elements-es5-adapter.*.js in globPattern.
The config I tried is giving a error. I even tried adding globStrict: false. Even that didn't help.
Below is my workbox config:
globDirectory: "dist",
globPatterns: ["en-in/**/*.{js,json,css}",
"en-in/node_modules/#webcomponents/webcomponentsjs/custom-elements-es5-adapter.*.js"],
globIgnores: [
"en-in/sw-reg.js",
"en-in/sw-custom.js",
"en-in/rev-manifest.json",
"en-in/package.json",
"en-in/workbox-v4.3.1/**/*",
"en-in/node_modules/**"
],
globStrict: false,
I am getting the below error:
One of the glob patterns doesn't match any files. Please remove or fix the following:
{
"globDirectory": "dist",
"globPattern": "en-in/node_modules/#webcomponents/webcomponentsjs/custom-elements-es5-adapter.*.js",
"globIgnores": [
"en-in/sw-reg.js",
"en-in/sw-custom.js",
"en-in/rev-manifest.json",
"en-in/package.json",
"en-in/workbox-v4.3.1/**/*",
"en-in/node_modules/**",
"**/service-worker.js"
]
}
The code that does glob-ing in Workbox looks like:
globbedFiles = glob.sync(globPattern, {
cwd: globDirectory,
follow: globFollow,
ignore: globIgnores,
strict: globStrict,
});
Because you're passing in "en-in/node_modules/**" as one of the globIgnores patterns, "en-in/node_modules/#webcomponents/webcomponentsjs/custom-elements-es5-adapter.*.js" is never going to match anything. In the glob module, ignore always takes precedence.
You have a number of approaches that would fix this:
As part of your build process, move the custom-elements-es5-adapter.*.js file out of node_modules and into a different directory, and load it from there.
Change your "en-in/**/*.{js,json,css}" in globPatterns to something like "en-in/{dir1,dir2,dir3}/**/*.{js,json,css}" so that it does not match node_modules by default, and remove "en-in/node_modules/**" from globIgnores. You can then leave "en-in/node_modules/#webcomponents/webcomponentsjs/custom-elements-es5-adapter.*.js" in globPatterns, and it will not longer be ignored.
Change your "en-in/node_modules/**" in globIgnores so that it doesn't match en-in/node_modules/#webcomponents. (I forget the syntax for doing that, but you might be able to figure it out if the other two options don't work.)
There are probably a few other alternatives as well. But hopefully that explains the root cause.
Here below the Polymer 3 notes at its page for service worker. They made easy of life :)
Service Worker
A Service Worker is loaded and registered in the index.html file. However, during development (to make debugging easier), the Service Worker does not actually exist, and only a stub file is used.
The production time Service Worker is automatically created during build time, i.e. by running npm run build or npm run build:static. This file is generated based on the polymer.json and sw-precache-config.js configuration files, and you can find it under each of the build directories:
build/
├── es5-bundled/
| └── service-worker.js
├── es6-bundled/
| └── service-worker.js
├── esm-bundled/
| └── service-worker.js
└── ...
By default, all of the source files (inside the /src directory) will be pre-cached, as specified in the sw-precache-config.js configuration file. If you want to change this behaviour, check out the sw-precache-config docs.
Source: https://pwa-starter-kit.polymer-project.org/building-and-deploying
I had the same issue and it was resolved when I explicitly changed the globPatterns to the following
globPatterns: [
'**/*.js',
'**/*.html',
'**/*.css',
'**/*.json',
'**/*.svg',
'**/*.png',
'**/*.gif',
'**/*.txt',
],
I've an external dependency declared in WORKSPACE as a new_git_repository and provided a BUILD file for it.
proj/
├── BUILD
├── external
│ ├── BUILD.myDep
│ └── code.bzl
└── WORKSPACE
in the BUILD.myDep file, I want to load code.bzl nearby, but when I load it (load("//:external/code.bzl", "some_func")) bazel tries to load #myDep//:external/code.bzl instead!
Of course it's not a target in #myDep repository, but in my local worksapce.
Seems I Rubber Duck ed the Stackoverflow. since the solution appeared when writing the question!
However, the solution is to explicitly mention the local workspace when loading the .bzl file:
Suppose we have declared the name in the WORKSPACE as below:
workspace(name = "local_proj")
Now instead of load("//:external/code.bzl", "some_func"), just load it explicitly as a local workspace file:
load("#local_proj//:external/code.bzl", "some_func")
NOTE: When using this trick just be careful about potential dependency loops (i.e. loading a generated file that itself is produced by a rule depending on the same external repo!)