I have a yaml file with the following structure:
transfers:
- name: xyz
cloud: aws
subheading:
impact: Low
reason: ---
artifacts:
- name: name1
type: type1
source:
hash: a1b2C3dd4 ---> VALUE TO OVERWRITE
I would like to overwrite the existing hash value with a value of the latest GIT_COMMIT.
I have tried the method from the following question: write yaml file in jenkins with groovy. However, the value of hash[0][0] remains unchanged. This is the case even when I replace env.GIT_COMMIT with a test hash string "testHash123". I'm unsure why this is the case?
def filename = ('path/to/file.yaml')
def datas = readYaml file: filename
//change hash
datas.transfers['artifacts'].source.hash[0][0] = env.GIT_COMMIT
writeYaml file: filename, data: datas, overwrite: true
Please try the following.
datas.transfers[0]['artifacts'][0]['source'].hash = env.GIT_COMMIT
The easiest way to figure this out is by printing, so you can understand the structure.
[transfers:[[name:xyz, cloud:aws, subheading:[impact:Low, reason:xxxx], artifacts:[[name:name1, type:type1, source:[hash:a1b2C3dd4]]]]]]
As you can see above the transfer is a sequence, so you need to extract the correct segment with an index.
Related
I have a yaml file(config.yaml) with tags/structure similar to what is mentioned below. I need to add a new tenant(tenant3) to the list of the existing tenants. How do I achieve it using the pipeline/groovy script? Any help/lead would be appreciated.
consumer_services:
- security
- token
id: 10000
tenants:
tenant_1:
state: all
web_token: true
cluster_pairs:
- cluster1
datacenter: local
client: CLIENT_TEST
tenant_2:
state: all
web_token: true
cluster_pairs:
- cluster2
datacenter: local
client: CLIENT_TEST
base_network:
subnets:
- 10.160.10.10
- 10.179.1.09
I think you need to do something like that:
#Grab('org.yaml:snakeyaml:1.17')
import org.yaml.snakeyaml.DumperOptions
import org.yaml.snakeyaml.Yaml
def options = new DumperOptions()
options.setDefaultFlowStyle(DumperOptions.FlowStyle.BLOCK)
Yaml yaml = new Yaml(options)
// load existing structure
def structure = yaml.load(new File("original.yml").text)
// modify the structure
structure.tenants.tenant_3 =
[
state : 'all',
web_token : true,
cluster_pairs: ['cluster3'],
datacenter : 'local',
client : 'CLIENT_TEST'
]
// save to the new file
new File("modified.yml").write(yaml.dump(structure))
So the steps are:
load the data from file into the memory
modify the structure in a way you like
store the modified structure to the file
I hope it will help.
I'm using OpenApi 3. A tool I use, Owasp Zap looks at the OpenAPI doc and creates fake requests. When it gets a 404, it complains that it doesn't have the media type that the OpenAPI promises.
But I didn't write anything in the OpenAPI doc about how 404s are handled. Obviously I can't write an infinite number of bad end points & document that they return 404s.
What is the right way to record this in the OpenAPI yaml or json?
Here is a minimal yaml file... I know for sure that this file does say anything about 404, ie. 404s aren't in the contract so tools are complaining that 404s are valid responses, but 404 is what a site should return when a resource is missing
---
"openapi": "3.0.0"
paths:
/Foo/:
get:
responses:
"200":
content:
application/json:
schema:
$ref: "#/components/schemas/Foo"
default:
description: Errors
content:
application/json:
schema:
$ref: "#/components/schemas/Error"
components:
schemas:
Foo:
type: object
required:
- name
properties:
name:
type: string
Error:
type: object
required:
- error
properties:
error:
type: string
message:
type: string
data:
type: object
This has been proposed already but not implemented: https://github.com/OAI/OpenAPI-Specification/issues/521
In the comments someone gave a suggestion: https://github.com/OAI/OpenAPI-Specification/issues/521#issuecomment-513055351, which reduces a little your code, but you would still have to insert N*M entries for N paths * M methods.
Since we don't have the ability to make the specification change to our needs, all that remains is we adapting ourselves.
From your profile, you seem to be a windows user. You can for example, create a new explorer context menu to your .yaml files (Add menu item to windows context menu only for specific filetype, Adding a context menu item in Windows for a specific file extension), and make it run a script that auto-fills your file.
Here, an example python script called yamlfill404.py that would be used in the context call in a way like path/to/pythonexecutable/python.exe path/to/python/script/yamlfill404.py %1, where %1 is the path to the file being right clicked.
Python file:
import yaml
from sys import argv
import re
order = ['openapi','paths','components']
level0re = re.compile('(?<=\n)[^ ][^:]+')
def _propfill(rootnode, nodes, value):
if len(nodes) == 1:
rootnode[nodes[0]] = value
if len(nodes) > 1:
nextnode = rootnode.get(nodes[0])
if rootnode.get(nodes[0]) is None:
nextnode = {}
rootnode[nodes[0]] = nextnode
_propfill(nextnode, nodes[1:], value)
def propfill(rootnode, nodepath, value):
_propfill(rootnode, [n.replace('__slash__','/') for n in nodepath.replace('\/','__slash__').split('/')], value)
def yamlfill(filepath):
with open(filepath, 'r') as file:
yamltree = yaml.safe_load(file)
#propfill(yamltree, 'components/schemas/notFoundResponse/...', '')
propfill(yamltree, 'components/responses/notFound/description', 'Not found response')
propfill(yamltree, 'components/responses/notFound/content/application\/json/schema/$ref', '#/components/schemas/notFoundResponse')
responses = [mv['responses'] if 'responses' in mv else [] for pk,pv in (yamltree['paths'].items() if 'paths' in yamltree else []) for mk,mv in pv.items()]
for response in responses:
propfill(response, '404/$ref', '#/components/responses/notFound')
yamlstring = yaml.dump(yamltree)
offsets = [i[1] for i in sorted([(order.index(f.group(0)) if f.group(0) in order else len(order),f.start()-1) for f in [f for f in level0re.finditer('\n'+yamlstring)]])]
offsets = [(offset,(sorted([o for o in offsets if o > offset]+[len(yamlstring)-1])[0])) for offset in offsets]
with open(filepath[:-5]+'_404.yaml', 'w') as file:
file.write(''.join(['\n'+yamlstring[o[0]:o[1]] for o in offsets]).strip())
yamlfill(argv[-1])
It processes the %1, which would be path/to/original.yaml and saves it as path/to/original_404.yaml (but you can change it to overwrite the original).
This example script changes the yaml formating (quotes type, spacing, ordering etc), because of the library used pyyaml. I had to reorder the file with the order = ['openapi','paths','components'], because it loses ordering. For less instrusion, maybe a more manual insertion would be better suited. Maybe one that uses only regex. Maye using awk, there are plenty of ways.
Unfortunately it is just a hack not not a solution.
I've some YAML files which I wanted to apply to create Custom Resources. But before applying it I want to change the spec and ENVs of the YAML snippet. So what could be the best way to do this?
What I'm doing now is:
Let suppose this is the YAML
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: nginx-chaos
namespace: default
spec:
appinfo:
appns: 'default'
applabel: 'app=nginx'
appkind: 'deployment'
# It can be true/false
annotationCheck: 'false'
# It can be active/stop
engineState: 'active'
chaosServiceAccount: pod-delete-sa
monitoring: false
# It can be delete/retain
jobCleanUpPolicy: 'delete'
experiments:
- name: pod-delete
spec:
components:
env:
# set chaos duration (in sec) as desired
- name: TOTAL_CHAOS_DURATION
value: '30'
# set chaos interval (in sec) as desired
- name: CHAOS_INTERVAL
value: '10'
# pod failures without '--force' & default terminationGracePeriodSeconds
- name: FORCE
value: 'false'
I Download this file from the raw link. (1 function for this)
Replace a field like jobCleanUpPolicy: 'delete' to jobCleanUpPolicy: 'retain' (1 fn)
Replace the next line when a match is found like value: 'false' to value: 'true' for FORCE ENV. (1 fn)
Applying the final manifest created! (1 function)
Can this be optimized?
If you create a struct representing the resource (or, even better, can import the package which defines the CRD) you could take the yaml string, marshal it into the struct and then edit the fields directly as fields of a struct instead
Hi You may use LitmusPortal to use this.Refer
Are variables within YAML files possible? For example:
theme:
name: default
css_path: compiled/themes/$theme.name
layout_path: themes/$theme.name
In this example, how can theme: name: default be used in other settings? What is the syntax?
I had this same question, and after a lot of research, it looks like it's not possible.
The answer from cgat is on the right track, but you can't actually concatenate references like that.
Here are things you can do with "variables" in YAML (which are officially called "node anchors" when you set them and "references" when you use them later):
Define a value and use an exact copy of it later:
default: &default_title This Post Has No Title
title: *default_title
{ or }
example_post: &example
title: My mom likes roosters
body: Seriously, she does. And I don't know when it started.
date: 8/18/2012
first_post: *example
second_post:
title: whatever, etc.
For more info, see this section of the wiki page about YAML: http://en.wikipedia.org/wiki/YAML#References
Define an object and use it with modifications later:
default: &DEFAULT
URL: stooges.com
throw_pies?: true
stooges: &stooge_list
larry: first_stooge
moe: second_stooge
curly: third_stooge
development:
<<: *DEFAULT
URL: stooges.local
stooges:
shemp: fourth_stooge
test:
<<: *DEFAULT
URL: test.stooges.qa
stooges:
<<: *stooge_list
shemp: fourth_stooge
This is taken directly from a great demo here: https://gist.github.com/bowsersenior/979804
After some search, I've found a cleaner solution wich use the % operator.
In your YAML file :
key : 'This is the foobar var : %{foobar}'
In your ruby code :
require 'yaml'
file = YAML.load_file('your_file.yml')
foobar = 'Hello World !'
content = file['key']
modified_content = content % { :foobar => foobar }
puts modified_content
And the output is :
This is the foobar var : Hello World !
As #jschorr said in the comment, you can also add multiple variable to the value in the Yaml file :
Yaml :
key : 'The foo var is %{foo} and the bar var is %{bar} !'
Ruby :
# ...
foo = 'FOO'
bar = 'BAR'
# ...
modified_content = content % { :foo => foo, :bar => bar }
Output :
The foo var is FOO and the bar var is BAR !
This is an old post, but I had a similar need and this is the solution I came up with. It is a bit of a hack, but it works and could be refined.
require 'erb'
require 'yaml'
doc = <<-EOF
theme:
name: default
css_path: compiled/themes/<%= data['theme']['name'] %>
layout_path: themes/<%= data['theme']['name'] %>
image_path: <%= data['theme']['css_path'] %>/images
recursive_path: <%= data['theme']['image_path'] %>/plus/one/more
EOF
data = YAML::load("---" + doc)
template = ERB.new(data.to_yaml);
str = template.result(binding)
while /<%=.*%>/.match(str) != nil
str = ERB.new(str).result(binding)
end
puts str
A big downside is that it builds into the yaml document a variable name (in this case, "data") that may or may not exist. Perhaps a better solution would be to use $ and then substitute it with the variable name in Ruby prior to ERB. Also, just tested using hashes2ostruct which allows data.theme.name type notation which is much easier on the eyes. All that is required is to wrap the YAML::load with this
data = hashes2ostruct(YAML::load("---" + doc))
Then your YAML document can look like this
doc = <<-EOF
theme:
name: default
css_path: compiled/themes/<%= data.theme.name %>
layout_path: themes/<%= data.theme.name %>
image_path: <%= data.theme.css_path %>/images
recursive_path: <%= data.theme.image_path %>/plus/one/more
EOF
This is how I was able to configure yaml files to refer to variable.
I have values.yaml where we have root level fields which are used as template variables inside values.yaml
values.yaml
.....
databaseUserPropName: spring.datasource.username
databaseUserName: sa
.....
secrets:
type: Opaque
name: dbservice-secrets
data:
- name: "{{ .Values.databaseUserPropName }}"
value: "{{ .Values.databaseUserName }}"
.....
When referencing these values in secret.yaml, we would use tpl function using syntax {{ tpl TEMPLATE_STRING VALUES }}
secret.yaml
when using inside range i:e iteration
{{ range .Values.deployments.secrets.data }}
{{ tpl .name $ }}: "{{ tpl .value $ }}"
{{ end }}
when directly referring as variable
{{ tpl .Values.deployments.secrets.data.name . }}
{{ tpl .Values.deployments.secrets.data.value . }}
$ - this is global variable and will always point to the root context
. - this variable will point to the root context based on where it used.
if your requirement is like parsing an replacing multiple variable and then use it as a hash/or anything then you can do something like this
require 'yaml'
require 'json'
yaml = YAML.load_file("xxxx.yaml")
blueprint = yaml.to_json % { var_a: "xxxx", var_b: "xxxx"}
hash = JSON.parse(blueprint)
inside the yaml just put variables like this
"%{var_a}"
Rails / ruby frameworks are able to do some templating ... it's frequently used to load env variables ...
# fooz.yml
foo:
bar: <%= $ENV[:some_var] %>
No idea if this works for javascript frameworks as I think that YML format is superset of json and it depends on what reads the yml file for you.
If you can use the template like that or the << >> or the {{ }} styles depending on your reader, after that you just ...
In another yml file ...
# boo.yml
development:
fooz: foo
Which allows you to basically insert a variable as your reference that original file each time which is dynamically set. When reading I was also seeing you can create or open YML files as objects on the fly for several languages which allows you to create a file & chain write a series of YML files or just have them all statically pointing to the dynamically created one.
I need to load a yaml file into Hash,
What should I do?
I would use something like:
hash = YAML.load(File.read("file_path"))
A simpler version of venables' answer:
hash = YAML.load_file("file_path")
Use the YAML module:
http://ruby-doc.org/stdlib-1.9.3/libdoc/yaml/rdoc/YAML.html
node = YAML::parse( <<EOY )
one: 1
two: 2
EOY
puts node.type_id
# prints: 'map'
p node.value['one']
# prints key and value nodes:
# [ #<YAML::YamlNode:0x8220278 #type_id="str", #value="one", #kind="scalar">,
# #<YAML::YamlNode:0x821fcd8 #type_id="int", #value="1", #kind="scalar"> ]'
# Mappings can also be accessed for just the value by accessing as a Hash directly
p node['one']
# prints: #<YAML::YamlNode:0x821fcd8 #type_id="int", #value="1", #kind="scalar">
http://yaml4r.sourceforge.net/doc/page/parsing_yaml_documents.htm
You may run into a problem mentioned at this related question, namely, that the YAML file or stream specifies an object into which the YAML loader will attempt to convert the data into. The problem is that you will need a related Gem that knows about the object in question.
My solution was quite trivial and is provided as an answer to that question. Do this:
yamltext = File.read("somefile","r")
yamltext.sub!(/^--- \!.*$/,'---')
hash = YAML.load(yamltext)
In essence, you strip the object-classifier text from the yaml-text. Then you parse/load it.