In relation to Jenkins DSL, what is the difference between:
def cwd = pwd()
and
cwd = pwd()
?
It's a difference of scope. When you assign a value to a variable without a "def" or other type, in a Groovy script, it's added to the "binding", the global variables for the script. That means it can be accessed from all functions within the script. It's a lot like if you had the variable defined at the top of the script.
You could wind up with unexpected behavior if multiple threads are acting on the script.
def a = {
x = 1
println x
}
def b = {
x = 2
println x
}
new Thread(a).start()
new Thread(b).start()
... could produce two ones, two twos, or a mix.
In contrast, using "def" makes a local variable:
def a = {
def x = 1
println x
}
def b = {
def x = 2
println x
}
new Thread(a).start()
new Thread(b).start()
... will always print a 1 and a 2, in arbitrary order.
It's a good question, but it's more a Groovy question.
From what I understand, defining a variable without def keyword will work from a script, but not if you were in a class method. Example from this blog post :
class MyTest {
def testMethod() {
y = 3
println y
}
}
t = new MyTest()
t.testMethod()
Variable t will be defined without problem but y definition will throw an exception.
What it means is that in our context (Jenkins pipeline) you could always define your variable without the def keyword because you are always in a script context and your variables will be bound to the script. However, I think it is good practice to use def keyword because it shows you know when you instantiate your variables, and it also can avoid some problems of duplicate variables definitions (if you define them with the def keyword at least compilation will fail if you defined the same variable twice).
Finally, from Groovy documentation :
When using def in Groovy, the actual type holder is Object (so you can
assign any object to variables defined with def, and return any kind
of object if a method is declared returning def).
So you might want to be specific and specify the type of variable you are defining. In your case you could define cwd as :
String cwd = pwd()
It would forbid you to do things like :
def cwd = pwd()
cwd = 1000 // Valid code
String cwd2 = pwd()
cwd2 = 1000 // Will fail compilation
Related
I have a Jenkins variable BUILDVERSION_DATE in stage which is calculated and formatted correctly. Everything works perfectly.
script {
def now = new Date();
def inOneHour = new Date(now.getTime() + 1 * 3600 * 1000);
println inOneHour.format("yyyy-MM-dd-HH-mm-ss", TimeZone.getTimeZone('UTC'))
def BUILDVERSION_DATE=inOneHour.format("yyyy-MM-dd-HH-mm-ss", TimeZone.getTimeZone('UTC'))
}
Now I would like to use this calculated variable in multiple stages (without code repetition).
I have tried to put this code into environment {...} section but it fails.
If it was static variable I know I can define it in environment segment.
But how this calculated variable be defined and used in multiple stages?
Thanks!
If you want to add this value to an environment variable named BUILDVERSION_DATE, then you can assign it to the env object intrinsic to Jenkins Pipeline:
env.BUILDVERSION_DATE=inOneHour.format("yyyy-MM-dd-HH-mm-ss", TimeZone.getTimeZone('UTC'))
I have a properties file where certain string shoudl be replaced with Jenkins parameter. I ahve tried using the variable directly in the Properties file which did not work.
properties file
DOCKER_TAG_SUFFIX=-REPLACE_RELEASE_VERSION
PROPERTY_FILE_PATH=someproperty
Jenkinsfile snippet
def jboss_parameters = readProperties file: jboss_propfile
jboss_parameters .replaceAll("RELEASE_VERSION",params.RELEASE_VERSION) # try1
jboss_parameters = readFile(jboss_propfile).replaceAll("REPLACE_RELEASE_VERSION",params.RELEASE_VERSION) # try2
# try 3
jboss_parameters.each{k,v ->
if (v == "REPLACE_RELEASE_VERSION" )
jboss_parameters.setProperty($k,params.RELEASE_VERSION)
}
# try 4
def jboss_source_file = new File(jboss_propfile)
def jboss_parameters = jboss_source_file.text.replace("REPLACE_RELEASE_VERSION",params.RELEASE_VERSION)
I am not able to find another way that works for me.
println jboss_parameters output
{DOCKER_TAG_SUFFIX=-REPLACE_RELEASE_VERSION, PROPERTY_FILE_PATH=someproperty}
The readProperties step returns a dictionary (map), not a string, that is crated from the properties file.
Your first attempt (# try1) fails because maps in groovy do not have a replaceAll function like strings have and therefore you will get an error.
Your third attempt (# try3) is failing because you are comparing the map values to REPLACE_RELEASE_VERSION without the - character and therefore the comparison always fails and no values are changed.
I tested the second attempt (# try 2) and it seems to be working, so i am not sure what is your issue, but it is easier to handle properties as a map instead of a string that is retuned from the readFile method.
So if you have only specific properties that need to be updated you can update them directly:
def jboss_parameters = readProperties file: jboss_propfile
jboss_parameters.DOCKER_TAG_SUFFIX = params.RELEASE_VERSION // update relevant property
Or if you have multiple properties that should be modified you can iterate and update each value using the collectEntries method. Something like:
def jboss_parameters = readProperties file: jboss_propfile
updated_parameters = jboss_parameters.collectEntries { key, value ->
[key, value.replaceAll("REPLACE_RELEASE_VERSION",params.RELEASE_VERSION)]
}
I have a very long workflow for building and testing our application. So long, in fact, that when we try to load the main workflow script, we get this exception:
java.lang.ClassFormatError: Invalid method Code length 67768 in class file WorkflowScript
I am not proud of this. I'm tying to split the workflow into smaller scripts that we load from the main workflow script, but are running into an issue with variable scoping. For example:
def a = 'foo' //some variable referenced in multiple workflow stages
node {
echo a
}
//... and then a whole bunch of other stages
might become
def a = 'foo' //some variable referenced in multiple workflow stages
node {
git: ...
load 'flowPartA.groovy'
}()
where flowPartA.groovy looks like:
{ ->
node {
echo a
}
}
Based on my understanding of the documentation, where flowPartA.groovy is interpreted as a closure, I expect the variable 'a' would remain in scope, but instead, I get an exception to the contrary.
groovy.lang.MissingPropertyException: No such property: a for class: groovy.lang.Binding
Am I missing something about the way workflow interprets the flow scripts? Is there a good way to take a huge workflow that uses many, many parameters and split it into smaller chunks?
You have to define a function in the external groovy and call it passing all required parameters:
def a = 'foo'
node('slave') {
git '…'
def flow = load 'flowPartA.groovy'
flow.echoFromA(a)
}
And flowPartA.groovy contains:
def echoFromA(String a) {
echo a
}
return this
See the documentation for more information.
I have an external and internal config in my grails application:
Config.groovy
root = "/home/baseConf"
test {
dir = root + "/testDir"
}
External.groovy
root = "/home/externalConf"
Inside controller i have:
println "${grailsApplication.config.root}"
println "${grailsApplication.config.test.dir}"
What is printed:
/home/externalConf
/home/baseConf/testDir
What I want to be printed:
/home/externalConf
/home/externalConf/testDir
What should I do to change many variables that are using one base variable in Config.groovy by exchanging this one base variable in external config file (as in example above)? Is something like this even possible?
You need to change your dir variable (inside test). Check the code below.
test {
dir = "${-> root}/testDir"
}
This change is necessary since you want dir to be evaluated when it is called and not when Config is loaded. This is called late-binding (lazy evaluation) (see the answer of Ian Roberts here: Reusing Grails variables inside Config.groovy).
It's important to note that it is related with Groovy language (not Grails).
An Eager Evaluation Strategy can be see below:
def x = 1
def s = "The value of x is: ${x}"
println s //The value of x is: 1
x = 2
println s //The value of x is: 1
For other side, a Lazy Evaluation Strategy will evaluate the expression on-demand (call-by-need):
def x = 1
def s = "The value of x is: ${-> x}"
println s //The value of x is: 1
x = 2
println s //The value of x is: 2
I use the following code to temporarily modify environment variables.
#contextmanager
def _setenv(**mapping):
"""``with`` context to temporarily modify the environment variables"""
backup_values = {}
backup_remove = set()
for key, value in mapping.items():
if key in os.environ:
backup_values[key] = os.environ[key]
else:
backup_remove.add(key)
os.environ[key] = value
try:
yield
finally:
# restore old environment
for k, v in backup_values.items():
os.environ[k] = v
for k in backup_remove:
del os.environ[k]
This with context is mainly used in test cases. For example,
def test_myapp_respects_this_envvar():
with _setenv(MYAPP_PLUGINS_DIR='testsandbox/plugins'):
myapp.plugins.register()
[...]
My question: is there a simple/elegant way to write _setenv? I thought about actually doing backup = os.environ.copy() and then os.environ = backup .. but I am not sure if that would affect the program behavior (eg: if os.environ is referenced elsewhere in the Python interpreter).
I suggest you the following implementation:
import contextlib
import os
#contextlib.contextmanager
def set_env(**environ):
"""
Temporarily set the process environment variables.
>>> with set_env(PLUGINS_DIR=u'test/plugins'):
... "PLUGINS_DIR" in os.environ
True
>>> "PLUGINS_DIR" in os.environ
False
:type environ: dict[str, unicode]
:param environ: Environment variables to set
"""
old_environ = dict(os.environ)
os.environ.update(environ)
try:
yield
finally:
os.environ.clear()
os.environ.update(old_environ)
EDIT: more advanced implementation
The context manager below can be used to add/remove/update your environment variables:
import contextlib
import os
#contextlib.contextmanager
def modified_environ(*remove, **update):
"""
Temporarily updates the ``os.environ`` dictionary in-place.
The ``os.environ`` dictionary is updated in-place so that the modification
is sure to work in all situations.
:param remove: Environment variables to remove.
:param update: Dictionary of environment variables and values to add/update.
"""
env = os.environ
update = update or {}
remove = remove or []
# List of environment variables being updated or removed.
stomped = (set(update.keys()) | set(remove)) & set(env.keys())
# Environment variables and values to restore on exit.
update_after = {k: env[k] for k in stomped}
# Environment variables and values to remove on exit.
remove_after = frozenset(k for k in update if k not in env)
try:
env.update(update)
[env.pop(k, None) for k in remove]
yield
finally:
env.update(update_after)
[env.pop(k) for k in remove_after]
Usage examples:
>>> with modified_environ('HOME', LD_LIBRARY_PATH='/my/path/to/lib'):
... home = os.environ.get('HOME')
... path = os.environ.get("LD_LIBRARY_PATH")
>>> home is None
True
>>> path
'/my/path/to/lib'
>>> home = os.environ.get('HOME')
>>> path = os.environ.get("LD_LIBRARY_PATH")
>>> home is None
False
>>> path is None
True
EDIT2
A demonstration of this context manager is available on GitHub.
_environ = dict(os.environ) # or os.environ.copy()
try:
...
finally:
os.environ.clear()
os.environ.update(_environ)
I was looking to do the same thing but for unit testing, here is how I have done it using the unittest.mock.patch function:
def test_function_with_different_env_variable():
with mock.patch.dict('os.environ', {'hello': 'world'}, clear=True):
self.assertEqual(os.environ.get('hello'), 'world')
self.assertEqual(len(os.environ), 1)
Basically using unittest.mock.patch.dict with clear=True, we are making os.environ as a dictionary containing solely {'hello': 'world'}.
Removing the clear=True will let the original os.environ and add/replace the specified key/value pair inside {'hello': 'world'}.
Removing {'hello': 'world'} will just create an empty dictionary, os.envrion will thus be empty within the with.
In pytest you can temporarily set an environment variable using the monkeypatch fixture. See the docs for details. I've copied a snippet here for your convenience.
import os
import pytest
from typing import Any, NewType
# Alias for the ``type`` of monkeypatch fixture.
MonkeyPatchFixture = NewType("MonkeyPatchFixture", Any)
# This is the function we will test below to demonstrate the ``monkeypatch`` fixture.
def get_lowercase_env_var(env_var_name: str) -> str:
"""
Return the value of an environment variable. Variable value is made all lowercase.
:param env_var_name:
The name of the environment variable to return.
:return:
The value of the environment variable, with all letters in lowercase.
"""
env_variable_value = os.environ[env_var_name]
lowercase_env_variable = env_variable_value.lower()
return lowercase_env_variable
def test_get_lowercase_env_var(monkeypatch: MonkeyPatchFixture) -> None:
"""
Test that the function under test indeed returns the lowercase-ified
form of ENV_VAR_UNDER_TEST.
"""
name_of_env_var_under_test = "ENV_VAR_UNDER_TEST"
env_var_value_under_test = "EnvVarValue"
expected_result = "envvarvalue"
# KeyError because``ENV_VAR_UNDER_TEST`` was looked up in the os.environ dictionary before its value was set by ``monkeypatch``.
with pytest.raises(KeyError):
assert get_lowercase_env_var(name_of_env_var_under_test) == expected_result
# Temporarily set the environment variable's value.
monkeypatch.setenv(name_of_env_var_under_test, env_var_value_under_test)
assert get_lowercase_env_var(name_of_env_var_under_test) == expected_result
def test_get_lowercase_env_var_fails(monkeypatch: MonkeyPatchFixture) -> None:
"""
This demonstrates that ENV_VAR_UNDER_TEST is reset in every test function.
"""
env_var_name_under_test = "ENV_VAR_UNDER_TEST"
expected_result = "envvarvalue"
with pytest.raises(KeyError):
assert get_lowercase_env_var(env_var_name_under_test) == expected_result
For unit testing I prefer using a decorator function with optional parameters. This way I can use the modified environment values for a whole test function. The decorator below also restores the original environment values in case the function raises an Exception:
import os
def patch_environ(new_environ=None, clear_orig=False):
if not new_environ:
new_environ = dict()
def actual_decorator(func):
from functools import wraps
#wraps(func)
def wrapper(*args, **kwargs):
original_env = dict(os.environ)
if clear_orig:
os.environ.clear()
os.environ.update(new_environ)
try:
result = func(*args, **kwargs)
except:
raise
finally: # restore even if Exception was raised
os.environ = original_env
return result
return wrapper
return actual_decorator
Usage in unit tests:
class Something:
#staticmethod
def print_home():
home = os.environ.get('HOME', 'unknown')
print("HOME = {0}".format(home))
class SomethingTest(unittest.TestCase):
#patch_environ({'HOME': '/tmp/test'})
def test_environ_based_something(self):
Something.print_home() # prints: HOME = /tmp/test
unittest.main()