With the current syntax of bitbucket's EXTRA_ARGS variable one directory is excluded from deployment like this:
EXTRA_ARGS: '--exclude=YOUR_DESIRE_FOLDER_PATH/*'
(Bitbucket Pipeline - how to exclude files or folders?)
But how to exclude multiple directories?
First, note that not every pipe has a support for --exclude option, as some pipes are just wrappers around the cli tools, like rsync or sftp. However, if you use the rsync-deploy pipe, you should be able to use multiple --exclude options:
script:
- pipe: atlassian/rsync-deploy:0.3.2
variables:
USER: 'ec2-user'
SERVER: '127.0.0.1'
REMOTE_PATH: '/var/www/build/'
LOCAL_PATH: 'build'
DEBUG: 'true'
EXTRA_ARGS: '--exclude=*.txt --exclude=src/*'
Related
How does one create test result files for bitbucket pipelines?
My bitbucket bitbucket-pipelines.yml contains:
options:
docker: true
pipelines:
default:
- step:
name: test bitbucket pipelines stuff..
script: # Modify the commands below to build your repository.
- /bin/bash -c 'mkdir test-results; echo Error Hello, World >> test-results/test1.txt; find'
and when running this pipeline i get
/bash -c 'mkdir test-results; echo Error Hello, World >> test-results/test1.txt; find'
<1s
+ /bin/bash -c 'mkdir test-results; echo Error Hello, World >> test-results/test1.txt; find'
.
(... censored/irrelevant stuff here)
./test-results
./test-results/test1.txt
then i get the "build teardown" saying it can't find test-results/test1.txt:
Build teardown
<1s
Searching for test report files in directories named
[test-results, failsafe-reports, test-reports, TestResults, surefire-reports] down to a depth of 4
Finished scanning for test reports. Found 0 test report files.
Merged test suites, total number tests is 0, with 0 failures and 0 errors.
i am surprised that it failed to find the ./test-results/test1.txt file.. hence the question.
Usually, each language/framework has some kind of utility to automatically produce such files as an outcome of a test suite run.
E.g. in Python you could simply run
pytest --junitxml=test-results/pytest.xml
See https://docs.pytest.org/en/latest/how-to/output.html#creating-junitxml-format-files
Manually crafting the xml youself feels brittle and tedious. Better find whatever library/option is available for your language/framework.
per https://support.atlassian.com/bitbucket-cloud/docs/test-reporting-in-pipelines/ seems it has to be XML files.. also it needs to be in a "j-unit xml format" ? an example of which can be found here https://www.ibm.com/docs/en/developer-for-zos/9.1.1?topic=formats-junit-xml-format
.. so try changing bitbucket-pipelines.yml to
options:
docker: true
pipelines:
default:
- step:
name: test bitbucket pipelines stuff..
script: # Modify the commands below to build your repository.
- export IMAGE_NAME2=easyad/easyad_nginx:$BITBUCKET_COMMIT
- /bin/bash bitbucket_pipeline_tests.sh
and in bitbucket_pipeline_tests.sh add
#!/bin/bash
mkdir test-results;
echo '<?xml version="1.0" encoding="UTF-8" ?>
<testsuites id="20140612_170519" name="New_configuration (14/06/12 17:05:19)" tests="225" failures="1262" time="0.001">
<testsuite id="codereview.cobol.analysisProvider" name="COBOL Code Review" tests="45" failures="17" time="0.001">
<testcase id="codereview.cobol.rules.ProgramIdRule" name="Use a program name that matches the source file name" time="0.001">
<failure message="PROGRAM.cbl:2 Use a program name that matches the source file name" type="WARNING">
WARNING: Use a program name that matches the source file name
Category: COBOL Code Review – Naming Conventions
File: /project/PROGRAM.cbl
Line: 2
</failure>
</testcase>
</testsuite>
</testsuites>' >>./test-results/test1.xml
then the pipeline run should say 17 / 45 tests failed, as indicated by the sample XML above...
I'm doing quite trivial java build in BitBucket Pipeline. the only twist is that it is in the repository subdirectory.
my bitbucket-pipelines.yml:
pipelines:
default:
- step:
caches:
- gradle
script: # Modify the commands below to build your repository.
# You must commit the Gradle wrapper to your repository
# https://docs.gradle.org/current/userguide/gradle_wrapper.html
- bash "./foo bar/gradlew" -p "./foo bar" distTar
- ls ./foo\ bar/build -R
- echo 'THE END'
artifacts:
- ./foo bar/build/distributions/xxx.tar
My ls confirms that xxx.tar is in the expected location
....
./foo bar/build/distributions:brigitte.tar
....
, but artifact page is empty.
Found it! It should be
# ...
artifacts:
- foo bar/build/distributions/brigitte.tar
artifacts paths are not real path so "dot slash" at the beginning was invalidating my path. Shame that it was not raised as a warning!
Extending the existing answer, I'd like to highlight the docs fragment that speaks about this
https://support.atlassian.com/bitbucket-cloud/docs/use-artifacts-in-steps/#Introduction
You can use glob patterns to define artifacts. Glob patterns that start with a * will need to be put in quotes. Note: As these are glob patterns, path segments “.” and “..” won’t work. Use paths relative to the build directory.
So that's it: artifacts can't be outside the build directory and artifact definitions in the pipelines must not contain . and .. path segments.
Absolute paths starting with / will neither work!
It is truly shameful how obscure this is like, why wouldn't the pipelines throw an error if that declaration is invalid? Damn, Bitbucket!
When outputting characters from a declarative pipeline running inside a linux container is it possible to change the encoding to match the true output from the terminal? I.e.
├── file1 +-- file1
├── file2 +-- file2
└── file3 +-- file3
^Formatting I want ^Formatting I get
.
I tried passing the following arguments to my Docker Agent:
-e JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF-8"
-e LC_ALL="en_US.UTF-8"
.
Combined with:
sh returnStdout: true, script: " "
And got ├── in place of the "+--", which seems to be the ANSI encoding for the "├──".
I am using the ansiColor Option but that didn't seem to help much.
.
I saw this similar question, but I was unsure on how to implement the solution in the pipeline.
Jenkins: console output characters
You can use Jenkins II to change the encoding to UTF-8.
Go to
Jenkins -> Manage Jenkins -> Configure System -> Global properties
and add two envirenment variables JAVA_TOOL_OPTIONS and LANG having values -Dfile.encoding=UTF-8 and en_US.UTF-8 respectively
.
After adding these you may need to restart Jenkins.
Reference: https://www.linkedin.com/pulse/how-resolve-utf-8-encoding-issue-jenkins-ajuram-salim/
UPDATE:
or you can update <arguments> in jenkins.xml file.
e.g.
<arguments>-Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Dfile.encoding=UTF-8 -jar "%BASE%\jenkins.war" --httpPort=8080 --webroot="%BASE%\war"</arguments>
Here is the official answer from cloudbees. Unfortunately all of these did not work for me.
https://support.cloudbees.com/hc/en-us/articles/360004397911-How-to-address-issues-with-unmappable-characters-
Add these to JVM Arguments in master and also on agents -
-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8
For me, the problem was really in specifying the optional 'encoding' parameter to the 'sh' pipeline step: sh: Shell Script
Of course, this will only work provided the file.encoding is set properly as described in other posts here.
I have a set of Jenkins jobs that are substantially the same. I have created a job template that creates them all. However, some have builders that others don't (i.e. the first in the chain doesn't copy artifacts from another project) and others have publishers that others don't (they don't all have JUnit tests).
I would like to conditionally include these modules depending on a variable, but I can't find a way of doing this:
I can't use a jinja2 template to include or exclude one item in a list
Including empty variables typically causes the build to fail
I could include yaml files, but I would need to include all of the builders section, and I would need one for each job, meaning a lot of repetition
Is this possible? I would like to include the comment section below in some of the jobs.
43 builders:
44 - shell: |
45 echo Removing working directory from previous run
46 rm -rf ${{WORKSPACE}}/css-build/working
47 # - copyartifact:
48 # project: "{previous-project}"
49 # whichbuild: last-successful
50 # optional: "{copy-optional}"
51 - shell: |
52 {init-shell}
53 ${{WORKSPACE}}/css-build/build-util.sh {shell-args} ${{WORKSPACE}}/{location} -w ${{WORKSPACE}}/css-b uild/working
Well, here is the workaround:
Define a new module (in this case it will be a builder) with a different name to the original. If the omit tag is present, don't do anything; otherwise, do what would have happened anyway.
def optional_copy(registry, xml_parent, data):
if data['omit'].lower() == 'true':
return
else:
new_data = collections.OrderedDict()
new_data['copyartifact'] = data
registry.dispatch('builder', xml_parent, new_data)
Register it to jjb in setup.py:
setup(
name='JJB config',
py_modules = ['optionals'],
entry_points={
'jenkins_jobs.builders': [
'optional-copy=optionals:optional_copy'
]
}
)
Then, in your yaml, you can use the optional-copy module and the omit property:
builders:
- shell: |
echo Removing working directory from previous run
rm -rf "{working-dir}"
- optional-copy:
omit: "{omit-copy}"
project: "{prev}"
whichbuild: last-successful
- shell: |
{init-shell}
${{WORKSPACE}}/css-build/build-util.sh -u {diirt-version} {shell-args} -p ${{WORKSPACE}}/{location} -w "{working-dir}"
I have got a workaround for your issue that does not require extending the job builder.
But it requires the availablity of the Conditional build step plugin on jenkins.
Example for optional builders:
- job-template
id: my-custom-template
builders:
- conditional-step:
condition-kind: always
steps: "{obj:optional_builders|[]}"
With this you can add builders to your job using a optional_builders variable (if you want to).
jobs:
#With optional_builders
- my-custom-template:
optional_builders:
- copyartifact:
project: "{previous-project}"
whichbuild: last-successful
#Without optional_builders
- my-custom-template:
Example for optional publishers:
publishers:
- conditional-publisher:
- condition-kind: always
action: "{obj:optional_publishers|[]}"
I am trying to generate Dockerfiles with Ansible template - see the role source and the template in Ansible Galaxy and Github
I need to genarate a standard Dockerfile line like:
...
VOLUME ["/etc/postgresql/9.4"]
...
However, when I put this in the input file:
...
instruction: CMD
value: "[\"/etc/postgresql/{{postgresql_version}}\"]"
...
It ends up rendered like:
...
VOLUME ['/etc/postgresql/9.4']
...
and I lose the " (which renders the Dockerfiles useless)
Any help ? How can I convince Jinja to not substitute " with ' ? I tried \" , |safe filter, even {% raw %} - it just keeps doing it!
Update:
Here is how to reproduce the issue:
Go get the peruncs.docker role from galaxy.ansible.com or Github (link is given above)
Write up a simple playbook (say demo.yml) with the below content and run: ansible-playbook -v demo.yml. The -v option will allow you to see the temp directory where the generated Dockerfile goes with the broken content, so you can examine it. Generating the docker image is not important to succeed, just try to get the Dockerfile right.
- name: Build docker image
hosts: localhost
vars:
- somevar: whatever
- image_tag: "blabla/booboo"
- docker_copy_files: []
- docker_file_content:
- instruction: CMD
value: '["/usr/bin/runit", "{{somevar}}"]'
roles:
- peruncs.docker
Thanks in advance!
Something in Ansible appears to be recognizing that as valid Python, so it's getting transformed into a Python list and then serialized using Python's str(), which is why you end up with the single-quoted values.
An easy way to work around this is to stick a space at the beginning of the value, which seems to prevent it from getting converted into Python:
- name: Build docker image
hosts: localhost
vars:
- somevar: whatever
- image_tag: "blabla/booboo"
- docker_copy_files: []
- docker_file_content:
- instruction: CMD
value: ' ["/usr/bin/runit", "{{somevar}}"]'
roles:
- peruncs.docker
This results in:
CMD ["/usr/bin/runit", "whatever"]