I'm doing quite trivial java build in BitBucket Pipeline. the only twist is that it is in the repository subdirectory.
my bitbucket-pipelines.yml:
pipelines:
default:
- step:
caches:
- gradle
script: # Modify the commands below to build your repository.
# You must commit the Gradle wrapper to your repository
# https://docs.gradle.org/current/userguide/gradle_wrapper.html
- bash "./foo bar/gradlew" -p "./foo bar" distTar
- ls ./foo\ bar/build -R
- echo 'THE END'
artifacts:
- ./foo bar/build/distributions/xxx.tar
My ls confirms that xxx.tar is in the expected location
....
./foo bar/build/distributions:brigitte.tar
....
, but artifact page is empty.
Found it! It should be
# ...
artifacts:
- foo bar/build/distributions/brigitte.tar
artifacts paths are not real path so "dot slash" at the beginning was invalidating my path. Shame that it was not raised as a warning!
Extending the existing answer, I'd like to highlight the docs fragment that speaks about this
https://support.atlassian.com/bitbucket-cloud/docs/use-artifacts-in-steps/#Introduction
You can use glob patterns to define artifacts. Glob patterns that start with a * will need to be put in quotes. Note: As these are glob patterns, path segments “.” and “..” won’t work. Use paths relative to the build directory.
So that's it: artifacts can't be outside the build directory and artifact definitions in the pipelines must not contain . and .. path segments.
Absolute paths starting with / will neither work!
It is truly shameful how obscure this is like, why wouldn't the pipelines throw an error if that declaration is invalid? Damn, Bitbucket!
Related
How does one create test result files for bitbucket pipelines?
My bitbucket bitbucket-pipelines.yml contains:
options:
docker: true
pipelines:
default:
- step:
name: test bitbucket pipelines stuff..
script: # Modify the commands below to build your repository.
- /bin/bash -c 'mkdir test-results; echo Error Hello, World >> test-results/test1.txt; find'
and when running this pipeline i get
/bash -c 'mkdir test-results; echo Error Hello, World >> test-results/test1.txt; find'
<1s
+ /bin/bash -c 'mkdir test-results; echo Error Hello, World >> test-results/test1.txt; find'
.
(... censored/irrelevant stuff here)
./test-results
./test-results/test1.txt
then i get the "build teardown" saying it can't find test-results/test1.txt:
Build teardown
<1s
Searching for test report files in directories named
[test-results, failsafe-reports, test-reports, TestResults, surefire-reports] down to a depth of 4
Finished scanning for test reports. Found 0 test report files.
Merged test suites, total number tests is 0, with 0 failures and 0 errors.
i am surprised that it failed to find the ./test-results/test1.txt file.. hence the question.
Usually, each language/framework has some kind of utility to automatically produce such files as an outcome of a test suite run.
E.g. in Python you could simply run
pytest --junitxml=test-results/pytest.xml
See https://docs.pytest.org/en/latest/how-to/output.html#creating-junitxml-format-files
Manually crafting the xml youself feels brittle and tedious. Better find whatever library/option is available for your language/framework.
per https://support.atlassian.com/bitbucket-cloud/docs/test-reporting-in-pipelines/ seems it has to be XML files.. also it needs to be in a "j-unit xml format" ? an example of which can be found here https://www.ibm.com/docs/en/developer-for-zos/9.1.1?topic=formats-junit-xml-format
.. so try changing bitbucket-pipelines.yml to
options:
docker: true
pipelines:
default:
- step:
name: test bitbucket pipelines stuff..
script: # Modify the commands below to build your repository.
- export IMAGE_NAME2=easyad/easyad_nginx:$BITBUCKET_COMMIT
- /bin/bash bitbucket_pipeline_tests.sh
and in bitbucket_pipeline_tests.sh add
#!/bin/bash
mkdir test-results;
echo '<?xml version="1.0" encoding="UTF-8" ?>
<testsuites id="20140612_170519" name="New_configuration (14/06/12 17:05:19)" tests="225" failures="1262" time="0.001">
<testsuite id="codereview.cobol.analysisProvider" name="COBOL Code Review" tests="45" failures="17" time="0.001">
<testcase id="codereview.cobol.rules.ProgramIdRule" name="Use a program name that matches the source file name" time="0.001">
<failure message="PROGRAM.cbl:2 Use a program name that matches the source file name" type="WARNING">
WARNING: Use a program name that matches the source file name
Category: COBOL Code Review – Naming Conventions
File: /project/PROGRAM.cbl
Line: 2
</failure>
</testcase>
</testsuite>
</testsuites>' >>./test-results/test1.xml
then the pipeline run should say 17 / 45 tests failed, as indicated by the sample XML above...
With the current syntax of bitbucket's EXTRA_ARGS variable one directory is excluded from deployment like this:
EXTRA_ARGS: '--exclude=YOUR_DESIRE_FOLDER_PATH/*'
(Bitbucket Pipeline - how to exclude files or folders?)
But how to exclude multiple directories?
First, note that not every pipe has a support for --exclude option, as some pipes are just wrappers around the cli tools, like rsync or sftp. However, if you use the rsync-deploy pipe, you should be able to use multiple --exclude options:
script:
- pipe: atlassian/rsync-deploy:0.3.2
variables:
USER: 'ec2-user'
SERVER: '127.0.0.1'
REMOTE_PATH: '/var/www/build/'
LOCAL_PATH: 'build'
DEBUG: 'true'
EXTRA_ARGS: '--exclude=*.txt --exclude=src/*'
I have a set of Jenkins jobs that are substantially the same. I have created a job template that creates them all. However, some have builders that others don't (i.e. the first in the chain doesn't copy artifacts from another project) and others have publishers that others don't (they don't all have JUnit tests).
I would like to conditionally include these modules depending on a variable, but I can't find a way of doing this:
I can't use a jinja2 template to include or exclude one item in a list
Including empty variables typically causes the build to fail
I could include yaml files, but I would need to include all of the builders section, and I would need one for each job, meaning a lot of repetition
Is this possible? I would like to include the comment section below in some of the jobs.
43 builders:
44 - shell: |
45 echo Removing working directory from previous run
46 rm -rf ${{WORKSPACE}}/css-build/working
47 # - copyartifact:
48 # project: "{previous-project}"
49 # whichbuild: last-successful
50 # optional: "{copy-optional}"
51 - shell: |
52 {init-shell}
53 ${{WORKSPACE}}/css-build/build-util.sh {shell-args} ${{WORKSPACE}}/{location} -w ${{WORKSPACE}}/css-b uild/working
Well, here is the workaround:
Define a new module (in this case it will be a builder) with a different name to the original. If the omit tag is present, don't do anything; otherwise, do what would have happened anyway.
def optional_copy(registry, xml_parent, data):
if data['omit'].lower() == 'true':
return
else:
new_data = collections.OrderedDict()
new_data['copyartifact'] = data
registry.dispatch('builder', xml_parent, new_data)
Register it to jjb in setup.py:
setup(
name='JJB config',
py_modules = ['optionals'],
entry_points={
'jenkins_jobs.builders': [
'optional-copy=optionals:optional_copy'
]
}
)
Then, in your yaml, you can use the optional-copy module and the omit property:
builders:
- shell: |
echo Removing working directory from previous run
rm -rf "{working-dir}"
- optional-copy:
omit: "{omit-copy}"
project: "{prev}"
whichbuild: last-successful
- shell: |
{init-shell}
${{WORKSPACE}}/css-build/build-util.sh -u {diirt-version} {shell-args} -p ${{WORKSPACE}}/{location} -w "{working-dir}"
I have got a workaround for your issue that does not require extending the job builder.
But it requires the availablity of the Conditional build step plugin on jenkins.
Example for optional builders:
- job-template
id: my-custom-template
builders:
- conditional-step:
condition-kind: always
steps: "{obj:optional_builders|[]}"
With this you can add builders to your job using a optional_builders variable (if you want to).
jobs:
#With optional_builders
- my-custom-template:
optional_builders:
- copyartifact:
project: "{previous-project}"
whichbuild: last-successful
#Without optional_builders
- my-custom-template:
Example for optional publishers:
publishers:
- conditional-publisher:
- condition-kind: always
action: "{obj:optional_publishers|[]}"
I am executing parameterised build in jenkins to count no. of lines in file which has 1 file parameter. Its file location is pqr. The name of the script file is linecount.sh which is saved at remote server. When i tried to execute it using command sh linecount.sh filename, it works perfectly from jenkins. But as i remove filename from the argument and execute same script as parameterised build it is showing below error on console :
Started by user Prasoon Gupta
[EnvInject] - Loading node environment variables.
Building in workspace users/Prasoon/sample_programs
Copying file to pqr
[sample_programs] $ /bin/sh -xe /tmp/hudson3529902665956638862.sh
+ sh linecount.sh
PRASOON4
linecount.sh: line 15: parameterBuild.txt: No such file or directory
Build step 'Execute shell' marked build as failure
Finished: FAILURE
I am uploading file (parameterBuild.txt) from my local machine. Why is it giving this error?
My doubt is in shell script I used argument as $1. How can I refer this when I am taking file as parameter.
The uploaded file will not retain the same name as it has on your local computer. It will be named after the File location argument specified in the file parameter settings:
In this example I will get a file called file.txt in my workspace root, regardless of what I call it on my computer.
So if I now build my job and enter the following in the parameter dialog (note that my local filename is table.html):
Then I get the following in the log (I have a build step which does ls -l):
Building on master in workspace /var/lib/jenkins/workspace/fs
Copying file to file.txt
[fs] $ /bin/sh -xe /tmp/hudson845437350739055843.sh
+ ls -l
total 4
-rw-r--r-- 1 jenkins jenkins 292 Feb 15 07:23 file.txt
Finished: SUCCESS
Note that table.html now is called file.txt, e.g. what I entered as File location.
So in you're case the command should be:
sh linecount.sh pqr
There is a a bug since ages that makes impossible to use fileParameter:
Handle file parameters
file parameter not working in pipeline job
There is a workaround for this issue https://github.com/janvrany/jenkinsci-unstashParam-library
and in a pipeline script you do:
library "jenkinsci-unstashParam-library"
node {
def file_in_workspace = unstashParam "file"
sh "cat ${file_in_workspace}"
}
If it's to do with Free-Style job & if your configuration looks similar to this - https://i.stack.imgur.com/vH7mQ.png then you can run simply do sh linecount.sh ${pqr} to get what you are looking for?
I am trying to generate Dockerfiles with Ansible template - see the role source and the template in Ansible Galaxy and Github
I need to genarate a standard Dockerfile line like:
...
VOLUME ["/etc/postgresql/9.4"]
...
However, when I put this in the input file:
...
instruction: CMD
value: "[\"/etc/postgresql/{{postgresql_version}}\"]"
...
It ends up rendered like:
...
VOLUME ['/etc/postgresql/9.4']
...
and I lose the " (which renders the Dockerfiles useless)
Any help ? How can I convince Jinja to not substitute " with ' ? I tried \" , |safe filter, even {% raw %} - it just keeps doing it!
Update:
Here is how to reproduce the issue:
Go get the peruncs.docker role from galaxy.ansible.com or Github (link is given above)
Write up a simple playbook (say demo.yml) with the below content and run: ansible-playbook -v demo.yml. The -v option will allow you to see the temp directory where the generated Dockerfile goes with the broken content, so you can examine it. Generating the docker image is not important to succeed, just try to get the Dockerfile right.
- name: Build docker image
hosts: localhost
vars:
- somevar: whatever
- image_tag: "blabla/booboo"
- docker_copy_files: []
- docker_file_content:
- instruction: CMD
value: '["/usr/bin/runit", "{{somevar}}"]'
roles:
- peruncs.docker
Thanks in advance!
Something in Ansible appears to be recognizing that as valid Python, so it's getting transformed into a Python list and then serialized using Python's str(), which is why you end up with the single-quoted values.
An easy way to work around this is to stick a space at the beginning of the value, which seems to prevent it from getting converted into Python:
- name: Build docker image
hosts: localhost
vars:
- somevar: whatever
- image_tag: "blabla/booboo"
- docker_copy_files: []
- docker_file_content:
- instruction: CMD
value: ' ["/usr/bin/runit", "{{somevar}}"]'
roles:
- peruncs.docker
This results in:
CMD ["/usr/bin/runit", "whatever"]