I am trying to create a javadoc based on the javadoc.bzl
I have added in my BUILD file under the root of my project to load the rule:
load("//rules:javadoc.bzl", "javadoc")
javadoc(
name = "api-docs",
srcs = glob(["**/*.java"])
)
But when I run it it fails and from what I see using the --sandbox_debug it does not seem to be calling javadoc with any source files
Specifically running: bazel build --sandbox_debug :api-docs
I get:
BUILD:3:8: error executing shell command: '/bin/bash -c mkdir api-docs
external/local_jdk/bin/javadoc -quiet -d api-docs zip -q -r
bazel-out/darwin-fastbuild/bin/api-docs.zip api-docs/*' failed (Exit
12): sandbox-exec failed: error executing command
and later on javadoc: error - No packages or classes specified.
If I understand correctly the error reported, I see external/local_jdk/bin/javadoc -quiet -d api-docs called but no files as input.
What am I doing wrong here?
glob doesn't pass over package boundaries and so the glob won't match any directories that have a BUILD file in them. I suspect you have a BUILD file in the directory that you are trying to match against.
In your src directory add the following to your BUILD file
filegroup(
name = "javasrcs",
srcs = glob(["**/*.java"]),
visibility = ["//visibility:public"],
)
Then in your top level BUILD file reference it like this:
load("//rules:javadoc.bzl", "javadoc")
javadoc(
name = "api-docs",
srcs = ["//src:javasrcs"],
)
assuming the sources are in a directory src
Related
I have a file file.txt with filenames ending with *.sha256, including the full paths of each file. This is a toy example:
file.txt:
/path/a/9b/x3.sha256
/path/7c/7j/y2.vcf.gz.sha256
/path/e/g/7z.sha256
Each line has a different path/file. The *.sha256 files have checksums.
I want to run the command "sha256sum -c" on each of these *.sha256 files and write the output to an output_file.txt. However, this command only accepts the name of the .sha256 file, not the name including its full path. I have tried the following:
while read in; do
sha256sum -c "$in" >> output_file.txt
done < file.txt
but I get:
"sha256sum: WARNING: 1 listed file could not be read"
which is due to the path included in the command.
Any suggestion is welcome
#!/bin/bash
while read in
do
thedir=$(dirname "$in")
thefile=$(basename "$in")
cd "$thedir"
sha256sum -c "$thefile" >>output_file.txt
done < file.txt
Modify your code to extract the directory and file parts of your in variable.
I've added the following nix expression to my project (verbatem from nixpkgs) and tried to build it, but I get the following error message:
these derivations will be built:
/nix/store/l6l5j1gas74pnzc2pb8w9fv2namxpgki-godot-3.0.6.drv
building '/nix/store/l6l5j1gas74pnzc2pb8w9fv2namxpgki-godot-3.0.6.drv'...
unpacking sources
unpacking source archive /nix/store/mlbp5ibpyq2rd710fl43pwr6a03ysz57-source
source root is source
patching sources
applying patch /nix/store/yk47p909lklbcai94izb5dfqjqqnnxmc-pkg_config_additions.patch
patching file platform/x11/detect.py
applying patch /nix/store/sdr3xp65cibpd06vq4fg5czv2s3m6a3c-dont_clobber_environment.patch
patching file SConstruct
configuring
no configure script, doing nothing
building
no Makefile, doing nothing
installing
cp: missing destination file operand after '/nix/store/mxvinscpfbv3k5j7dvpa83pd4w1p1f4f-godot-3.0.6/bin/godot'
Try 'cp --help' for more information.
builder for '/nix/store/l6l5j1gas74pnzc2pb8w9fv2namxpgki-godot-3.0.6.drv' failed with exit code 1
What could be causing this? Note that when I simply add godot to my nix config (in NixOS using nixos-18.09 channel), it installs and runs just fine. Aren't these the same expressions? If so, why does it work globally and not locally?
NOTE: Here is the part of the nix expression that invokes the cp shell command (related to the error above):
installPhase = ''
mkdir -p "$out/bin"
cp bin/godot.* $out/bin/godot
mkdir "$dev"
cp -r modules/gdnative/include $dev
mkdir -p "$man/share/man/man6"
cp misc/dist/linux/godot.6 "$man/share/man/man6/"
mkdir -p "$out"/share/{applications,icons/hicolor/scalable/apps}
cp misc/dist/linux/godot.desktop "$out/share/applications/"
cp icon.svg "$out/share/icons/hicolor/scalable/apps/godot.svg"
cp icon.png "$out/share/icons/godot.png"
substituteInPlace "$out/share/applications/godot.desktop" \
--replace "Exec=godot" "Exec=$out/bin/godot"
'';
I have set up bazel to build a number of CLI tools that perform various database maintenance tasks. Each one is a py_binary or cc_binary target that is called from the command line with the path to some data file: it processes that file and stores the results in a database.
Now, I need to create a dependent package that contains data files and shell scripts that call these CLI tools to perform application-specific database operations.
However, there doesn't seem to be a way to depend on the existing py_binary or cc_binary targets from a new package that only contains sh_binary targets and data files. Trying to do so results in an error like:
ERROR: /workspace/shbin/BUILD.bazel:5:12: in deps attribute of sh_binary rule //shbin:run: py_binary rule '//pybin:counter' is misplaced here (expected sh_library)
Is there a way to call/depend on an existing bazel binary target from a shell script using sh_binary?
I have implemented a full example here:
https://github.com/psigen/bazel-mixed-binaries
Notes:
I cannot use py_library and cc_library instead of py_binary and cc_binary. This is because (a) I need to call mixes of the two languages to process my data files and (b) these tools are from an upstream repository where they are already designed as CLI tools.
I also cannot put all the data files into the CLI tool packages -- there are multiple application-specific packages and they cannot be mixed.
You can either create a genrule to run these tools as part of the build, or create a sh_binary that depends on the tools via the data attribute and runs them them.
The genrule approach
This is the easier way and lets you run the tools as part of the build.
genrule(
name = "foo",
tools = [
"//tool_a:py",
"//tool_b:cc",
],
srcs = [
"//source:file1",
":file2",
],
outs = [
"output_file1",
"output_file2",
],
cmd = "$(location //tool_a:py) --input=$(location //source:file1) --output=$(location output_file1) && $(location //tool_b:cc) < $(location :file2) > $(location output_file2)",
)
The sh_binary approach
This is more complicated, but lets you run the sh_binary either as part of the build (if it is in a genrule.tools, similar to the previous approach) or after the build (from under bazel-bin).
In the sh_binary you have to data-depend on the tools:
sh_binary(
name = "foo",
srcs = ["my_shbin.sh"],
data = [
"//tool_a:py",
"//tool_b:cc",
],
)
Then, in the sh_binary you have to use the so-called "Bash runfiles library" built into Bazel to look up the runtime-path of the binaries. This library's documentation is in its source file.
The idea is:
the sh_binary has to depend on a specific target
you have to copy-paste some boilerplate code to the top of the sh_binary (reason is described here)
then you can use the rlocation function to look up the runtime-path of the binaries
For example your my_shbin.sh may look like this:
#!/bin/bash
# --- begin runfiles.bash initialization ---
...
# --- end runfiles.bash initialization ---
path=$(rlocation "__main__/tool_a/py")
if [[ ! -f "${path:-}" ]]; then
echo >&2 "ERROR: could not look up the Python tool path"
exit 1
fi
$path --input=$1 --output=$2
The __main__ in the rlocation path argument is the name of the workspace. Since your WORKSPACE file does not have a "workspace" rule in, which would define the workspace's name, Bazel will use the default workspace name, which is __main__.
An easier approach for me is to add the cc_binary as a dependency in the data section. In prefix/BUILD
cc_binary(name = "foo", ...)
sh_test(name = "foo_test", srcs = ["foo_test.sh"], data = [":foo"])
Inside foo_test.sh, the working directory is different, so you need to find the right prefix for the binary
#! /usr/bin/env bash
executable=prefix/foo
$executable ...
A clean way to do this is to use args and $(location):
Contents of BUILD:
py_binary(
name = "counter",
srcs = ["counter.py"],
main = "counter.py",
)
sh_binary(
name = "run",
srcs = ["run.sh"],
data = [":counter"],
args = ["$(location :counter)"],
)
Contents of counter.py (your tool):
print("This is the counter tool.")
Contents of run.sh (your bash script):
#!/bin/bash
set -eEuo pipefail
counter="$1"
shift
echo "This is the bash script, about to call the counter tool."
"$counter"
And here's a demo showing the bash script calling the Python tool:
$ bazel run //example:run 2>/dev/null
This is the bash script, about to call the counter tool.
This is the counter tool.
It's also worth mentioning this note (from the docs):
The arguments are not passed when you run the target outside of bazel (for example, by manually executing the binary in bazel-bin/).
I am using genhtml command to generate html coverage report from Bazel generated coverage.dat file:
genhtml bazel-testlogs/path/to/TestTarget/coverage.dat --output-directory coverage
The problem with using genhtml is that I have to provide the paths to the coverage.dat files (which are generated in bazel-testlogs/..) Is it possible to fetch those coverage.dat files as an output from another rule?
I would like to not have to call genthml command directly, but have Bazel handle everything.
I was not able to find a way to get coverage.dat files as an output of a bazel rule. However, I was able to wrap all the locations of all the .dat files as srcs to a filegroup in WORKSPACE directory:
filegroup(
name = "coverage_files",
srcs = glob(["bazel-out/**/coverage.dat"]),
)
and then use that filegroup in a custom .bzl rule that wraps the genthml command to generate html coverage report. So now I only have to call
bazel coverage //path/... --instrumentation_filter=/path[/:]
command to generate the coverage.dat files, generate html report and zip it up. Thus, bazel handles everything.
Bazel added support for C++ coverage (though I couldn't find much documentation for it).
I was able to generate a combined coverage.dat file with
bazel coverage -s \
--instrument_test_targets \
--experimental_cc_coverage \
--combined_report=lcov \
--coverage_report_generator=#bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main \
//...
The coverage file gets added to bazel-out/_coverage/_coverage_report.dat
For Java based project we can get code coverage in following way
To get coverage for complete module ->
Running coverage for complete project module. Run following command ->
bazel coverage ... --compilation_mode=dbg --subcommands --announce_rc --verbose_failures --jobs=auto --sandbox_debug --build_runfile_links --combined_report=lcov --coverage_report_generator=#bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main
Then run following command from parent project directory to get html view. Html report is generated in output-directory-name we specified. From that open index.html to see coverage report.
genhtml -o <output-directory-name> bazel-out/_coverage/_coverage_report.dat
bazel-out directory usually gets created in project parent directory(e.g. where bazel WORKSPACE file is present)
To get coverage for specific IT/Test in a module ->
Running coverage for for specific IT/Test in a module. Run following command from project/sub-project directory ->
bazel coverage <class-name-of-Test-or-IT> --compilation_mode=dbg --subcommands --announce_rc --verbose_failures --jobs=auto --sandbox_debug --build_runfile_links --combined_report=lcov --coverage_report_generator=#bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main
Then run following command from parent project directory to get html view. Html report is generated in output-directory-name we specified. From that open index.html to see coverage report.
genhtml -o <output-directory-name> bazel-out/_coverage/_coverage_report.dat
I'm trying to create a custom build rule to build a pip package from a py_binary output. To create my pip package I want to invoke a shell script. The shell script builds a pip package by creating a zip file from the runfiles of the py_binary output.
For example suppose I have
py_binary(
name = "some_binary",
srcs = ["some_binary.py"],
srcs_version = "PY2AND3",
)
Building this rule produces
bazel-bin/some_binary.runfiles
I would now like to create a custom build rule that will invoke my shell script with the location bazel-bin/some_binary.runfiles
I tried creating a macro
def build_pip_package(
name, py_binary=None, setup_file=None):
"""Create a pip package from a py_binary.
The source file should be a text file with python formatting i.e.
Args:
name: Name for the rule.
py_binary: Build rule producing the py_binary
setup_file: Build rule producing the setup.py file to use to produce
the package.
"""
output= "somefile"
native.genrule(
name=name,
outs=[output],
cmd="echo $(location //:build_pip_package_script) "
+ "--py_runfiles_path=$(locations %s)" % py_binary
+ " --setup_file=$(location %s) " % setup_file,
tools=["//:build_pip_package_script"],
srcs=[setup_file, py_binary])
This ends up invoking my shell script with
bazel-out/local-fastbuild/bin/some_binary/model_train some_binary.py
How can I invoke my script with the location of the runfiles directory corresponding to the some_binary target.
If you put the py_binary in the tools attribute instead of srcs bazel will include the runfiles tree. You can access it via "$(location %s).runfiles" % py_binary