I have a mutli-module project which I'm migrating from Maven to Bazel. During this migration people will need to be able to work on both build systems.
After an mvn clean install Maven copies some of the BUILD files into the target folder.
When I later try to run bazel build //... it thinks the BUILD files under the various target folders are valid packages and fails due to some mismatch.
I've seen deleted_packages but AFAICT it requires I specify the list of folders to "delete" while I can't do that for 200+ modules.
I'm looking for the ability to say bazel build //... --deleted_packages=**/target.
Is this supported? (my experimentation says it's not but I might be wrong). If it's not supported is there an existing hack for it?
Can you use your shell to find the list of packages to ignore?
deleted=$(find . -name target -type d)
bazel build //... --deleted_packages="$deleted"
#Laurent's answer gave me the lead but Bazel didn't accept relative paths and required I add both classes and test-classes folders under target to delete the package so I decided to answer with the complete solution:
#!/bin/bash
#find all the target folders under the current working dir
target_folders=$(find . -name target -type d)
#find the repo root (currently assuming it's git based)
repo_root=$(git rev-parse --show-toplevel)
repo_root_length=${#repo_root}
#the current bazel package prefix is the PWD minus the repo root and adding a slash
current_bazel_package="/${PWD:repo_root_length}"
deleted_packages=""
for target in $target_folders
do
#cannonicalize the package path
full_package_path="$current_bazel_package${target:1}"
classes_full="${full_package_path}/classes"
test_classes_full="${full_package_path}/test-classes"
deleted_packages="$deleted_packages,$classes_full,$test_classes_full"
done
#remove the leading comma and call bazel-real with the other args
bazel-real "$#" --deleted_packages=${deleted_packages:1}
This script was checked in under tools/bazel which is why it calls bazel-real at the end.
I'm sorry I don't think this is supported. Some brainstorming:
Is it an option to point maven outputs somewhere else?
Is is an option not to use //... but explicit target(s)?
Maybe just remove the bad BUILD files before running bazel?
Related
My repo has some gitignored paths that are sometimes populated during non-Bazel development and contain symlinks to other code paths that can sometimes cause the Bazel loader to too aggressively look for packages, which can cause bazel query to wrongly match packages through symlinks. Workspace root .bazelignore allows me to hide these directories but is there an option to tell Bazel to ignore a directory using a more local config rather than a single global file?
Specifically, I'm using Yarn Workspaces in Node.js. The source tree looks like the following:
BUILD
package.json
yarn.lock
myapp
├─ BUILD
└─ package.json
Running yarn install outside of Bazel creates a directory called node_modules, which includes third-party dependencies as well as a symlink farm to local packages:
BUILD
package.json
yarn.lock
node_modules
├─ myapp -> ../myapp
└─ some_3p_dep
myapp
├─ BUILD
└─ package.json
Expressing this pattern through Bazel is straightforward, and I've written a custom rule that just uses ctx.actions.declare_directory to create node_modules in the bindir (I have to tar/untar the symlink to conceal it from Bazel to prevent it from complaining, but the rule works great).
However, this results in a problem if a user runs the yarn install command locally without Bazel then runs something like bazel query //... | xargs bazel test to run all tests. In general bazel query //... appears to wrongly list not just targets in //myapp but also targets in //node_modules/myapp... after all node_modules/myapp/BUILD technically exists. This is problematic at the very least because it duplicates targets, but also in practice it causes ArtifactPrefixConflictException for any outputs that //node_modules/myapp declares because the paths overlap with the //:node_modules target/output. If I wanted to, I could technically work around the latter by messing with output names, but the duplication would still exist, wasting compute time and leading to different Bazel results depending on if the user is also doing a non-Bazel devloop.
.bazelignore appears to work fine to get Bazel not to traverse node_modules, but it only supports full paths (https://github.com/bazelbuild/bazel/issues/8106) and requires every new node_modules that may come into existence, which is cumbersome because it's a global file. Is there an option that we can declare the ignore more globally or another workaround I'm missing?
I have a genrule in Bazel that is supposed to manipulate some files. I think I'm not accessing these files by the correct path, so I want to look at the directory structure that Bazel is creating so I can debug.
I added some echo statements to my genrule and I can see that Bazel is working in the directory /home/lyft/.cache/bazel/_bazel_lyft/8de0a1069de8d166c668173ca21c04ae/sandbox/linux-sandbox/1/execroot/. However, after Bazel finishes running, this directory is gone, so I can't look at the directory structure.
How can I prevent Bazel from deleting its temporary files so that I can debug what's happening?
Since this question is a top result for "keep sandbox files after build bazel" Google search and it wasn't obvious for me from the accepted answer, I feel the need to write this answer.
Short answer
Use --sandbox_debug. If this flag is passed, Bazel will not delete the files inside the sandbox folder after the build finishes.
Longer answer
Run bazel build with --sandbox_debug option:
$ bazel build mypackage:mytarget --sandbox_debug
Then you can inspect the contents of the sandbox folder for the project.
To get the location of the sandbox folder for current project, navigate to project and then run:
$ bazel info output_base
/home/johnsmith/.cache/bazel/_bazel_johnsmith/d949417420413f64a0b619cb69f1db69 # output will be something like this
Inside that directory there will be sandbox folder.
Possible caveat: (I'm NOT sure about this but) It's possible that some of the files are missing in sandbox folder, if you previously ran a build without --sandbox_debug flag and it partially succeeded. The reason is Bazel won't rerun parts of the build that already succeeded, and consequently the files corresponding to the successful build parts might not end up in the sandbox.
If you want to make sure all the sandbox files are there, clean the project first using either bazel clean or bazel clean --expunge.
You can use --spawn_strategy=standalone.
You can also use --sandbox_debug to see which directories are mounted to the sandbox.
You can also set the genrule's cmd to find . > $# to debug what's available to the genrule.
Important: declare all srcs/outs/tools that the genrule will read/write/use, and use $(location //label/of:target) to look up their path. Example:
genrule(
name = "x1",
srcs = ["//foo:input1.txt", "//bar:generated_file"],
outs = ["x1out.txt", "x1err.txt"],
tools = ["//util:bin1"],
cmd = "$(location //util:bin1) --input1=$(location //foo:input1.txt) --input2=$(location //bar:generated_file) --some_flag --other_flag >$(location x1out.txt) 2>$(location x1err.txt)",
)
For some reasons, practical or not, rxjs npm package stores BAZEL.build configuration in the package, so when I'm trying to build my project (which has node_modules folder) bazel tries automatically to build something that it's not supposed to build at all.
My question would be - what is canonical way of ignoring some specific folder while building bazel project recursively?
The only way to achieve what I'm looking for that I know of is to point to it explicitly in the command line
bazel build //... --deleted_packages=node_modules/rxjs/src (see user manual)
But I don't want to type this every time.
Bazel recently added a feature for ignoring folders (similar to gitignore).
Simply add node_modules to the .bazelignore file in the root of your project.
Yes, this is expressible as a bazel target pattern:
bazel build -- //... -//node_modules/rxjs/src/...
Full documentation is available at https://docs.bazel.build/versions/master/user-manual.html#target-patterns
I need to copy
C:\opencv-3.4.0.-opencl\bin\Debug\*.dll =>
myproj\build\bin\Debug\*.dll
and also
C:\opencv-3.4.0.-opencl\bin\Release\*.dll =>
myproj\build\bin\Release\*.dll
I'd like to do it in one command for Build/Release if possible.
You can copy files on a post-build command. A step through tutorial can be found here.
The basic concept is that you can use batch file commands, as a post-build step in Visual Studio to do basically anything you want as you build.
A further tutorial can be found here
For CMAKE
The easiest way is to follow the advice above but instead of putting it in the post-build options in VS just add a custom command
You can try using CPack to handle multiple configuration at one go. See an example in the following tutorial
https://cmake.org/cmake/help/latest/guide/tutorial/index.html#packaging-debug-and-release-step-12
By default, CMake’s model is that a build directory only contains a
single configuration, be it Debug, Release, MinSizeRel, or
RelWithDebInfo. It is possible, however, to setup CPack to bundle
multiple build directories and construct a package that contains
multiple configurations of the same project.
Then you will need to use either of the following method for each configuration to copy the files you need
configure_file
https://cmake.org/cmake/help/latest/command/configure_file.html
or
add_custom_command
https://cmake.org/cmake/help/latest/command/add_custom_command.html
Here is an example from reddit
https://www.reddit.com/r/cmake/comments/gmewhu/copy_one_file_in_src_directory_to_build_directory/
# Copy <filename> to build directory
set(copy_source_dir "${CMAKE_SOURCE_DIR}/src/<path>")
set(copy_dest_dir "${CMAKE_BINARY_DIR}/Build/<path>/$<CONFIG>")
set(copy_file_name "<filename>")
add_custom_command(
TARGET ${PROJECT_NAME} POST_BUILD
COMMAND ${CMAKE_COMMAND} -E make_directory ${copy_dest_dir}
)
add_custom_command(
TARGET ${PROJECT_NAME} POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy "${copy_source_dir}/${copy_file_name}" "${copy_dest_dir}/${copy_file_name}"
COMMENT "Copying ${copy_file_name} to build directory"
)
In my repository I have some files with the name "build" (automatically generated and/or imported, spread around elsewhere from where I have my bazel build files). These seem to be interpreted by Bazel as its BUILD files, and fail the full build I try to run with bazel build //...
Is there some way I can tell Bazel in a settings configuration file to ignore certain directories altogether? Or perhaps specify the build file names as something other than BUILD, like BUILD.bazel?
Or are my options:
To banish the name build from the entire repository.
To add a gigantic --deleted_packages=<...> to every run of build.
To not use full builds, instead specifying explicit targets.
I think this is a duplicate of the two questions you linked, but to expand on what you asked about in your comment:
You don't have to rename them BUILD.bazel, my suggestion is to add an empty BUILD.bazel to those directories. So you'd end up with:
my-project/
BUILD
src/
build/
stuff-bazel-shouldn't-mess-with
BUILD.bazel # Empty
Then Bazel will check for targets in BUILD.bazel, see that there are none, and won't try to parse the build/ directory.
And there is a distressing lack of documentation about BUILD vs. BUILD.bazel, at least that I could find.