Imagine I have a java_binary target triggered by a custom rule that generates source code and places the generated sources under a directory, let's call it "root".
So after the code generation we will have something like this:
// bazel-bin/...../src/com/example/root
root:
-> Foo.java
-> Bar.java
-> utils
-> Baz.java
Now, I have another target, a java_library, that depends on the previously generated sources, so it depends on the custom rule.
My custom rule definition currently looks something like this:
def _code_generator(ctx):
outputDir = ctx.actions.declare_directory("root")
files = [
ctx.actions.declare_file("root/Foo.java"),
ctx.actions.declare_file("root/Bar.java"),
ctx.actions.declare_file("root/utils/Baz.java"),
// and many,
// many other files
]
outputs = []
outputs.append(outputDir)
outputs.extend(files)
ctx.actions.run(
executable = // executable pointing to the java_binary
outputs = outputs
// ....
)
This works. But as you can see, every anticipated file that is to be generated, is hard-coded in the rule definition. This makes it very fragile, should the code generation produce a different set of files in the future (which it will).
(Without specifying each of the files, as shown above, Bazel will fail the build saying that the files have no generating action)
So I was wondering, is there a way to read the content of the root directory and automatically, somehow, declare each of the files as an output?
What I tried:
The documentation of declare_directory says:
The contents of the directory are not directly accessible from Starlark, but can be expanded in an action command with Args.add_all().
And add_all says:
[...] Each directory File item is replaced by all Files recursively contained in that directory.
This sounds like there could be a way to get access to the individual files in the directory, but I am not sure how.
I tried:
outputDir = ctx.actions.declare_directory("root")
//...
args = ctx.actions.args()
args.add_all(outputDir)
with the intention to access the individual files later from args, but the build fails with: "Error in add_all: expected value of type sequence or depset for values, got File".
Any other ideas on how to implement the rule, so that I don't have to hard-code each and every file that will be generated?
Related
Bazel has been working great for me recently, but I've stumbled upon a question for which I have yet to find a satisfactory answer:
How can one collect all files bearing a certain extension from the workspace?
Another way of phrasing the question: how could one obtain the functional equivalent of doing a glob() across a complete Bazel workspace?
Background
The goal in this particular case is to collect all markdown files to run some checks and generate a static site from them.
At first glance, glob() sounds like a good idea, but will stop as soon as it runs into a BUILD file.
Current Approaches
The current approach is to run the collection/generation logic outside of the sandbox, but this is a bit dirty, and I'm wondering if there is a way that is both "proper" and easy (ie, not requiring that each BUILD file explicitly exposes its markdown files.
Is there any way to specify, in the workspace, some default rules that will be added to all BUILD files?
You could write an aspect for this to aggregate markdown files in a bottom-up manner and create actions on those files. There is an example of a file_collector aspect here. I modified the aspect's extensions for your use case. This aspect aggregates all .md and .markdown files across targets on the deps attribute edges.
FileCollector = provider(
fields = {"files": "collected files"},
)
def _file_collector_aspect_impl(target, ctx):
# This function is executed for each dependency the aspect visits.
# Collect files from the srcs
direct = [
f
for f in ctx.rule.files.srcs
if ctx.attr.extension == f.extension
]
# Combine direct files with the files from the dependencies.
files = depset(
direct = direct,
transitive = [dep[FileCollector].files for dep in ctx.rule.attr.deps],
)
return [FileCollector(files = files)]
markdown_file_collector_aspect = aspect(
implementation = _file_collector_aspect_impl,
attr_aspects = ["deps"],
attrs = {
"extension": attr.string(values = ["md", "markdown"]),
},
)
Another way is to do a query on file targets (input and output files known to the Bazel action graph), and process these files separately. Here's an example querying for .bzl files in the rules_jvm_external repo:
$ bazel query //...:* | grep -e ".bzl$"
//migration:maven_jar_migrator_deps.bzl
//third_party/bazel_json/lib:json_parser.bzl
//settings:stamp_manifest.bzl
//private/rules:jvm_import.bzl
//private/rules:jetifier_maven_map.bzl
//private/rules:jetifier.bzl
//:specs.bzl
//:private/versions.bzl
//:private/proxy.bzl
//:private/dependency_tree_parser.bzl
//:private/coursier_utilities.bzl
//:coursier.bzl
//:defs.bzl
The Bazel Starlark API does strange things with files in external repositories. I have the following Starlark snippet:
print(ctx.genfiles_dir)
print(ctx.genfiles_dir.path)
print(output_filename)
ret = ctx.new_file(ctx.genfiles_dir, output_filename)
print(ret.path)
It is creating the following output:
DEBUG: build_defs.bzl:292:5: <derived root>
DEBUG: build_defs.bzl:293:5: bazel-out/k8-fastbuild/genfiles
DEBUG: build_defs.bzl:294:5: google/protobuf/descriptor.upb.c
DEBUG: build_defs.bzl:296:5: bazel-out/k8-fastbuild/genfiles/external/com_google_protobuf/google/protobuf/descriptor.upb.c
That extra external/com_google_protobuf comes seemingly out of nowhere, and it makes my rule fail:
I tell protoc to generate into ctx.genfiles_dir.path (which is bazel-out/k8-fastbuild/genfiles).
So protoc generates bazel-out/k8-fastbuild/genfiles/google/protobuf/descriptor.upb.c
Bazel fails because I didn't generate bazel-out/k8-fastbuild/genfiles/external/com_google_protobuf/google/protobuf/descriptor.upb.c
Likewise, when I try to call file.short_path on a source file from an external repository, I get a result like ../com_google_protobuf/google/protobuf/descriptor.proto. This seems quite unhelpful, so I just wrote some manual code to strip off the leading ../com_google_protobuf/.
Am I missing something? How can I write this rule in a way that doesn't feel like I'm fighting Bazel the whole time?
Am I missing something?
The basic problem, as you already realized, is that you have two path "namespaces" the one that protoc sees (i.e. import paths) and the one that bazel sees (i.e. the path you pass to declare_file().
2 things to note:
1) All paths declared with declare_file() get the path <bin dir>/<package path incl. workspace>/<path you passed to declare_file()>
2) All actions are executed from <bin dir> (unless output_to_genfils=True in which case this switches to <gen dir> as in your example.
Trying to solve the exact same problem you encountered, I resorted to stripping the known path from the output_file's path to determine which directory to pass as p:
# This code is run from the context of the external protobuf dependency
proto_path = "google/a/b.proto"
output_file = ctx.actions.declare_file(proto_path)
# output_file.path would be `<gen_dir>/external/protobuf/google/a/b.proto`
# Strip the known proto_path from output_file.path
protoc_prefix = output_file.path[:-len(proto_path)]
print(protoc_prefix) # Prints: <gen_dir>/external/protobuf
command = "{protoc} {proto_paths} {cpp_out} {plugin} {plugin_options} {proto_file}".format(
...
cpp_out = "--cpp_out=" + protoc_prefix,
...
)
Alternatives
You may also be able to construct the same path with ctx.bin_dir, ctx.label.workspace_name, ctx.label.package, and ctx.label.name.
Misc.
proto_library recently gained an attribute strip_import_prefix. When used, the above is not correct, as all dependent files are symlinked into a new directory from which they have the relative paths declared with strip_import_prefix.
The path format is:
<bin dir>/<repo>/<package>/_virtual_base/<label name>/<path `import`ed in .proto files>
i.e.
<bin dir>/external/protobuf/_virtual_base/b_proto/google/a/b.proto
Assuming you are building an external repo called protobuf, which contains a BUILD file at its root with a target named b_proto, which in turn, relies on a proto_library wrapping google/a/b.proto AND uses the strip_import_prefix attribute.
I'd like to the use app.config file of my F# to store versioning information. I discovered the FSharp.Configuration type provider which seemed like it'd be simple enough. However, I'm running in to an error I can't diagnose.
Below is a screen shot of a version.config file (identical to the one in the link above) and a scratch pad.
As you can see, calling Settings auto populates a drop-down of everything in the <appSettings> chunk of the config but when I try to run something,
I get an error saying that the thing I'm looking for can't be found in the <appSettings> section of the config file.
What's causing this error, especially considering that it clearly is finding it in the config file, given it's auto-populating? What can I do to prevent this from happening again?
You have bumped into this issue.
When you run the Configuration provider in FSI it will look not for the app's config file but FSI's config file. One way to get around this is by specifying the exe's config file explicitly. Here's an example:
open FSharp.Configuration
open System
type Settings = AppSettings<"app.config">
[<EntryPoint>]
let main argv =
let path = System.IO.Path.Combine [|__SOURCE_DIRECTORY__ ;"bin";"release";"ConfigApplication.exe" |]
Settings.SelectExecutableFile path
Settings.TestBool <- false // change a setting
printfn "%A" Settings.Test2 // read another setting
Console.ReadLine() |> ignore
0 // return an integer exit code
This will take the App.config file in the source directory, but use the ConfigApplication.exe.config file in the binaries directory.
If you just need to set the DB's connection string, it's actually easier, if the SQL type provider has a config setting parameter, just specify the config file there (and set it to Always copy in VS), if you add that to .gitignore you can have many different app.config files with different connection strings.
You could also use the YAML provider, it has two advantages, it's not XML and it's not an erasing type provider.
Is there any way to share a variable by including a fsx script within another fsx script.
e.g script buildConsts.fsx contains
let buildDir = "./build/"
I want to reference this in other build scripts e.g.
#load #".\buildConsts.fsx"
let testDlls = !! (buildDir + "*Test*.dll")
When I attempt to run the script the 'buildDir' variable the script fails to compile.
This is a fairly common approach that is used with tools such as MSBuild and PSAKE to modularise scripts. Is this the correct approach with FAKE ?
What you're doing should work - what exactly is the error message that you're getting?
I suspect that the problem is that F# automatically puts the contents of a file in a module and you need to open the module before you can access the constants. The module is named based on the file name, so in your case buildConsts.fsx will generate a module named BuildConsts. You should be able to use it as follows:
#load #".\buildConsts.fsx"
open BuildConsts
let testDlls = !! (buildDir + "*Test*.dll")
You can also add an explicit module declaration to buildconsts.fsx, which is probably a better idea as it is less fragile (won't change when you rename the file):
moule BuildConstants
let buildDir = "./build/"
I'm using aapt tool to remove some files from different folders of my apk. This works fine.
But when I want to add files to the apk, the aapt tool add command doesn't let me specify the path to where I want the file to be added, therefore I can add files only to the root folder of the apk.
This is strange because I don't think that developers would never want to add files to a subfolder of the apk (res folder for example). Is this possible with aapt or any other method? Cause removing files from any folder works fine, and adding file works only for the root folder of the apk. Can't use it for any other folder.
Thanks
The aapt tool retains the directory structure specified in the add command, if you want to add something to an existing folder in an apk you simply must have a similar folder on your system and must specify each file to add fully listing the directory. Example
$ aapt list test.apk
res/drawable-hdpi/pic1.png
res/drawable-hdpi/pic2.png
AndroidManifest.xml
$ aapt remove test.apk res/drawable-hdpi/pic1.png
$ aapt add test.apk res/drawable-hdpi/pic1.png
The pic1.png that will is added resides in a folder in the current working directory of the terminal res/drawable-hdpi/ , hope this answered your question
There is actually a bug in aapt that will make this randomly impossible. The way it is supposed to work is as the other answer claims: paths are kept, unless you pass -k. Let's see how this is implemented:
The flag that controls whether the path is ignored is mJunkPath:
bool mJunkPath;
This variable is in a class called Bundle, and is controlled by two accessors:
bool getJunkPath(void) const { return mJunkPath; }
void setJunkPath(bool val) { mJunkPath = val; }
If the user specified -k at the command line, it is set to true:
case 'k':
bundle.setJunkPath(true);
break;
And, when the data is being added to the file, it is checked:
if (bundle->getJunkPath()) {
String8 storageName = String8(fileName).getPathLeaf();
printf(" '%s' as '%s'...\n", fileName, storageName.string());
result = zip->add(fileName, storageName.string(),
bundle->getCompressionMethod(), NULL);
} else {
printf(" '%s'...\n", fileName);
result = zip->add(fileName, bundle->getCompressionMethod(), NULL);
}
Unfortunately, the one instance of Bundle used by the application is allocated in main on the stack, and there is no initialization of mJunkPath in the constructor, so the value of the variable is random; without a way to explicitly set it to false, on my system I (seemingly deterministically) am unable to add files at specified paths.
However, you can also just use zip, as an APK is simply a Zip file, and the zip tool works fine.
(For the record, I have not submitted the trivial fix for this as a patch to Android yet, if someone else wants to the world would likely be a better place. My experience with the Android code submission process was having to put up with an incredibly complex submission mechanism that in the end took six months for someone to get back to me, in some cases with minor modifications that could have just been made on their end were their submission process not so horribly complex. Given that there is a really easy workaround to this problem, I do not consider it important enough to bother with all of that again.)