Edit: The example below did actually work, I misinterpreted the output the compiler gave me. The answer may still be helpful to some.
Is there a way for an action in a rule to generate a file that is consumed by a later action in that same rule?
E.g.:
def _example_rule_impl(ctx):
thefile = ctx.actions.declare_file("required_file.json")
ctx.actions.write(
output = thefile,
content = "CONTENT",
)
args = ctx.actions.args()
args.add("--config", thefile)
ctx.actions.run(
inputs = ctx.files.srcs + ctx.files.deps + [thefile],
outputs = outs,
arguments = [args],
progress_message = "Compiling...",
executable = ctx.executable._compiler,
)
The main problem with this seems to be, that all action outputs seem to be written to bazel-out, but the run action requires the generated file to be written next to the srcs and deps files in the execroot for it to work. Is there a way to have an action write to the execroot or is this not the correct approach?
Actions taking the outputs of other actions as inputs is a very typical thing to do, and should basically just work as you've set it up. Bazel takes care of handling input and output files, e.g., on linux using a lot of symlinks, mounting things in sandboxes, uploading and downloading files for remote execution, etc.
See
https://docs.bazel.build/versions/main/output_directories.html
Also, regarding this line:
inputs = ctx.files.srcs + ctx.files.deps + [thefile],
Depending on what you need to do, you may want to use depsets for performance reasons. See https://docs.bazel.build/versions/main/skylark/depsets.html
and in particular at the end https://docs.bazel.build/versions/main/skylark/depsets.html#performance
Related
In LibreOffice, It is possible to run python scripts like this:
sURL = "vnd.sun.star.script:file.function?language=Python&location=document"
oScript = scriptProv.getScript(sURL)
x = oScript.Invoke(args, Array(), Array())
In that example 'file' is a filename, and 'function' is the name of a function in that file.
Is it possible to embed script in that URL? sURL="vnd.." & scriptblock & "?language.."
(It seems like the kind of thing that might be possible with the correct URL, or might not be possible if just not supported).
We can use Python's eval() function. Here is an example inspired by JohnSUN's explanation in the discussion. Note: xray() uses XrayTool to show output, but you could replace that line with any output method of your choosing, such as writing to a file.
def runArbitraryCode(*args):
url = args[0]
codeString = url.split("&codeToRun=")[1]
x = eval(codeString)
xray(x)
Now enter this formula in Calc and Ctrl+click on it.
=HYPERLINK("vnd.sun.star.script:misc_examples.py$runArbitraryCode?language=Python&location=user&codeToRun=5+1")
Result: 6
Obligatory caveat: Running eval() on an unknown string is about the worst idea imaginable in terms of security. So hopefully you're the one controlling the URL and not some black hat hacker!
I have a code analysis tool that I'd like to run for each cc_library (and cc_binary, silently implied for rest of the question). The tool has a CLI interfaces taking:
A tool project file
Compiler specifics, such as type sizes, built-ins, macros etc.
Files to analyze
File path, includes, defines
Rules to (not) apply
Files to add to the project
Options for synchronizing files with build data
JSON compilation database
Parse build log
Analyze and generate analysis report
I've been looking at how to integrate this in Bazel so that the files to analyze AND the associated includes and defines are updated automatically, and that any analysis result is properly cached. Generating JSON compilation database (using third party lib) or parsing build log both requires separate runs and updating the source tree. For this question I consider that a workaround I'm trying to remove.
What I've tried so far is using aspects, adding an analysis aspect to any library. The general idea is having a base project file holding library invariant configuration, appended with the cc_library files to analysis, and finally an analysis is triggered generating the report. But I'm having trouble to execute, and I'm not sure it's even possible.
This is my aspect implementation so far, trying to iterate through cc_library attributes and target compilation context:
def _print_aspect_impl(target, ctx):
# Make sure the rule has a srcs attribute
if hasattr(ctx.rule.attr, 'srcs'):
# Iterate through the files
for src in ctx.rule.attr.srcs:
for f in src.files.to_list():
if f.path.endswith(".c"):
print("file: ")
print(f.path)
print("includes: ")
print(target[CcInfo].compilation_context.includes)
print("quote_includes: ")
print(target[CcInfo].compilation_context.quote_includes)
print("system_includes: ")
print(target[CcInfo].compilation_context.system_includes)
print("define: " + define)
print(ctx.rule.attr.defines)
print("local_defines: ")
print(ctx.rule.attr.local_defines)
print("") # empty line to separate file prints
return []
What I cannot figure out is how to get ALL includes and defines used when compiling the library:
From libraries depended upon, recursively
copts, defines, includes
From the toolchain
features, cxx_builtin_include_directories
Questions:
How do I get the missing flags, continuing on presented technique?
Can I somehow retrieve the compile action command string?
Appended to analysis project using the build log API
Some other solution entirely?
Perhaps there is something one can do with cc_toolchain instead of aspects...
Aspects are the right tool to do that. The information you're looking for is contained in the providers, fragments, and toolchains of the cc_* rules the aspect has access to. Specifically, CcInfo has the target-specific pieces, the cpp fragment has the pieces configured from the command-line flag, and CcToolchainInfo has the parts from the toolchain.
CcInfo in target tells you if the current target has that provider, and target[CcInfo] accesses it.
The rules_cc my_c_compile example is where I usually look for pulling out a complete compiler command based on a CcInfo. Something like this should work from the aspect:
load("#rules_cc//cc:action_names.bzl", "C_COMPILE_ACTION_NAME")
load("#rules_cc//cc:toolchain_utils.bzl", "find_cpp_toolchain")
[in the impl]:
cc_toolchain = find_cpp_toolchain(ctx)
feature_configuration = cc_common.configure_features(
ctx = ctx,
cc_toolchain = cc_toolchain,
requested_features = ctx.features,
unsupported_features = ctx.disabled_features,
)
c_compiler_path = cc_common.get_tool_for_action(
feature_configuration = feature_configuration,
action_name = C_COMPILE_ACTION_NAME,
)
[in the loop]
c_compile_variables = cc_common.create_compile_variables(
feature_configuration = feature_configuration,
cc_toolchain = cc_toolchain,
user_compile_flags = ctx.fragments.cpp.copts + ctx.fragments.cpp.conlyopts,
source_file = src.path,
)
command_line = cc_common.get_memory_inefficient_command_line(
feature_configuration = feature_configuration,
action_name = C_COMPILE_ACTION_NAME,
variables = c_compile_variables,
)
env = cc_common.get_environment_variables(
feature_configuration = feature_configuration,
action_name = C_COMPILE_ACTION_NAME,
variables = c_compile_variables,
)
That example only handles C files (not C++), you'll have to change the action names and which parts of the fragment it uses appropriately.
You have to add toolchains = ["#bazel_tools//tools/cpp:toolchain_type"] and fragments = ["cpp"] to the aspect invocation to use those. Also see the note in find_cc_toolchain.bzl about the _cc_toolchain attr if you're using legacy toolchain resolution.
The information coming from the rules and the toolchain is already structured. Depending on what your analysis tool wants, it might make more sense to extract it directly instead of generating a full command line. Most of the provider, fragment, and toolchain is well-documented if you want to look at those directly.
You might pass required_providers = [CcInfo] to aspect to limit propagation to rules which include it, depending on how you want to manage propagation of your aspect.
The Integrating with C++ Rules documentation page also has some more info.
Bazel has been working great for me recently, but I've stumbled upon a question for which I have yet to find a satisfactory answer:
How can one collect all files bearing a certain extension from the workspace?
Another way of phrasing the question: how could one obtain the functional equivalent of doing a glob() across a complete Bazel workspace?
Background
The goal in this particular case is to collect all markdown files to run some checks and generate a static site from them.
At first glance, glob() sounds like a good idea, but will stop as soon as it runs into a BUILD file.
Current Approaches
The current approach is to run the collection/generation logic outside of the sandbox, but this is a bit dirty, and I'm wondering if there is a way that is both "proper" and easy (ie, not requiring that each BUILD file explicitly exposes its markdown files.
Is there any way to specify, in the workspace, some default rules that will be added to all BUILD files?
You could write an aspect for this to aggregate markdown files in a bottom-up manner and create actions on those files. There is an example of a file_collector aspect here. I modified the aspect's extensions for your use case. This aspect aggregates all .md and .markdown files across targets on the deps attribute edges.
FileCollector = provider(
fields = {"files": "collected files"},
)
def _file_collector_aspect_impl(target, ctx):
# This function is executed for each dependency the aspect visits.
# Collect files from the srcs
direct = [
f
for f in ctx.rule.files.srcs
if ctx.attr.extension == f.extension
]
# Combine direct files with the files from the dependencies.
files = depset(
direct = direct,
transitive = [dep[FileCollector].files for dep in ctx.rule.attr.deps],
)
return [FileCollector(files = files)]
markdown_file_collector_aspect = aspect(
implementation = _file_collector_aspect_impl,
attr_aspects = ["deps"],
attrs = {
"extension": attr.string(values = ["md", "markdown"]),
},
)
Another way is to do a query on file targets (input and output files known to the Bazel action graph), and process these files separately. Here's an example querying for .bzl files in the rules_jvm_external repo:
$ bazel query //...:* | grep -e ".bzl$"
//migration:maven_jar_migrator_deps.bzl
//third_party/bazel_json/lib:json_parser.bzl
//settings:stamp_manifest.bzl
//private/rules:jvm_import.bzl
//private/rules:jetifier_maven_map.bzl
//private/rules:jetifier.bzl
//:specs.bzl
//:private/versions.bzl
//:private/proxy.bzl
//:private/dependency_tree_parser.bzl
//:private/coursier_utilities.bzl
//:coursier.bzl
//:defs.bzl
I have a binary that takes as input a single file and produces an unknown number of header and source C++ files into a single directory. I would like to be able to write a target like:
x_library(
name = "my_x_library",
src = "source.x",
)
where x_library is a macro that ultimately produces the cc_library from the output files. However, I can't bundle all the output files inside the rule implementation or inside the macro. I tried this answer but it doesn't seem to work anymore.
What's the common solution to this problem? Is it possible at all?
Small example of a macro using a genrule (not a huge fan) to get one C file and one header and provide them as a cc_library:
def x_library(name, src):
srcfile = "{}.c".format(name)
hdrfile = "{}.h".format(name)
native.genrule(
name = "files_{}".format(name),
srcs = [src],
outs = [srcfile, hdrfile],
cmd = "./generator.sh $< $(OUTS)",
tools = ["generator.sh"],
)
native.cc_library(
name = name,
srcs = [srcfile],
hdrs = [hdrfile],
)
Used it like this then:
load(":myfile.bzl", "x_library")
x_library(
name = "my_x_library",
src = "source.x",
)
cc_binary(
name = "tgt",
srcs = ["mysrc.c"],
deps = ["my_x_library"],
)
You should be able to extend that with any number of files (and for C++ content; IIRC the suffices are use for automagic decision how to call the tools) as long as your generator input -> generated content is known and stable (generally a good thing for a build). Otherwise you can no longer use genrule as you need your custom rule (probably a good thing anyways) to use TreeArtifact as described in the linked answer. Or two, one with .cc suffix and one with .hh so that you can pass them to cc_library.
Suppose I have this macro which creates a native.cc_binary target:
def build_it(name, **kwargs):
native.cc_binary(
name = name + ".out",
linkopts = [
"-Lsomedir",
"-lsomelib"
],
**kwargs)
And I also have this rule which takes some sources, runs a tool on them, and generates a value, writing that value to an output file:
def _write_value_impl:
args = [f.path for f in ctx.files.srcs] + [ctx.outputs.out.path]
ctx.actions.run(
inputs = ctx.files.srcs,
outputs = [ctx.outputs.out],
arguments = args,
executable = ctx.executable._tool
)
write_value = rule(
implementation=_write_value_impl,
attrs = {
"srcs": attr.label_list(allow_files = True),
"out": attr.output(mandatory = True),
"_tool": attr.label(
executable = True,
allow_files = True,
default = Label("//tools:generate_value")
}
)
Okay, what I'd like to do is modify the macro so that it adds the value generated by the write_value rule to the linkopts. Something like this:
def build_it(name, value, **kwargs):
native.cc_binary(
name = name + ".out",
linkopts = [
"-Lsomedir",
"-lsomelib",
"-Wl,--defsym=SOME_SYMBOL={}".format(value)
],
**kwargs)
How do I make this work? The problem is that the target of build_it is generated at analysis time, but the value it needs is generated at evaluation time. Also, the value got put into a file. How do I get the value out of the file and give it to the macro?
I suspect that instead of a macro, I need a rule, but how do I get the rule to call native.cc_binary?
You can write a repository_rule() to create files and generate values prior to the loading phase, and then files in the #external_repo//... will be accessible by the rules during analysis. https://docs.bazel.build/versions/master/skylark/repository_rules.html
This can't be done in Bazel, precisely because of what you mentioned. All the inputs to rules need to be determined in the analysis phase rather than the execution phase. Bazel wants to build the complete action graph before executing any actions, and this would require the write_value rule to run before build_it could be analyzed.
A workaround might be to generate the BUILD file yourself outside of Bazel beforehand, and then use the generated BUILD file during your build.
Another workaround is to hard-code the linkopts to specify them to what you expect them to be. Then in write_value check if they are what you expected and if not throw an exit code. That way Bazel will at least warn you when they aren't going to match, but it will take some effort to update both places so they're aligned again.
For your specific problem there is a concept of linker scripts, and even implicit linker scripts. Perhaps you could generate one and supply that to cc_binary in the srcs attribute. You may need to name it as a .o file (even though it's not an object file). The GCC linker documentation says:
If you specify a linker input file which the linker can not recognize
as an object file or an archive file, it will try to read the file as
a linker script.