I'm trying to translate my company's project from legacy build tool to bazel. Now I'm facing this problem and searched a lot, but unfortunately, I haven't had a clue so far.
Here's the thing:
For compliance with open source audit, we must provide a list of open-source software which are built into our binary. As external dependencies are introduced by repository rules, my intuitive thought is to query these rules and get the URLs. However, subcommand query/cquery hasn't provided such functionality yet AFAIK, it can print rule/target/buildfiles but no repository rules nor their attributes.
Is there a way that I can gather such information from repository rules in WORKSPACE? It's not viable to do it manually as there are thousands of projects in my company and the dependencies also change frequently.
For example, a workspace rule:
http_archive(
name = "testrunner",
urls = ["https://github.com/testrunner/v2.zip"],
sha256 = "..."
)
This dependency is used by a rule named "my_target", so what i expected is that the dependency could be queried like this:
> bazel queryExtDep my_target
External Dependency of my_target: name->testrunner, urls = "https://github.com/testrunner/v2.zip"
--experimental_repository_resolved_file will give you all that information in a single Starlark file, which you can easily process with Starlark or Python etc to extract the information you're looking for.
The resolved file looks something like this:
resolved = [
...,
{
"original_rule_class": "#bazel_tools//tools/build_defs/repo:git.bzl%git_repository",
"original_attributes": {
"name": "com_google_protobuf",
"remote": "https://github.com/google/protobuf",
"branch": "master"
},
"repositories": [
{
"rule_class": "#bazel_tools//tools/build_defs/repo:git.bzl%git_repository",
"attributes": {
"remote": "https://github.com/google/protobuf",
"commit": "78ba021b846e060d5b8f3424259d30a1f3ae4eef",
"shallow_since": "2018-02-07",
"init_submodules": False,
"verbose": False,
"strip_prefix": "",
"patches": [],
"patch_tool": "patch",
"patch_args": [
"-p0"
],
"patch_cmds": [],
"name": "com_google_protobuf"
}
}
]
}
]
This includes the original attributes, which is where that URL you're looking for is. It also includes any additional information returned by the repository rule (ie for git_repository, the actual commit a given ref refers to).
I got that example from blog post introducing that flag, which also has some more background.
Related
I have the issue I would like to automate via a script so tat .jnlp will be added as an allowable type of file , is there a command like or powershell or regedit that will add it?
The latest file types policies are published in the Chromium source code. You could clearly see that the danger_level of .jnlp type files is DANGEROUS. Therefor Edge will warn users that this file may harm their computers. Let users continue or discard the file.
If you ensure that the content(download file) on the site is safe, you can use this policy to specify the file types that are allowed to be downloaded continuously from a specific site: ExemptDomainFileTypePairsFromFileTypeDownloadWarnings.
Example:
[ { "file_extension": "jnlp", "domains": ["contoso.com"] }, { "file_extension": "exe", "domains": ["contoso.com"] }, { "file_extension": "swf", "domains": ["*"] } ]
If you want to achieve the same function through the registry, you can set it under this path: SOFTWARE\Policies\Microsoft\Edge\ExemptDomainFileTypePairsFromFileTypeDownloadWarnings
Bazel has been working great for me recently, but I've stumbled upon a question for which I have yet to find a satisfactory answer:
How can one collect all files bearing a certain extension from the workspace?
Another way of phrasing the question: how could one obtain the functional equivalent of doing a glob() across a complete Bazel workspace?
Background
The goal in this particular case is to collect all markdown files to run some checks and generate a static site from them.
At first glance, glob() sounds like a good idea, but will stop as soon as it runs into a BUILD file.
Current Approaches
The current approach is to run the collection/generation logic outside of the sandbox, but this is a bit dirty, and I'm wondering if there is a way that is both "proper" and easy (ie, not requiring that each BUILD file explicitly exposes its markdown files.
Is there any way to specify, in the workspace, some default rules that will be added to all BUILD files?
You could write an aspect for this to aggregate markdown files in a bottom-up manner and create actions on those files. There is an example of a file_collector aspect here. I modified the aspect's extensions for your use case. This aspect aggregates all .md and .markdown files across targets on the deps attribute edges.
FileCollector = provider(
fields = {"files": "collected files"},
)
def _file_collector_aspect_impl(target, ctx):
# This function is executed for each dependency the aspect visits.
# Collect files from the srcs
direct = [
f
for f in ctx.rule.files.srcs
if ctx.attr.extension == f.extension
]
# Combine direct files with the files from the dependencies.
files = depset(
direct = direct,
transitive = [dep[FileCollector].files for dep in ctx.rule.attr.deps],
)
return [FileCollector(files = files)]
markdown_file_collector_aspect = aspect(
implementation = _file_collector_aspect_impl,
attr_aspects = ["deps"],
attrs = {
"extension": attr.string(values = ["md", "markdown"]),
},
)
Another way is to do a query on file targets (input and output files known to the Bazel action graph), and process these files separately. Here's an example querying for .bzl files in the rules_jvm_external repo:
$ bazel query //...:* | grep -e ".bzl$"
//migration:maven_jar_migrator_deps.bzl
//third_party/bazel_json/lib:json_parser.bzl
//settings:stamp_manifest.bzl
//private/rules:jvm_import.bzl
//private/rules:jetifier_maven_map.bzl
//private/rules:jetifier.bzl
//:specs.bzl
//:private/versions.bzl
//:private/proxy.bzl
//:private/dependency_tree_parser.bzl
//:private/coursier_utilities.bzl
//:coursier.bzl
//:defs.bzl
I am writing a sample C++ project that uses Bazel to serve as an example idiom for other collaborators to follow.
Here is the repository: https://github.com/thinlizzy/bazelexample
I am interested to know if I am doing it 'right', more specifically about this file: https://github.com/thinlizzy/bazelexample/blob/38cc07931e58ff5a888dd6a83456970f76d7e5b3/demo/BUILD
when regarding to pick particular implementations.
cc_library(
name = "demo",
srcs = ["demo.cpp"],
deps = [
"//example:frontend",
],
)
cc_binary(
name = "main_win",
deps = [
":demo",
"//example:impl_win",
],
)
cc_binary(
name = "main_linux",
deps = [
":demo",
"//example:impl_linux",
],
)
Is this following a correct/expected idiom for Bazel projects? I am doing this way already for other projects, by concentrating all the platform-specific dependencies in separate targets and then the binaries just depend on them.
Someone in bazel-discuss list told me to use select, instead, but my attempts failed to 'detect' the operating system. I'm sure I did something wrong, but the lack of info and examples don't tell me much how to use it properly.
#bazel_tools contains predefined platform conditions:
$ bazel query #bazel_tools//src/conditions:all
#bazel_tools//src/conditions:windows_msys
#bazel_tools//src/conditions:windows_msvc
#bazel_tools//src/conditions:windows
#bazel_tools//src/conditions:remote
#bazel_tools//src/conditions:host_windows_msys
#bazel_tools//src/conditions:host_windows_msvc
#bazel_tools//src/conditions:host_windows
#bazel_tools//src/conditions:freebsd
#bazel_tools//src/conditions:darwin_x86_64
#bazel_tools//src/conditions:darwin
You can use them directly in the BUILD file:
cc_library(
name = "impl",
srcs = ["Implementation.cpp"] + select({
"#bazel_tools//src/conditions:windows": ["ImplementationWin.cpp"],
"#bazel_tools//src/conditions:darwin": ["ImplementationMacOS.cpp"],
"//conditions:default": ["ImplementationLinux.cpp"],
}),
# .. same for hdrs and data
)
cc_binary(
name = "demo",
deps = [":impl"],
)
See the documentation for select for details on the syntax.
Add a .bazelrc to your project. Add the lines build:vs2019 --cxxopt=/std:c++14 and build:gcc --cxxopt=-std=c++14. Build your code bazel build --config=msvc //... or bazel build --config=gcc //....
#Vertexwahn's answer caused some confusion on my end, so I hope this answer helps clarify a bit. While his answer does not directly tie into the question, it may be of use to others trying to build on entirely different platforms without file specific inclusions.
Here is a link to where I answered that particular question: How do I specify portable build configurations for different operating systems for Bazel?
I'm using the Artifactory plugin in Jenkins. I have a situation where I need to pull docker images from Artifactory which have a particular property set. I can search for them using the REST API, but I have not found a way to do this using the Jenkins plugin. Anybody know if there is a way?
Thanks.
This is the download spec schema from the documentation:
{
"files": [
{
"pattern" or "aql": "[Mandatory]",
"target": "[Mandatory]",
"props": "[Optional]",
"recursive": "[Optional, Default: true]",
"flat" : "[Optional, Default: false]"
}
]
}
Where from my point of view you have a couple of options to search by propeties; one would be using aql where you have such option. Second one would be directly using props, which main purpose seems to be exactly that:
props: List of "key=value" pairs separated by a semi-colon. (For
example, "key1=value1;key2=value2;key3=value3). Only artifacts with
all of the specified properties and values will be downloaded.
Is there an easy way to get hold of a path object so I can check if a given label path exists. Say for example if path.exists("#external_project_name//:filethatmightexist.txt"):. I can see that the repository context has this. But I need to have a wrapping repository rule. Is it possible to do this in a macro or Skylark native call instead?
Even with a repository_rule, I had a lot of trouble with this due to what you already pointed out:
if you create a Label with a path that doesn't exist, it will cause the build to fail
But if you're willing to do a repository rule, here's a possible solution...
In this example, my rule allows specification of a default configuration if a config file is not present. The configuration can be checked into .gitignore and overridden for individual developers, but work out of the box for most cases.
I think I understand why the ctx.actions have sibling arguments now, same idea here. The trick is config_file_location is a true label, and then config_file is a string attribute. I chose BUILD arbitrarily, but since all workspaces have a top level BUILD that's public seemed legit-ish.
WORKSPACE Definition
...
workspace(name="E02_mysql_database")
json_datasource_configuration(name="E02_datasources",
config_file_location="#E02_mysql_database//:BUILD",
config_file="database.json")
The definition for json_datasource_configuration looks like this:
json_datasource_configuration = repository_rule(
attrs = {
"config_file_location": attr.label(
doc="""
Path relative to the repository root for a datasource config file.
"""),
"config_file": attr.string(
doc="""
Config file, maybe absent
"""),
"default_config": attr.string(
# better way to do this?
default="None",
doc = """
If no config is at the path, then this will be the default config.
Should look something like:
{
"datasource_name": {
"host": "<host>"
"port": <port>
"password": "<password>"
"username": "<username>"
"jdbc_connection_string": "<optional>"
}
}
There can be more than datasource configured... maybe, eventually.
""",
),
},
local = True,
implementation = _json_config_impl,
)
Then in the rule I can test for the file existence, and if not present, do other logic.
def _json_config_impl(ctx):
"""
Allows you to specify a file on disk to use for data connection.
If you pass a default
"""
config_path = ctx.path(ctx.attr.config_file_location).dirname.get_child(ctx.attr.config_file)
config = ""
if config_path.exists:
config = ctx.read(config_path)
elif ctx.attr.default_config == "None":
fail("Could not find config at %s, you must supply a default_config if this is intentional" % ctx.attr.config_file)
else:
config = ctx.attr.default_config
...
probably too late to help, but your question is the only thing I found referencing this goal. If someone knows a better way I am looking for other options. It's complicated to explain to other developers why the rule has to work the way it does.
Also note, if you change the config file, you have to clean to get the workspace to re-read the config. I haven't been able to figure out any way to fix that. glob() does not work in the workspace.