After downloading an archive throug http_archive I'd like to run a script to generate a BUILD file from the folder structure and Cmake files in it (I currently do that by hand and it is easy enough that it could be scripted). I don't find anything on how to open, read and write files in the starlark documentation but since http_archive itself is loaded from a bzl file (haven't found the source of that file yet though...) and generates BUILD files (by unpacking them from archives) I guess it must be possible to write a wrapper for http_archive that also generates the BUILD file?
This is a perfect use case for a custom repository rule. That lets you run arbitrary commands to generate the files for the repository, along with some helpers for common operations like downloading a file over HTTP using the repository cache (if configured). A repository rule conceptually similar to a normal rule, but with much less infrastructure because it's running during the loading phase when most of the Bazel infrastructure doesn't apply yet.
The starlark implementation of http_archive is in http.bzl. The core of it is a single call to ctx.download_and_extract. Your custom rule should do that too. http_archive then calls workspace_and_buildfile and patch from util.bzl, which do what they sound like. Instead of workspace_and_buildfile, you should call ctx.execute to run your command to generate the BUILD file. You could call patch if you want, or skip that functionality if you're not going to use it.
The repository_ctx page in the documentation is the top-level reference for everything your repository rule's implementation function can do, if you want to extend it further.
When using http_archive, you can use the build_file argument to create a BUILD file. To generate it dynamically, I think you can use the patch_cmds argument to run external commands.
Related
A few years ago I wrote a set of wrappers for Bazel that enabled me to use it to build FPGA code. The FPGA bit is only relevant because the full clean build takes many CPU days so I really care about caching and minimizing rebuilds.
Using Bazel v0.28 I never found a way to have my Bazel package depend on a single source file from somewhere else in the git repo. It felt like this wasn't something Bazel was designed for.
We want to do this because we have a library of VHDL source files that are parameterized and the parameters are set in the instantiating VHDL source. (VHDL generics). If we declare this library as a Bazel package in its own right then a change to one library file would rebuild everything (at huge time cost) when in practice only a couple of steps might need to be rebuilt.
I worked around this with a python script to copy all the individual source files into a subdirectory and then generate the BUILD file to reference these copies. The resulting build process is:
call python preparation script
bazel build //:allfpgas
call python result extractor
This is clearly quite ugly but the benefits were huge so we live with it.
Now we want to leverage Bazel to build our Java, C++ etc so I wanted to revisit and try and make everything work with Bazel alone.
In the latest Bazel is there a way to have a BUILD package depend on individual source files outside of the package directory? If Bazel cant, would buck pants or please.build work better for our use case?
The Bazel rules for most languages support doing something like this already. For example, the Python rules bundle source files from multiple packages together, and the C++ rules manage include files from other packages. Somehow the rule has to pass the source files around in providers, so that another rule can generate actions which use them. Hard to be more specific without knowing which rules you're using.
If you just want to copy the files, you can do that in bazel with a genrule. In the package with the source file:
exports_files(["templated1.vhd", "templated2.vhd"])
In the package that uses it:
genrule(
name = "copy_templates",
srcs = ["//somewhere:templated1.vhd", "//somewhere:templated2.vhd"],
outs = ["templated1.vhd", "templated2.vhd"],
cmd = "cp $(SRCS) $(RULEDIR)",
)
some_library(
srcs = ["templated1.vhd", "templated2.vhd", "other.vhd"],
)
If you want to deduplicate that across multiple packages that use it, put the filenames in a list and write a macro to create the genrule.
"(new_)git_repository" and "(new_)http_archive" workspace rules deal with external projects in such way that any external dependency is copied to temporary directory linked to workspace as ${WORKSPACE}/bazel-workspace/external/${EXTERNAL_DEP_NAME} on build or prefetch.
I'd like to save external dependencies locally in my repo, so if remote repository vanishes i'd have copy of dependency even on a new machine, where it wasn't cached.
Can I somehow change default behaviour without writing custom workspace rule?
Bazel does have a flag you could use for this: --experimental_repository_cache. It is designed to be a system-wide cache so that multiple projects on one machine don't have to re-download dependencies, but you could use it per-repository. Basically you'd say:
bazel build --experimental_repository_cache=$PWD/my_cache //foo
Then all external repositories would be downloaded to the my_cache directory in your project.
This is a cache keyed by the hash of your external dependencies' content, so it's not going to be very human-readable, but it would let you keep your external dependencies in your VCS fairly easily.
(Theoretically you could even check in a .bazelrc file to specify this option by default, but --experimental_repository_cache only takes an absolute path right now, so it's a bit impractical. I filed a bug to handle the relative path use case.)
I might be wrong but it sounds like you want to just check it in the VCS. If we're talking about an http archive then download it manually, stick it under the relevant "third_party" sub folder with the BUILD file you craft for it and you're done.
If you want to use Bazel mechanisms to download and check-in the external dependencies then this isn't currently supported.
Maybe you should open an issue
So, at work, I frequently have to create virtually identical ant scripts. Basically the application we provide to our clients is designed to be easily extensible, and we offer a service of designing and creating custom modules for it. Because of the complexity of our application, with lots of cross dependencies, I tend to develop the module within our core dev environment, compile it using IntelliJ, and then run a basic ant script that does the following tasks:
1) Clean build directory
2) Create build directory and directory hierarchy based on package paths.
3) Copy class files (and source files to a separate sources directory).
4) Jar it up.
The thing is, to do this I need to go through the script line by line and change a bunch of property names, so it works for the new use case. I also save all the scripts in case I need to go back to them.
This isn't the worst thing in the world, but I'm always looking for a better way to do things. Hence my idea:
For each specific implementation I would provide an ant script (or other file) of just properties. Key-value pairs, which would have specific prefixes for each key based on what it's used for. I would then want my ant script to run the various tasks, executing each one for the key-value pairs that are appropriate.
For example, copying the class files. I would have a property with a name like "classFile.filePath". I would want the script to call the task for every property it detects that starts with "classFile...".
Honestly, from my current research so far, I'm not confident that this is possible. But... I'm super stubborn, and always looking for new creative options. So, what options do I have? Or are there none?
It's possible to dynamically generate ANT scripts, for example the following does this using an XML input file:
Use pure Ant to search if list of files exists and take action based on condition
Personally I would always try and avoid this level of complexity. Ant is not a programming language.
Looking at what you're trying to achieve it does appear you could benefit from packaging your dependencies as jars and using a Maven repository manager like Nexus or Artifactory for storage. This would simplify each sub-project build. When building projects that depend on these published libraries you can use a dependency management tool like Apache ivy to download them.
Hope that helps your question is fairly broad.
I want to pick up certificates from a different paths and add them to the final zip file. I have a xml file which has the location of all the certs. I need maven to read from this xml file and pick up the certs. I am struck here.
As reading files is generally not an intended maven use case, getting the cert into your final artifact will be cumbersome.
I see mainly 2 options for you.
First, use the gmaven-plugin and execute a Groovy script to copy your files to an appropriate place ( somewhere into the target folder i guess).
This anwser on SO provides hints how to do that.
An alternative would be to pack this process into your own maven-plugin, which is less difficult than it sounds. The advantage would be, that the copy logic is sealed in the plugin (in contrast to openly accessible Groovy script in the pom) and is therefor more maintainable and secure.
I have a collection of repos under the same root directory. Each repo contains a build.fsx to compile & test etc.
I want to create one FAKE build.fsx in the root directory that can trigger the build.fsx scripts in the sibling repo directories.
I'm not worried about the loop, but how best to call another build.fsx from within the originating build.fsx?
I am sure Shell.Exec("./packages/tools/FAKE.exe", "./otherdir/build.fsx") would work, but is there a more seamless approach built into FAKE?
I don't think this is built into FAKE, so if you want the 'sibling' build scripts to be used independently of the main script then your Shell.Exec approach is likely a very good one (and what I would use). That said ...
A slight variation on that approach would be to load the sibling .fsx files and then compose their build targets in the 'parent' build script:
#load "Sibling.fsx"
"LocalTarget"
==> "SiblingTarget"
You will get errors if the target names conflict, and it could be confusing, so a naming convention would be smart (eg "Build_Sibling_1", "Clean_Sibling_2", etc).