As documented here, there are two flags for specifying a cache directory where Bazel can check before downloading a file from the net: --repository_cache and --distdir
What's the difference between them?
Both the repository cache and the distribution files directory have more extensive prose documentation in the user guide. The short version is the repository cache is a content-addressable cache used internally by Bazel. The distribution files directory is keyed by archive file name and expected to be completely user-provided.
Related
I am trying to run bazel from a company-installed location. It decides to use my home directory for some caching and extracting installation files. There is no much space (company policy). So, the question is: how to redirect it to a different location? I could not find much on the web.
WARNING: Output base '/home/name/.cache/bazel/_bazel_name/b05102fc551c33214ce89fb43ea90837' is on NFS. This may lead to surprising failures and undetermined behavior.
Extracting Bazel installation...
FATAL: Failed to extract embedded binaries: Failed to write zipped file '/home/name/.cache/bazel/_bazel_name/install/f6ca571514ebc1be8327564b4455aae2.tmp.10086/_embedded_binaries/A-server.jar': (error: 122): Disk quota exceeded
Also, not sure why it thinks that this is NFS. I try to run it onlinux.
found it: bazel --output_user_root=<loc> fixes the issue.
I am attempting to use Bazel to compile a dhall program based on dhall-kubernetes to generate a Kubernetes YAML file.
The basic dhall compile without dhall-kubernetes using a simple bazel macro works ok.
I have made an example using dhall's dependency resolution to download dhall-kubernetes - see here. This also works but is very slow (I think because dhall downloads each remote file separately), and introduces a network dependency to the bazel rule execution, which I would prefer to avoid.
My preferred approach is to use Bazel to download an archive release version of dhall-kubernetes, then have the rule access it locally (see here). My solution requires a relative path in Prelude.dhall and package.dhall for the examples/k8s package to reference dhall-kubernetes. While it works, I am concerned that this is subverting the Bazel sandbox by requiring special knowledge of the folder structure used internally by Bazel. Is there a better way?
Prelude.dhall:
../../external/dhall-kubernetes/1.17/Prelude.dhall
WORKSPACE:
workspace(name = "dhall")
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
DHALL_KUBERNETES_VERSION = "4.0.0"
http_archive(
name = "dhall-kubernetes",
sha256 = "0bc2b5d2735ca60ae26d388640a4790bd945abf326da52f7f28a66159e56220d",
url = "https://github.com/dhall-lang/dhall-kubernetes/archive/v%s.zip" % DHALL_KUBERNETES_VERSION,
strip_prefix = "dhall-kubernetes-4.0.0",
build_file = "#//:BUILD.dhall-kubernetes",
)
BUILD.dhall-kubernetes:
package(default_visibility=['//visibility:public'])
filegroup(
name = "dhall-k8s-1.17",
srcs = glob([
"1.17/**/*",
]),
)
examples/k8s/BUILD:
package(default_visibility = ["//visibility:public"])
genrule(
name = "special_ingress",
srcs = ["ingress.dhall",
"Prelude.dhall",
"package.dhall",
"#dhall-kubernetes//:dhall-k8s-1.17"
],
outs = ["ingress.yaml"],
cmd = "dhall-to-yaml --file $(location ingress.dhall) > $#",
visibility = [
"//visibility:public"
]
)
There is a way to instrument dhall to do "offline" builds, meaning that the package manager fetches all Dhall dependencies instead of Dhall fetching them.
In fact, I implemented something exactly this for Nixpkgs, which you may be able to translate to Bazel:
Add Nixpkgs support for Dhall
High-level explanation
The basic trick is to take advantage of a feature of Dhall's import system, which is that if a package protected by a semantic integrity check (i.e. a "semantic hash") is cached then Dhall will use the cache instead of fetching the package. You can build upon this trick to have the package manager bypass Dhall's remote imports by injecting dependencies in this way.
You can find the Nix-related logic for this here:
Nix function for building a Dhall package
... but I will try to explain how it works in a package-manager-independent way.
Package structure
First, the final product of a Dhall "package" built using Nix is a directory with the following structure:
$ nix-build --attr 'dhallPackages.Prelude'
…
$ tree -a ./result
./result
├── .cache
│ └── dhall
│ └── 122026b0ef498663d269e4dc6a82b0ee289ec565d683ef4c00d0ebdd25333a5a3c98
└── binary.dhall
2 directories, 2 files
The contents of this directory are:
./cache/dhall/1220XXX…XXX
A valid cache directory for Dhall containing a single build product: the binary encoding of the interpreted Dhall expression.
You can create such a binary file using dhall encode and you can compute the file name by replacing the XXX…XXX above with the sha256 encoding of the expression, which you can obtain using the dhall hash command.
./binary.dhall
A convenient Dhall file containing the expression missing sha256:XXX…XXX. Interpreting this expression only succeeds if the expression we built matching the hash sha256:XXX…XXX is already cached.
The file is called binary.dhall because this is the Dhall equivalent of a "binary" package distribution, meaning that the import can only be obtained from a binary cache and cannot be fetched and interpreted from source.
Optional: ./source.dhall
This is a file containing a fully αβ-normalized expression equivalent to the expression that was cached. By default, this should be omitted for all packages except perhaps the top-level package, since it contains the same expression that is stored inside of ./cache/1220XXX…XXX, albeit less efficiently (since the binary encoding is more compact)
This file is called ./source.dhall because this is the Dhall equivalent of a "source" package distribution, which contains valid source code to produce the same result.
User interface
The function for building a package takes four arguments:
The package name
This is not material to the build. It's just to name things since every Nix package has to have a human-readable name.
The dependencies for the build
Each of these dependencies is a build product that produces a directory tree just like the one I described above (i.e. a ./cache directory, a ./binary.dhall file, and an optional ./source.dhall file)
A Dhall expression
This is can be arbitrary Dhall source code, with only one caveat: all remote imports transitively referenced by the expression must be protected by integrity checks AND those imports must match one of the dependencies to this Dhall package (so that the import can be satisfied via the cache instead of the Dhall runtime fetching the URL)
A boolean option specifying whether to keep the ./source.dhall file, which is False by default
Implementation
The way that the Dhall package builder works is:
First, build the Haskell Dhall package with the -f-with-http flag
This flag compiles out support for HTTP remote imports, that way if the user forgets to supply a dependency for a remote import they will get an error message saying Import resolution is disabled
We'll be using this executable for all of the subsequent steps
Create a cache directory within the current working directory named .cache/dhall
... and populate the cache directory with the binary files stored inside each dependency's ./cache/ directory
Configure the interpreter to use the cache directory we created
... by setting XDG_CACHE_HOME to point to the .cache directory we just created in our current working directory
Interpret and α-normalize the Dhall source code for our package
... using the dhall --alpha command. Save the result to $out/source.dhall where $out is the directory that will store the final build product
Obtain the expression's hash
... using the dhall hash command. We will need this hash for the following two steps.
Create the corresponding binary cache file
... using the dhall encode command and save the file to $out/cache/dhall/1220${HASH}
Create the ./binary.dhall file
... by just writing out a text file to $out/binary.dhall containing missing sha256:${HASH}
Optional: Delete the ./source.dhall file
... if the user did not request to keep the file. Omitting this file by default helps conserve space within the package store by not storing the same expression twice (as both a binary file and source code).
Packaging conventions
Once you have this function, there are a couple of conventions that can help simplify doing things "in the large"
By default, a package should build a project's ./package.dhall file
Make it easy to override the package version
Make it easy to override the file built within the package
In other words, if a user prefers to import individual files like https://prelude.dhall-lang.org/List/map instead of the top-level ./package.dhall file there should be a way for them to specify a dependency like Prelude.override { file = "./List/map"; } to obtain a package that builds and caches that individual file.
Conclusion
I hope that helps! If you have more questions about how to do this you can either ask them here or you can also discuss more on our Discourse forum, especially on the thread where this idiom first originated:
Dhall Discourse - Offline use of Prelude
My command is .\azcopy cp "source" "dest" --recursive=true
Where both source and dest are storage containers.
When I run the cp command, it seems like azcopy iterates over every file and transfers to destination.
Is there way to only copy the file if the file does not exist or is different in the destination?
azcopy sync does something similar, but only supports dest/origin of local/container and not container/container, as is my understanding.
We've just added container to container support in version 10.3
If you want to stick with AzCopy v10, it looks like there is an --overwrite parameter which you can set to true (default), false, or prompt. By setting to false, it won't overwrite any files that already exist. However, it also won't overwrite any files which are newer in the source -- not sure if that is a deal-breaker for you.
Your understanding is right, currently, the azcopy sync supports only between the local file system to the blob storage container, not container/container. Since Synchronization is one-way. As a workaround, you could perform the synchronous process in two steps by syncing from the specified blob storage source to the local file path and then syncing them to the Blob storage destination from the local file path.
Another option is to use AzCopy v8.1. The /XO and /XN parameters allow you to exclude older or newer source resources from being copied, respectively. If you only want to copy source resources that don't exist in the destination, you can specify both parameters in the AzCopy command:
/Source:http://myaccount.blob.core.windows.net/mycontainer /Dest:http://myaccount.blob.core.windows.net/mycontainer1 /SourceKey:<sourcekey> /DestKey:<destkey> /S /XO /XN
(Background: In a docker container on a NVidia Jetson TX2 board I have decompressed NVidia's Linux For Tegra tarball which contains lots of drivers and shared object files, some of which provide GStreamer element factories which produce elements that I use in my GStreamer pipeline. I am trying to run the pipeline in the docker container.)
However, there is an element in my GStreamer pipeline (on this Tegra board), called 'omxh264enc', which I haven't been able to create.
I've put the corresponding 'libnvomx.so' in my drivers folder which is in the exported paths GST_PLUGIN_PATH and the LD_LIBRARY_PATH.
ldd -r does not show any missing libraries for libnvomx.so
HOWEVER when I try and run the pipeline, output includes
WARN omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /root/.config:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)
INFO omx gstomx.c:2831:plugin_init: Using default configuration
ERROR omx gstomx.c:2894:plugin_init: Core '/usr/lib/aarch64-linux-gnu/tegra/libnvomx.so' does not exist for element 'omxh264enc'
WARN GST_PLUGIN_LOADING gstplugin.c:526:gst_plugin_register_func: plugin "/gst_1.8.3/libs/gstreamer-1.0/libnvomx.so" failed to initialise
and when I use GST_DEBUG=3 gst-inspect-1.0 libnvomx.so, libnvomx.so is blacklisted.
Plugin Details:
Name libnvomx.so
Description Plugin for blacklisted file
Filename /gst_1.8.3/libs/gstreamer-1.0/libnvomx.so
Version 0.0.0
License BLACKLIST
Source module BLACKLIST
Binary package BLACKLIST
Origin URL BLACKLIST
I have copied libnvomx.so into /usr/lib/aarch64-linux-gnu/tegra but this did not make a difference (probably because libnvomx is blacklisted).
I don't know where to find the gstomx.conf file, where maybe I can change the path /usr/lib/aarch64-linux-gnu/tegra/libnvomx.so to my designated drivers folder (/gst_1.8.3/libs/gstreamer-1.0/). I have used 'find' on /etc and some other folders but didn't find it (I didn't actually find a .config folder on the system).
(There are also other plugins with missing symbols, nvidia_drv.so with undefined symbol TimerSet and libglx.so with undefined symbol serverClient. I would like to find out what is suppose to provide these symbols? But these are not (direct) dependencies of libnvomx.so)
So how can I initialise / unblacklist libnvomx.so so I can use 'omxh264enc'? Do I need to find / make a gstomx configuration file or can I make it work with the default configuration? I read somewhere there may be a solution using a 'symlink' but at the moment I'm not familiar with what these are or how these work.
Let me know if you need more info, thanks.
I don't understand what is the need/use of the git unpack-objects command.
If I have a pack file outside of my repository and run the git unpack-objects on it the pack file will be "decompressed" and all the object files will be placed in .git/objects. But what is the need for this? If I just place the pack and index files in .git/objects I can still see all the commits and have a functional repo and have less space occupied by my .git since the pack file is compact.
So why would anyone need to run this command?
Pack file uses the format that is used with normal transfer over the network. So, I can think of two main reasons to use the manual command instead of network:
having a similar update workflow in an environment without network configuration between the machines or where that cannot be used for other reasons
debugging/inspecting the contents of the transfer
For 1), you could just use a disk or any kind of mobile media for your files. It could be encrypted, for instance.