I'd like to define/manage a config file within a nix file and have it end up in the nix store, how can I do this?
Currently I'm managing this (outside of the nix store) manually by having the file within a path like ~/example/config.json, and used like the below:
systemd.services = {
example = {
description = "abcxyz";
serviceConfig = {
WorkingDirectory = "%h/example/";
Type = "simple";
ExecStart = "${example-pkg}/bin/app -c config.json";
Restart = "always";
RestartSec = 60;
};
wantedBy = [ "default.target" ];
};
For example Nixos defines a postgresql service that has a config file located at /nix/store/pcp1r7f8mylwvri381an01r64knj1wwb-postgresql.conf/postgresql.conf - how could I replicate this functionality in my own service above?
Taking a look at https://github.com/NixOS/nixpkgs/blob/226ff16bd6fbe6d47e714379eeba8598cd61a90a/nixos/modules/services/databases/postgresql.nix#L21
configFile = pkgs.writeTextDir "postgresql.conf" (concatStringsSep "\n" (mapAttrsToList (n: v: "${n} = ${toStr v}") cfg.settings));
It's not clear to me what configFile is? Will it be the path to the postgresql.conf file?
Either way pkgs.writeTextDir "config.ini" seems to create a directory instead, should I be using pkgs.writeText instead?
The above seems to work:
configFile = pkgs.writeText "config.json"
''
example config file bla bla
'';
And then you can use configFile via string interpolation:
ExecStart = "${example-pkg}/bin/app -c ${configFile}";
Related
I'm creating a rules file that generate some scripts with ctx.actions.expand_template and runs these scripts with ctx.actions.run.
ctx.actions.run uses the script file generated by ctx.actions.expand_template and the 'generated file type' file(filelist file contains several file name, path information) generated from other rule files which has a dependency relationship with this rule file as input attribute.
When the script is executed in ctx.actions.run, the generated file type filelist mentioned above is not found.
If I check the sandbox path where the actual build takes place, this filelist does not exist.
What should I do?
This is a part of my rule file
def _my_rule_impl(ctx):
...
my_script = ctx.actions.declare_file("my_script.sh")
ctx.actions.expand_template(
output = compile_script,
template = ctx.file._my_template,
substitutions = {
"{TOP}": "{}".format(top_name),
"{FLISTS}": " ".join(["-f {}".format(f.short_path) for f in flists_list]),
...
},
)
compile_srcs = flists_list + srcs_list + [my_script]
outputs = ctx.outputs.executable
executable = compile_script.path
ctx.actions.run(
inputs = depset(compile_srcs),
outputs = [outputs],
executable = executable,
env = {
"HOME": "/home/grrrr",
},
)
allfiles = depset(compile_srcs)
runfiles = ctx.runfiles(files = compile_srcs)
return [DefaultInfo(
files = allfiles,
runfiles = runfiles,
)]
my_rule = rule(
implementation = _my_rule_impl,
attrs = {
"deps": attr.label_list(
mandatory = True,
),
"_my_template": attr.label(
allow_single_file = True,
default = Label("#my_rules//my_test:my_script.sh.template"),
),
...
},
executable = True,
)
As a result of checking with print, this path is the location where the script is executed.
/home/grrrr/.cache/bazel/_bazel_grrrr/.../sandbox/processwrapper-sandbox/.../execroot/my_rules/
As a result of checking with print, the script refers to sources including a filelist in this path. However, there are only source file type files. There is not a filelist.
/home/grrrr/.cache/bazel/_bazel_grrrr/.../sandbox/processwrapper-sandbox/.../execroot/my_rules/my_test
However, There is a filelist in this path. I'm wondering why this filelist is not in above directory.
/home/grrrr/.cache/bazel/_bazel_grrrr/.../sandbox/processwrapper-sandbox/.../execroot/my_rules/bazel-out/k8-fastbuild/bin/my_test
It's resolved by using sandboxfs instead of sandbox.
Here is the useful page regarding sandboxfs.
https://bazel.build/docs/sandboxing#sandboxfs
I have some html files in "root/html_files/*.html" directory. I want to iterate on these html files and run some bazel rules on them, from "root/tests/" directory.
So, I made a filegroup of those html files and tried to get access to them in "root/tests/" directory, but that is't working.
I wonder if it is possible?
My BUILD file in "root/" directory:
HTMLS_LIST = glob(["html_files/*.html",])
filegroup(
name = "html_files",
srcs = HTMLS_LIST,
visibility = [ "//visibility:public" ],)
My BUILD file in "root/tests/" directory:
load("//tests:automation_test.bzl", "make_and_run_html_tests")
make_and_run_html_tests(
name = 'test_all_htmls',
srcs = ['test/automation_test.py'],
html_files = '//:html_files')
My bzl file in "root/tests/" directory:
def make_and_run_html_tests(name, html_files, srcs):
tests = []
for html_file in html_files: # I want to iterate on sources of html filegroup here
folders = html_file.split("/")
filename = folders[-1].split(".")[0]
test_rule_name = 'test_' + filename + '_file'
native.py_test(
name = test_rule_name,
srcs = srcs,
main = srcs[0],
data = [
html_file,
],
args = [html_file],
)
testname = ":" + test_rule_name
tests.append(testname)
native.test_suite(
name = name,
tests = tests,
)
And my python unittest file in "root/tests/" directory:
import sys
import codecs
import unittest
class TestHtmlDocumetns(unittest.TestCase):
def setUp(self):
self.html_file_path = sys.argv[1]
def test_html_file(self):
fid = codecs.open(self.html_file_path, 'r')
print(fid.read())
self.assertTrue(fid)
if __name__ == '__main__':
unittest.main(argv=[sys.argv[1]])
You can't access the references to the files inside another filegroup / rule from within a macro like that. You'd need to create a rule and access them via the ctx.files attr
However, you can iterate over them if you were to remove the filegroup, and pass the glob directly to the macro:
HTMLS_LIST = glob(["html_files/*.html"])
make_and_run_html_tests(
name = 'test_all_htmls',
srcs = ['test/automation_test.py'],
html_files = HTMLS_LIST
)
The glob is resolved to an array before expanding the macro
I'm experimenting with using libhttpserver and, by extension, libmicrohttpd from inside a Bazel build. The build process for these libraries seems to go:
./bootstrap
mkdir build
cd build
../configure
make
which is a slight variation, that I haven't seen before, on the more classic configure && make workflow.
Has anyone managed to make such a library work under Bazel?
Does anyone have a public example I can crib from?
The closest thing I've found to supporting this is #rules_foreign_cc//tools/build_defs:configure.bzl#configure_make but that seems to have no concept of the bootstrap step. Even hacking it doesn't seem to work as the bootstrap script ends up failing with:
mkdir: cannot create directory 'tmpwrk23': Read-only file system
autopoint: *** cannot create directory tmpwrk23
autopoint: *** Stop.
autoreconf: autopoint failed with exit status: 1
I'm about ready to just reach for a genrule() but that seems very error prone...
I went down the same path as you and was able to get libhttpserver compiling with bazel using the rules_foreign_cc project after coming across the 6/23/20 update on this blog post. I added code below, but in general rules_foreign_cc has a make rule, which you can set override make_commands and call ./bootstrap.
WORKSPACE:
...
http_archive(
name = "rules_foreign_cc",
strip_prefix = "rules_foreign_cc-master",
url = "https://github.com/bazelbuild/rules_foreign_cc/archive/master.zip",
)
load("#rules_foreign_cc//:workspace_definitions.bzl", "rules_foreign_cc_dependencies")
rules_foreign_cc_dependencies(register_default_tools = True)
all_content = """filegroup(name = "all", srcs = glob(["**"]), visibility = ["//visibility:public"])"""
http_archive(
name = "rules_cc",
urls = ["https://github.com/bazelbuild/rules_cc/archive/262ebec3c2296296526740db4aefce68c80de7fa.zip"],
strip_prefix = "rules_cc-262ebec3c2296296526740db4aefce68c80de7fa",
)
http_archive(
name = "libgnutls",
build_file_content = all_content,
strip_prefix = "gnutls-3.6.15",
urls = ["https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/gnutls-3.6.15.tar.xz"],
)
http_archive(
name = "libhttpserver",
build_file_content = all_content,
strip_prefix = "libhttpserver-master",
urls = ["https://github.com/etr/libhttpserver/archive/master.zip"],
)
http_archive(
name = "libmicrohttpd",
build_file_content = all_content,
strip_prefix = "libmicrohttpd-0.9.71",
urls = ["https://ftp.gnu.org/gnu/libmicrohttpd/libmicrohttpd-0.9.71.tar.gz"],
)
BUILD:
load("#rules_foreign_cc//tools/build_defs:configure.bzl", "configure_make")
load("#rules_foreign_cc//tools/build_defs:make.bzl", "make")
load("#rules_cc//cc:defs.bzl", "cc_binary", "cc_library")
package(default_visibility = ["//visibility:public"])
configure_make(
name = "libgnutls",
lib_source = "#libgnutls//:all",
configure_options = ["--with-included-unistring"],
out_include_dir = "include/gnutls",
shared_libraries = ["libgnutls.so"],
)
configure_make(
name = "libmicrohttpd",
lib_source = "#libmicrohttpd//:all",
deps = [":libgnutls"],
)
make(
name = "libhttpserver",
lib_source = "#libhttpserver//:all",
make_commands = [
"./bootstrap",
"mkdir build_dir",
"cd build_dir",
"../configure --prefix=${INSTALLDIR}",
"make",
"make install",
],
deps = [":libmicrohttpd", ":libgnutls"],
)
cc_binary(
name = "hello-world",
srcs = ["hello_world.cc"],
deps = [
":libhttpserver"
],
)
hello_world.cc (example on libhttpserver github page, run "curl -XGET -v http://localhost:8080/hello" to test)
#include <iostream>
#include <httpserver.hpp>
using namespace std;
using namespace httpserver;
class hello_world_resource : public http_resource {
public:
const std::shared_ptr<http_response> render(const http_request&) {
return std::shared_ptr<http_response>(new string_response("Hello, World!"));
}
};
int main(int argc, char** argv) {
cout << "hello!" << std::endl;
webserver web_server = create_webserver(8080);
hello_world_resource resource;
web_server.register_resource("/hello", &resource);
web_server.start(true);
return 0;
}
I'm working on my first custom Bazel rules. The rules allow the running of bats command line tests.
I've included the rule definition below verbatim. I'm pretty happy with it so far but there's one part which feels really ugly and non-standard. If the rule user adds a binary dependency to the rule then I make sure that the binary appears on the PATH so that it can be tested. At the moment I do this by making a list of the binary paths and then appending them with $PWD which is expanded inside the script to the complete execution path. This feels hacky and error prone.
Is there a more idiomatic way to do this? I don't believe I can access the execution path in the rule due to it not being created until the execution phase.
Thanks for your help!
BATS_REPOSITORY_BUILD_FILE = """
package(default_visibility = [ "//visibility:public" ])
sh_binary(
name = "bats",
srcs = ["libexec/bats"],
data = [
"libexec/bats-exec-suite",
"libexec/bats-exec-test",
"libexec/bats-format-tap-stream",
"libexec/bats-preprocess",
],
)
"""
def bats_repositories(version="v0.4.0"):
native.new_git_repository(
name = "bats",
remote = "https://github.com/sstephenson/bats",
tag = version,
build_file_content = BATS_REPOSITORY_BUILD_FILE
)
BASH_TEMPLATE = """
#!/usr/bin/env bash
set -e
export TMPDIR="$TEST_TMPDIR"
export PATH="{bats_bins_path}":$PATH
"{bats}" "{test_paths}"
"""
def _dirname(path):
prefix, _, _ = path.rpartition("/")
return prefix.rstrip("/")
def _bats_test_impl(ctx):
runfiles = ctx.runfiles(
files = ctx.files.srcs,
collect_data = True,
)
tests = [f.short_path for f in ctx.files.srcs]
path = ["$PWD/" + _dirname(b.short_path) for b in ctx.files.deps]
sep = ctx.configuration.host_path_separator
ctx.file_action(
output = ctx.outputs.executable,
executable = True,
content = BASH_TEMPLATE.format(
bats = ctx.executable._bats.short_path,
test_paths = " ".join(tests),
bats_bins_path = sep.join(path),
),
)
runfiles = runfiles.merge(ctx.attr._bats.default_runfiles)
return DefaultInfo(
runfiles = runfiles,
)
bats_test = rule(
attrs = {
"srcs": attr.label_list(
allow_files = True,
),
"deps": attr.label_list(),
"_bats": attr.label(
default = Label("#bats//:bats"),
executable = True,
cfg = "host",
),
},
test = True,
implementation = _bats_test_impl,
)
This should be easy to support from Bazel 0.8.0 which will be released in ~2 weeks.
In your skylark implementation you should do ctx.expand_location(binary) where binary should be something like $(execpath :some-label) so you might want to just format the label you got from the user with the $(execpath) and bazel will make sure to give you the execution location of that label.
Some relevant resources:
$location expansion in Bazel
https://github.com/bazelbuild/bazel/issues/2475
https://github.com/bazelbuild/bazel/commit/cff0dc94f6a8e16492adf54c88d0b26abe903d4c
What I figured currently is creating a AllTest and run it with junit. But, I am not satisfied with it. I want this rule can create as many tests as many java test file in created in codebase.
def junit_suite_test(name, srcs, deps, size="small", resources=[], classpath_resources=[], jvm_flags=[], tags=[], data=[]):
tests = []
package = PACKAGE_NAME.replace("src/test/java/", "").replace("/", ".")
for src in srcs:
if src.endswith("Test.java"):
if "/" in src:
src = package + "." + src.replace("/", ".")
tests += [src.replace(".java", ".class")]
native.genrule(
name = name + "-AllTests-gen",
outs = ["AllTests.java"],
cmd = """
cat <<EOF >> $#
package %s;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
#RunWith(Suite.class)
#Suite.SuiteClasses({%s})
public class AllTests {}
EOF
""" % (package, ",".join(tests))
)
native.java_test(
name = name,
srcs = srcs + ["AllTests.java"],
test_class = package + ".AllTests",
resources = resources,
classpath_resources = classpath_resources,
data = data,
size = size,
tags = tags,
jvm_flags = jvm_flags,
deps = deps + [
],
)
Hi you can do something like that:
[java_test(name = s[:-5], srcs = s) for s in glob(["*.java"])]
That will create on test target per java file.
With that method, your macro would looks like:
def junit_suite_test(name, srcs, deps, size="small", resources=[], classpath_resources=[], jvm_flags=[], tags=[], data=[]):
[native.java_test(
name = name,
srcs = src,
resources = resources,
classpath_resources = classpath_resources,
data = data,
size = size,
tags = tags,
jvm_flags = jvm_flags,
deps = deps,
) for src in srcs if src.endswith("Test.java")]
Of course you probably needs some adaptation to feed in the good sources.
However, I would recommend against doing that over your solution as too much parallelisms can actually be slower in fine. The test log and the XML file will report the actual failing test case and you can use shard_count to increase parallelism is really needed.