I want to write Google ZX scripts in an idiomatic way in Nix, but how can I do that? nixpkgs only offers writeShellScriptBin which is tied to bash as interpreter. How can I change the interpreter?
To solve it idiomatically, you could create a writeZxScriptBin attribute similar to writeShellScriptBin. Luckily, zx is already packaged in <nixpkgs>. Hence, we can write a zx-script-writer-bin.nix like this:
# This returns a writer that creates an executable Google ZX script in `<pkg>/bin/%name`
# with the provided content. Similar to `writeShellScriptBin`, it consumes the name of
# the script and the NodeJS/ZX-script content as parameters.
{ pkgs }:
name: text:
let
mjsBin = pkgs.writeTextFile {
name = "${name}-mjs";
executable = true;
# For Google ZX, the .mjs extension is mandatory.
destination = "/bin/${name}.mjs";
text = ''
#!${pkgs.zx}/bin/zx
${text}
'';
};
in
# Shebang script enables to call the zx-script without the .mjs version.
pkgs.writeTextFile {
name = "${name}";
executable = true;
destination = "/bin/${name}";
text = ''
#!${mjsBin}/bin/${name}.mjs
'';
}
We can invoke it like this:
$ nix-build -E "
let
pkgs = import <nixpkgs> {};
writeZxScriptBin = pkgs.callPackage ./write-zx-script-bin.nix {};
in
writeZxScriptBin \"foo\" ''
const a = 7
console.log(\`hello! sum=''\${3 + a}\`)
''" && result/bin/foo
Note that there is some escaping required. Alternatively, you can work with builtins.readFile to write the script code in pure JavaScript
Related
I want to parse a file to a Nix list value inside flake.nix.
I have a shell script which does that
perl -007 -nE 'say for m{[(]use-package \s* ([a-z-0-9]+) \s* (?!:nodep)}xsgm' init.el
How can I execute external command while evaluating flake.nix?
programs.emacs = {
enable = true;
extraConfig = builtins.readFile ./init.el;
extraPackages = elpa: (shellCommandToParseFile ./init.el); # Runs shell script
};
You can run ./init.el by the same way you perform any other impure step in Nix: With a derivation.
This might look something vaguely like:
programs.emacs = {
enable = true;
extraConfig = ../init.el;
extraPackages = elpa:
let
packageListNix =
pkgs.runCommand "init-packages.nix" { input = ../init.el; } ''
${pkgs.perl}/bin/perl -007 -nE '
BEGIN {
say "{elpa, ...}: with elpa; [";
say "use-package";
};
END { say "]" };
while (m{[(]use-package \s* ([a-z-0-9]+) \s* (;\S+)?}xsgm) {
next if $2 eq ";builtin";
say $1;
}' "$input" >"$out"
'';
in (import "${packageListNix}" { inherit elpa; });
};
...assuming that, given the contents of your ./init.el, the contents of your resulting el-pkgs.nix is actually valid nix source code.
That said, note that like any other derivation (that isn't either fixed-output or explicitly impure), this happens inside a sandbox with no network access. If the goal of init.el is to connect to a network resource, you should be committing its output to your repository. A major design goal of flakes is to remove impurities; they're not suitable for impure derivations.
NB: add --extra-experimental-features nix-command --extra-experimental-features flakes if you've allow experimental feature in nix
This repository propose to load a shell this way:
nix develop github:informalsystems/cosmos.nix#cosmos-shell
It seems to work.
In order to see if I've really understood how nix flake works (I haven't) I am trying to write a flake.nix so that I only have to write
nix develop
There is a field devShells in output in flake.nix in this repo. Warning devShells not devShell. This is a collection of shells defined in configuration.nix
my flake.nix :
{
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; #not useful for this question
cosmos_inform_system.url = github:informalsystems/cosmos.nix;
flake-utils.url = github:numtide/flake-utils;
};
outputs = { self, nixpkgs, cosmos_inform_system, flake-utils, ... }:
flake-utils.lib.eachDefaultSystem (system:
let
shells = cosmos_inform_system.devShells;
in {
devShell.default = shells.cosmos-shell;
});
}
warning: flake output attribute 'devShell' is deprecated; use 'devShells..default' instead
error: attribute 'cosmos-shell' missing
There is cosmos-shell in configuration.nix this a field devShells. Therefore I don't understand the error
In order to remove the warning, I replace this line
devShell = shells.cosmos-shell;
by this line
devShells.${system}.default = shells.cosmos-shell;
error: flake attribute 'devShell.aarch64-linux' is not a derivation
I still have the warning.
Take a look at this flake.nix file for an example of how to do that.
nix flake check
works with that
{
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
cosmos_inform_system.url = github:informalsystems/cosmos.nix;
flake-utils.url = github:numtide/flake-utils;
};
outputs = { self, nixpkgs, cosmos_inform_system, flake-utils, ... }:
flake-utils.lib.eachDefaultSystem (system:
let
shells = cosmos_inform_system.devShells;
in {
devShells.default = shells.${system}.cosmos-shell;
}
);
}
but
nix develop
error: hash mismatch in fixed-output derivation '/nix/store/yz9kjwjhszzpp7g4wnbxj43zwga1zzsy-ica-go-modules.drv':
specified: sha256-ykGo5TQ+MiFoeQoglflQL3x3VN2CQuyZCIiweP/c9lM=
got: sha256-D31e+G/7KAmF3Gkk4IOmU2g/eLlqkrkpwJa7CEjdaAk=
[1/158 built (1 failed)] building ica-go-modules (installPhase): installing
Nevertheless I have exactly the same problem with nix develop github:informalsystems/cosmos.nix#cosmos-shell
I still think my flake file is the same as nix develop github:informalsystems/cosmos.nix#cosmos-shell
I have a long string which I want to extract to a separate file.
prepare = lib.hm.dag.entryAfter ["writeBoundary"] '' very long script with ${...} '';
I can read it with builtins.readFile but it does not replace ${...} nix place holders.
How can I read string from a file and resolve nix variables inside it?
You could use the substitute family of bash functions (see Nixpkgs manual) or use the substituteAll Nix function which produces a derivation that performs the substitution.
substituteAll { src = ./sample.sh; inherit bash; hi = hello; }
sample.sh:
#!#bash#/bin/bash
#hi#
Result:
$ nix repl <nixpkgs>
nix-repl> substituteAll { src = ./sample.sh; inherit bash; hi = hello; }
«derivation /nix/store/7klv763a0ipgvwf3j84aazkzx2d5rljz-sample.sh.drv»
nix-repl> :b substituteAll { src = ./sample.sh; inherit bash; hi = hello; }
this derivation produced the following outputs:
out -> /nix/store/v03hpzw9ykrbyqpmalnijnyibq3waqhw-sample.sh
nix-repl>
/nix/store/v03hpzw9ykrbyqpmalnijnyibq3waqhw-sample.sh:
#!/nix/store/a4yw1svqqk4d8lhwinn9xp847zz9gfma-bash-4.4-p23/bin/bash
/nix/store/jmmw0d3nmklwafcwylvrjb9v69wrbcxf-hello-2.10
Is there a simple way to add a bash function to the environment provided by stdenv? When developing with 'nix-shell', I can run commands like 'unpackPhase' or 'buildPhase' because mkDerivation puts them in scope--its super useful. My derivation attributes are added to the environment as well. But I'd also like to see a way to automatically add an attribute as a function in the build/shell environment.
To explain what I'm getting at, I can currently get functions into the environment by either using eval statements or toFile and source. For instance, with eval, something like:
{ stdenv, ... } : stdenv.mkDerivation
{ shellHook = ''eval "$myFunctions"'';
myFunctions = ''
myFunction1(){
echo "doing myFunction1"
}
myFunction2(){
echo "doing myFunction2"
}
''
}
will "source" myFunction1 and myFunction2 when I enter the nix-shell.
However, I was expecting something like a mkFunction utility to compliment mkWrapper, mkProgram, and other such basic utilities. I'd expect to use it like in the below, where the the attributes defined using mkFunction are automatically "sourced" as above.
{ stdenv, mkFunction, ... } : stdenv.mkDerivation
{ myFunction1 = mkFunction '' echo "doing myFunction1" '';
myFunction2 = mkFunction '' echo "doing myFunction2" '';
shellHook = '' echo "No need to source, myFunctions are already in scope." '';
}
I think such a utility would be useful. I thought setup hooks might cover this use, but I'm not really sure how to use them. And its not so bad doing it the first way (If I hadn't figured it out during the course of writing the question I wouldn't have bothered asking). But it seems like the kind of utility that nix would already have available, so I'm asking anyway.
Setup hooks can indeed do that, here's a simple example of one that defines a function foo:
with import <nixpkgs> {};
let
fooHook = stdenv.mkDerivation {
name = "foo-hook";
# Setting phases directly is usually discouraged, but in this case we really
# only need fixupPhase because that's what installs setup hooks
phases = [ "fixupPhase" ];
setupHook = writeText "my-setup-hook" ''
foo() { echo "Foo was called!"; }
'';
};
in mkShell {
buildInputs = [ fooHook ];
shellHook = "foo";
}
Running nix-shell on this yields the desired result:
$ nix-shell
Foo was called!
[nix-shell:~]$
Another simple possibility is to use the runHook function provided by stdenv.
with import <nixpkgs> {};
mkShell rec {
name = "runs-hook";
myFun = "echo The name is ${name}";
}
In the snippet above, a variable myFun is made available in the shell/build environment. You call it with:
> runHook myFun
The name is runs-hook
This is really similar to the eval method from the question.
The bazel build flag --workspace_status_command supports calling a script to retrieve e.g. repository metadata, this is also known as build stamping and available in rules like java_binary.
I'd like to create a custom rule using this metadata.
I want to use this for a common support function. It should receive the git version and some other attributes and create a version.go output file usable as a dependency.
So I started a journey looking at rules in various bazel repositories.
Rules like rules_docker support stamping with stamp in container_image and let you reference the status output in attributes.
rules_go supports it in the x_defs attribute of go_binary.
This would be ideal for my purpose and I dug in...
It looks like I can get what I want with ctx.actions.expand_template using the entries in ctx.info_file or ctx.version_file as a dictionary for substitutions. But I didn't figure out how to get a dictionary of those files. And those two files seem to be "unofficial", they are not part of the ctx documentation.
Building on what I found out already: How do I get a dict based on the status command output?
If that's not possible, what is the shortest/simplest way to access workspace_status_command output from custom rules?
I've been exactly where you are and I ended up following the path you've started exploring. I generate a JSON description that also includes information collected from git to package with the result and I ended up doing something like this:
def _build_mft_impl(ctx):
args = ctx.actions.args()
args.add('-f')
args.add(ctx.info_file)
args.add('-i')
args.add(ctx.files.src)
args.add('-o')
args.add(ctx.outputs.out)
ctx.actions.run(
outputs = [ctx.outputs.out],
inputs = ctx.files.src + [ctx.info_file],
arguments = [args],
progress_message = "Generating manifest: " + ctx.label.name,
executable = ctx.executable._expand_template,
)
def _get_mft_outputs(src):
return {"out": src.name[:-len(".tmpl")]}
build_manifest = rule(
implementation = _build_mft_impl,
attrs = {
"src": attr.label(mandatory=True,
allow_single_file=[".json.tmpl", ".json_tmpl"]),
"_expand_template": attr.label(default=Label("//:expand_template"),
executable=True,
cfg="host"),
},
outputs = _get_mft_outputs,
)
//:expand_template is a label in my case pointing to a py_binary performing the transformation itself. I'd be happy to learn about a better (more native, fewer hops) way of doing this, but (for now) I went with: it works. Few comments on the approach and your concerns:
AFAIK you cannot read in (the file and perform operations in Skylark) itself...
...speaking of which, it's probably not a bad thing to keep the transformation (tool) and build description (bazel) separate anyways.
It could be debated what constitutes the official documentation, but ctx.info_file may not appear in the reference manual, it is documented in the source tree. :) Which is case for other areas as well (and I hope that is not because those interfaces are considered not committed too yet).
For sake of comleteness in src/main/java/com/google/devtools/build/lib/skylarkbuildapi/SkylarkRuleContextApi.java there is:
#SkylarkCallable(
name = "info_file",
structField = true,
documented = false,
doc =
"Returns the file that is used to hold the non-volatile workspace status for the "
+ "current build request."
)
public FileApi getStableWorkspaceStatus() throws InterruptedException, EvalException;
EDIT: few extra details as asked in the comment.
In my workspace_status.sh I would have for instance the following line:
echo STABLE_GIT_REF $(git log -1 --pretty=format:%H)
In my .json.tmpl file I would then have:
"ref": "${STABLE_GIT_REF}",
I've opted for shell like notation of text to be replaced, since it's intuitive for many users as well as easy to match.
As for the replacement, relevant (CLI kept out of this) portion of the actual code would be:
def get_map(val_file):
"""
Return dictionary of key/value pairs from ``val_file`.
"""
value_map = {}
for line in val_file:
(key, value) = line.split(' ', 1)
value_map.update(((key, value.rstrip('\n')),))
return value_map
def expand_template(val_file, in_file, out_file):
"""
Read each line from ``in_file`` and write it to ``out_file`` replacing all
${KEY} references with values from ``val_file``.
"""
def _substitue_variable(mobj):
return value_map[mobj.group('var')]
re_pat = re.compile(r'\${(?P<var>[^} ]+)}')
value_map = get_map(val_file)
for line in in_file:
out_file.write(re_pat.subn(_substitue_variable, line)[0])
EDIT2: This is how the Python script is how I expose the python script to rest of bazel.
py_binary(
name = "expand_template",
main = "expand_template.py",
srcs = ["expand_template.py"],
visibility = ["//visibility:public"],
)
Building on Ondrej's answer, I now use somthing like this (adapted in SO editor, might contain small errors):
tools/bazel.rc:
build --workspace_status_command=tools/workspace_status.sh
tools/workspace_status.sh:
echo STABLE_GIT_REV $(git rev-parse HEAD)
version.bzl:
_VERSION_TEMPLATE_SH = """
set -e -u -o pipefail
while read line; do
export "${line% *}"="${line#* }"
done <"$INFILE" \
&& cat <<EOF >"$OUTFILE"
{ "ref": "${STABLE_GIT_REF}"
, "service": "${SERVICE_NAME}"
}
EOF
"""
def _commit_info_impl(ctx):
ctx.actions.run_shell(
outputs = [ctx.outputs.outfile],
inputs = [ctx.info_file],
progress_message = "Generating version file: " + ctx.label.name,
command = _VERSION_TEMPLATE_SH,
env = {
'INFILE': ctx.info_file.path,
'OUTFILE': ctx.outputs.version_go.path,
'SERVICE_NAME': ctx.attr.service,
},
)
commit_info = rule(
implementation = _commit_info_impl,
attrs = {
'service': attr.string(
mandatory = True,
doc = 'name of versioned service',
),
},
outputs = {
'outfile': 'manifest.json',
},
)