I want to parse a file to a Nix list value inside flake.nix.
I have a shell script which does that
perl -007 -nE 'say for m{[(]use-package \s* ([a-z-0-9]+) \s* (?!:nodep)}xsgm' init.el
How can I execute external command while evaluating flake.nix?
programs.emacs = {
enable = true;
extraConfig = builtins.readFile ./init.el;
extraPackages = elpa: (shellCommandToParseFile ./init.el); # Runs shell script
};
You can run ./init.el by the same way you perform any other impure step in Nix: With a derivation.
This might look something vaguely like:
programs.emacs = {
enable = true;
extraConfig = ../init.el;
extraPackages = elpa:
let
packageListNix =
pkgs.runCommand "init-packages.nix" { input = ../init.el; } ''
${pkgs.perl}/bin/perl -007 -nE '
BEGIN {
say "{elpa, ...}: with elpa; [";
say "use-package";
};
END { say "]" };
while (m{[(]use-package \s* ([a-z-0-9]+) \s* (;\S+)?}xsgm) {
next if $2 eq ";builtin";
say $1;
}' "$input" >"$out"
'';
in (import "${packageListNix}" { inherit elpa; });
};
...assuming that, given the contents of your ./init.el, the contents of your resulting el-pkgs.nix is actually valid nix source code.
That said, note that like any other derivation (that isn't either fixed-output or explicitly impure), this happens inside a sandbox with no network access. If the goal of init.el is to connect to a network resource, you should be committing its output to your repository. A major design goal of flakes is to remove impurities; they're not suitable for impure derivations.
Related
I want to write Google ZX scripts in an idiomatic way in Nix, but how can I do that? nixpkgs only offers writeShellScriptBin which is tied to bash as interpreter. How can I change the interpreter?
To solve it idiomatically, you could create a writeZxScriptBin attribute similar to writeShellScriptBin. Luckily, zx is already packaged in <nixpkgs>. Hence, we can write a zx-script-writer-bin.nix like this:
# This returns a writer that creates an executable Google ZX script in `<pkg>/bin/%name`
# with the provided content. Similar to `writeShellScriptBin`, it consumes the name of
# the script and the NodeJS/ZX-script content as parameters.
{ pkgs }:
name: text:
let
mjsBin = pkgs.writeTextFile {
name = "${name}-mjs";
executable = true;
# For Google ZX, the .mjs extension is mandatory.
destination = "/bin/${name}.mjs";
text = ''
#!${pkgs.zx}/bin/zx
${text}
'';
};
in
# Shebang script enables to call the zx-script without the .mjs version.
pkgs.writeTextFile {
name = "${name}";
executable = true;
destination = "/bin/${name}";
text = ''
#!${mjsBin}/bin/${name}.mjs
'';
}
We can invoke it like this:
$ nix-build -E "
let
pkgs = import <nixpkgs> {};
writeZxScriptBin = pkgs.callPackage ./write-zx-script-bin.nix {};
in
writeZxScriptBin \"foo\" ''
const a = 7
console.log(\`hello! sum=''\${3 + a}\`)
''" && result/bin/foo
Note that there is some escaping required. Alternatively, you can work with builtins.readFile to write the script code in pure JavaScript
Is there a simple way to add a bash function to the environment provided by stdenv? When developing with 'nix-shell', I can run commands like 'unpackPhase' or 'buildPhase' because mkDerivation puts them in scope--its super useful. My derivation attributes are added to the environment as well. But I'd also like to see a way to automatically add an attribute as a function in the build/shell environment.
To explain what I'm getting at, I can currently get functions into the environment by either using eval statements or toFile and source. For instance, with eval, something like:
{ stdenv, ... } : stdenv.mkDerivation
{ shellHook = ''eval "$myFunctions"'';
myFunctions = ''
myFunction1(){
echo "doing myFunction1"
}
myFunction2(){
echo "doing myFunction2"
}
''
}
will "source" myFunction1 and myFunction2 when I enter the nix-shell.
However, I was expecting something like a mkFunction utility to compliment mkWrapper, mkProgram, and other such basic utilities. I'd expect to use it like in the below, where the the attributes defined using mkFunction are automatically "sourced" as above.
{ stdenv, mkFunction, ... } : stdenv.mkDerivation
{ myFunction1 = mkFunction '' echo "doing myFunction1" '';
myFunction2 = mkFunction '' echo "doing myFunction2" '';
shellHook = '' echo "No need to source, myFunctions are already in scope." '';
}
I think such a utility would be useful. I thought setup hooks might cover this use, but I'm not really sure how to use them. And its not so bad doing it the first way (If I hadn't figured it out during the course of writing the question I wouldn't have bothered asking). But it seems like the kind of utility that nix would already have available, so I'm asking anyway.
Setup hooks can indeed do that, here's a simple example of one that defines a function foo:
with import <nixpkgs> {};
let
fooHook = stdenv.mkDerivation {
name = "foo-hook";
# Setting phases directly is usually discouraged, but in this case we really
# only need fixupPhase because that's what installs setup hooks
phases = [ "fixupPhase" ];
setupHook = writeText "my-setup-hook" ''
foo() { echo "Foo was called!"; }
'';
};
in mkShell {
buildInputs = [ fooHook ];
shellHook = "foo";
}
Running nix-shell on this yields the desired result:
$ nix-shell
Foo was called!
[nix-shell:~]$
Another simple possibility is to use the runHook function provided by stdenv.
with import <nixpkgs> {};
mkShell rec {
name = "runs-hook";
myFun = "echo The name is ${name}";
}
In the snippet above, a variable myFun is made available in the shell/build environment. You call it with:
> runHook myFun
The name is runs-hook
This is really similar to the eval method from the question.
The bazel build flag --workspace_status_command supports calling a script to retrieve e.g. repository metadata, this is also known as build stamping and available in rules like java_binary.
I'd like to create a custom rule using this metadata.
I want to use this for a common support function. It should receive the git version and some other attributes and create a version.go output file usable as a dependency.
So I started a journey looking at rules in various bazel repositories.
Rules like rules_docker support stamping with stamp in container_image and let you reference the status output in attributes.
rules_go supports it in the x_defs attribute of go_binary.
This would be ideal for my purpose and I dug in...
It looks like I can get what I want with ctx.actions.expand_template using the entries in ctx.info_file or ctx.version_file as a dictionary for substitutions. But I didn't figure out how to get a dictionary of those files. And those two files seem to be "unofficial", they are not part of the ctx documentation.
Building on what I found out already: How do I get a dict based on the status command output?
If that's not possible, what is the shortest/simplest way to access workspace_status_command output from custom rules?
I've been exactly where you are and I ended up following the path you've started exploring. I generate a JSON description that also includes information collected from git to package with the result and I ended up doing something like this:
def _build_mft_impl(ctx):
args = ctx.actions.args()
args.add('-f')
args.add(ctx.info_file)
args.add('-i')
args.add(ctx.files.src)
args.add('-o')
args.add(ctx.outputs.out)
ctx.actions.run(
outputs = [ctx.outputs.out],
inputs = ctx.files.src + [ctx.info_file],
arguments = [args],
progress_message = "Generating manifest: " + ctx.label.name,
executable = ctx.executable._expand_template,
)
def _get_mft_outputs(src):
return {"out": src.name[:-len(".tmpl")]}
build_manifest = rule(
implementation = _build_mft_impl,
attrs = {
"src": attr.label(mandatory=True,
allow_single_file=[".json.tmpl", ".json_tmpl"]),
"_expand_template": attr.label(default=Label("//:expand_template"),
executable=True,
cfg="host"),
},
outputs = _get_mft_outputs,
)
//:expand_template is a label in my case pointing to a py_binary performing the transformation itself. I'd be happy to learn about a better (more native, fewer hops) way of doing this, but (for now) I went with: it works. Few comments on the approach and your concerns:
AFAIK you cannot read in (the file and perform operations in Skylark) itself...
...speaking of which, it's probably not a bad thing to keep the transformation (tool) and build description (bazel) separate anyways.
It could be debated what constitutes the official documentation, but ctx.info_file may not appear in the reference manual, it is documented in the source tree. :) Which is case for other areas as well (and I hope that is not because those interfaces are considered not committed too yet).
For sake of comleteness in src/main/java/com/google/devtools/build/lib/skylarkbuildapi/SkylarkRuleContextApi.java there is:
#SkylarkCallable(
name = "info_file",
structField = true,
documented = false,
doc =
"Returns the file that is used to hold the non-volatile workspace status for the "
+ "current build request."
)
public FileApi getStableWorkspaceStatus() throws InterruptedException, EvalException;
EDIT: few extra details as asked in the comment.
In my workspace_status.sh I would have for instance the following line:
echo STABLE_GIT_REF $(git log -1 --pretty=format:%H)
In my .json.tmpl file I would then have:
"ref": "${STABLE_GIT_REF}",
I've opted for shell like notation of text to be replaced, since it's intuitive for many users as well as easy to match.
As for the replacement, relevant (CLI kept out of this) portion of the actual code would be:
def get_map(val_file):
"""
Return dictionary of key/value pairs from ``val_file`.
"""
value_map = {}
for line in val_file:
(key, value) = line.split(' ', 1)
value_map.update(((key, value.rstrip('\n')),))
return value_map
def expand_template(val_file, in_file, out_file):
"""
Read each line from ``in_file`` and write it to ``out_file`` replacing all
${KEY} references with values from ``val_file``.
"""
def _substitue_variable(mobj):
return value_map[mobj.group('var')]
re_pat = re.compile(r'\${(?P<var>[^} ]+)}')
value_map = get_map(val_file)
for line in in_file:
out_file.write(re_pat.subn(_substitue_variable, line)[0])
EDIT2: This is how the Python script is how I expose the python script to rest of bazel.
py_binary(
name = "expand_template",
main = "expand_template.py",
srcs = ["expand_template.py"],
visibility = ["//visibility:public"],
)
Building on Ondrej's answer, I now use somthing like this (adapted in SO editor, might contain small errors):
tools/bazel.rc:
build --workspace_status_command=tools/workspace_status.sh
tools/workspace_status.sh:
echo STABLE_GIT_REV $(git rev-parse HEAD)
version.bzl:
_VERSION_TEMPLATE_SH = """
set -e -u -o pipefail
while read line; do
export "${line% *}"="${line#* }"
done <"$INFILE" \
&& cat <<EOF >"$OUTFILE"
{ "ref": "${STABLE_GIT_REF}"
, "service": "${SERVICE_NAME}"
}
EOF
"""
def _commit_info_impl(ctx):
ctx.actions.run_shell(
outputs = [ctx.outputs.outfile],
inputs = [ctx.info_file],
progress_message = "Generating version file: " + ctx.label.name,
command = _VERSION_TEMPLATE_SH,
env = {
'INFILE': ctx.info_file.path,
'OUTFILE': ctx.outputs.version_go.path,
'SERVICE_NAME': ctx.attr.service,
},
)
commit_info = rule(
implementation = _commit_info_impl,
attrs = {
'service': attr.string(
mandatory = True,
doc = 'name of versioned service',
),
},
outputs = {
'outfile': 'manifest.json',
},
)
So, I have a parser, written in TCL. There are many commands in the parsing file. Now, I need to add support for line breaks.
For ex.
my_command \
arg1 \
arg2 \
arg3
I have something like this.
while { ! [eof $currentFileDescriptor] } {
set line [gets $currentFileDescriptor]
set lst [lindex [regexp -all -inline {^(\s*(\S*)\s*)*(\{(.*)\})?(\s*(\S*)\s*)*$} $line] 0]
set tok [string toupper [lindex $lst 0]]
switch -glob $tok {
"\#*" { }
"MY_COMMAND_1" { parseMyCommand1 $handler $lst }
.....#other commands }
}
incr lnum
}
I am looking for an optimal and effective solution.
It looks like you have defined a domain specific language (DSL) with the parsing implemented in Tcl. You may as well use the Tcl parsing itself to deal with things like line continuation and quote handling. The method to do this is to create a safe interpreter and in the safe interpreter only provide the commands required for your DSL. You then interpret your config file in the safe child interpreter. The wiki page has some examples.
The advantage of this method is that the parsing is handled by the normal Tcl parser. However you can be in complete control of what commands are exposed in the safe interpreter. You can also control the amount of resources it can use (stack and memory) and limit it's visibility of the filesystem or network.
If you don't want to get into this then you just need to implement recognition of backslashed newlines and buffer such lines until you have a complete line. Something like the following (untested):
set linenum 0
set buffer ""
while {[gets $input line] != -1} {
incr linenum
if {[regexp {\\$} $line]} {
append buffer [string range $line 0 end-1]
continue
} else {
append buffer $line
}
ParseCompleteLine $linenum $buffer
set buffer ""
}
I have a program that accepts a destination folder where files will be created. My program should be able to handle absolute paths as well as relative paths. My problem is that I don't know how to expand ~ to the home directory.
My function to expand the destination looks like this. If the path given is absolute it does nothing otherwise it joins the relative path with the current working directory.
import "path"
import "os"
// var destination *String is the user input
func expandPath() {
if path.IsAbs(*destination) {
return
}
cwd, err := os.Getwd()
checkError(err)
*destination = path.Join(cwd, *destination)
}
Since path.Join doesn't expand ~ it doesn't work if the user passes something like ~/Downloads as the destination.
How should I solve this in a cross platform way?
Go provides the package os/user, which allows you to get the current user, and for any user, their home directory:
usr, _ := user.Current()
dir := usr.HomeDir
Then, use path/filepath to combine both strings to a valid path:
if path == "~" {
// In case of "~", which won't be caught by the "else if"
path = dir
} else if strings.HasPrefix(path, "~/") {
// Use strings.HasPrefix so we don't match paths like
// "/something/~/something/"
path = filepath.Join(dir, path[2:])
}
(Note that user.Current() is not implemented in the go playground (likely for security reasons), so I can't give an easily runnable example).
In general the ~ is expanded by your shell before it gets to your program. But there are some limitations.
In general is ill-advised to do it manually in Go.
I had the same problem in a program of mine and what I have understood is that if I use the flag format as --flag=~/myfile, it is not expanded. But if you run --flag ~/myfile it is expanded by the shell (the = is missing and the filename appears as a separate "word").
Normally, the ~ is expanded by the shell before your program sees it.
Adjust how your program acquires its arguments from the command line in a way compatible with the shell expansion mechanism.
One of the possible problems is using exec.Command like this:
cmd := exec.Command("some-binary", someArg) // say 'someArg' is "~/foo"
which will not get expanded. You can, for example use instead:
cmd := exec.Command("sh", "-c", fmt.Sprintf("'some-binary %q'", someArg))
which will get the standard ~ expansion from the shell.
EDIT: fixed the 'sh -c' example.
If you are expanding tilde '~' for use with exec.Command() you should use the users local shell for expansion.
// 'sh', 'bash' and 'zsh' all respect the '-c' argument
cmd := exec.Command(os.Getenv("SHELL"), "-c", "cat ~/.myrc")
cmd.Stdout = os.Stdout
if err := cmd.Run(); err != nil {
fmt.Fprintln(os.Stderr, err)
}
However; when loading application config files such as ~./myrc this solution is not acceptable. The following has worked well for me across multiple platforms
import "os/user"
import "path/filepath"
func expand(path string) (string, error) {
if len(path) == 0 || path[0] != '~' {
return path, nil
}
usr, err := user.Current()
if err != nil {
return "", err
}
return filepath.Join(usr.HomeDir, path[1:]), nil
}
NOTE: usr.HomeDir does not respect $HOME instead determines the home directory by reading the /etc/passwd file via the getpwuid_r syscall on (osx/linux). On windows it uses the OpenCurrentProcessToken syscall to determine the users home directory.
I know this is an old question but there is another option now. You can use go-homedir to expand the tidle to the user's homedir:
myPath := "~/.ssh"
fmt.Printf("path: %s; with expansion: %s", myPath, homedir.Expand(myPath))
This works on go >= 1.12:
if strings.HasPrefix(path, "~/") {
home, _ := os.UserHomeDir()
path = filepath.Join(home, path[2:])
}