If I query for available packages for Go:
nix-env -qa go -b --put-path --json
I get:
{
"nixpkgs.go_1_16": {
"name": "go-1.16.15",
"pname": "go",
"version": "1.16.15",
"system": "x86_64-linux",
"outputs": {
"out": "/nix/store/nqi39ksavkfrxkrz3d0797n5wmzi9r30-go-1.16.15"
}
},
"nixpkgs.go": {
"name": "go-1.17.7",
"pname": "go",
"version": "1.17.7",
"system": "x86_64-linux",
"outputs": {
"out": "/nix/store/3v2l94h7pllq6za9km3388cyd5agrln7-go-1.17.7"
}
},
"nixpkgs.go_1_18": {
"name": "go-1.18",
"pname": "go",
"version": "1.18",
"system": "x86_64-linux",
"outputs": {
"out": "/nix/store/7jyfpb96xv3hr8dpfhnbb0f7zscwm7sr-go-1.18"
}
}
}
I'm curios, how does nix-env knows the store path in advance? As far as I'm aware, the store path is generated by nix on build time, but nix-env already knows it without building(i see nothing gets downloaded from network for example).
The build process doesn't directly take .nix code (as downloaded from channels like https://nixos.org/channels/nixos-21.11) as inputs. Instead, it takes derivations as inputs; those derivations are generated by evaluating code written in the Nix language, and then are written into the Nix store as .drv files (which can be readily decoded to JSON for reading; see nix show-derivation). These .drv files are what is used as input by the actual build process (run by the nixbld sandboxed user accounts).
Those derivations fully specify what the build process will do -- all its inputs, and all its outputs. Because the outputs (except for those of fixed-output derivations) are addressed by the hash of the build process used to generate them, it's not necessary to actually run a build before the hash is known.
The exception I mentioned above is a fixed-output derivation; these are used for downloading resources that will be used by the build process off the network. For a Nix build step to be allowed network access, it needs to assert ahead-of-time what the output of that build step will be -- and the step is considered failed if it produces any output that does not match what was stated. In these, the hash of the content itself is used for the store location, so changes to the download process don't require re-downloading content.
The ability to predict the output location pre-build is critical because packages are pulled down from the Hydra binary cache by their hashes. If we didn't know the hashes until we had already built the package, the cache would be useless: We wouldn't know what name to look for the binary under unless we already had that binary in our posession! (And similarly, we wouldn't be able to plan more than one step ahead if we didn't know the names and hashes of our inputs without actually building them).
Let's take a practical example. Take it as given that as of the time that I'm writing this, I don't have GCC installed via Nix.
$ nix repl
Welcome to Nix version 2.3.16. Type :? for help.
nix-repl> :l <nixpkgs>
Added 15472 variables.
nix-repl> gcc8
«derivation /nix/store/9fpas3flqf424g46b8ldkbz7sgd9r7qk-gcc-wrapper-8.5.0.drv»
nix-repl> :b gcc8
...and either a download or a long build process runs here.
Notice how evaluating gcc8 gives us a /nix/store/*-gcc*.drv file. That file contains a plan describing how Nix would build gcc 8 if we wanted it to.
We can demonstrate that that plan includes the output location:
$ nix show-derivation /nix/store/9fpas3flqf424g46b8ldkbz7sgd9r7qk-gcc-wrapper-8.5.0.drv | jq '.[].outputs'
{
"info": {
"path": "/nix/store/znasm5jz3pp57ivspw5ahgn7rzfk791w-gcc-wrapper-8.5.0-info"
},
"man": {
"path": "/nix/store/j46y5mrppjw7nw9g8ckd3h438k8jjvkr-gcc-wrapper-8.5.0-man"
},
"out": {
"path": "/nix/store/v9pv2w7qiw1cpbjn4wjdkxkzld7pfki4-gcc-wrapper-8.5.0"
}
}
Related
On nixos, I am trying to compile the hello world example listed in the agda documentation.
In my working directory, I have the following:
The hello-world agda program, hello-world.agda:
module hello-world where
open import IO
main = run (putStrLn "Hello, World!")
A nix shell file, shell.nix:
{ pkgs ? import <nixpkgs> { } }:
with pkgs;
mkShell {
buildInputs = [
(agda.withPackages (ps: [
ps.standard-library
]))
];
}
To enter a shell with the standard-library dependency available, I ran $ nix-shell shell.nix.
Then, trying to compile the program, I ran $ agda --compile hello-world.agda, as advised by the linked agda hello world documentation.
But that gave me the following error:
$ agda --compile hello-world.agda
Checking hello-world (/home/matthew/backup/projects/agda-math/hello-world.agda).
/home/matthew/backup/projects/agda-math/hello-world.agda:3,1-15
Failed to find source of module IO in any of the following
locations:
/home/matthew/backup/projects/agda-math/IO.agda
/home/matthew/backup/projects/agda-math/IO.lagda
/nix/store/7pg293b76ppv2rw2saf5lcbckn6kdy7z-Agda-2.6.2.2-data/share/ghc-9.0.2/x86_64-linux-ghc-9.0.2/Agda-2.6.2.2/lib/prim/IO.agda
/nix/store/7pg293b76ppv2rw2saf5lcbckn6kdy7z-Agda-2.6.2.2-data/share/ghc-9.0.2/x86_64-linux-ghc-9.0.2/Agda-2.6.2.2/lib/prim/IO.lagda
when scope checking the declaration
open import IO
It seems it should be finding the standard library, since I'm running from the nix-shell with agda's standard-library specified, but that error on open import IO looks like the standard library is somehow still not found.
Any idea what the problem is likely to be?
Or what else I can do to get agda working on nixos?
You might need to create a defaults file in AGDA_DIR (which typically refers to ~/.agda/ unless overwritten through environment variable) with the libraries you want to make available to your program:
echo standard-library >> ~/.agda/defaults
which should make the compiler automatically load the standard library.
Alternatively, you can pass them in on the command-line:
agda -l standard-library
or use a project-local .agda-lib file as follows:
name: my-libary
depend: standard-library
include: -- Optionally specify include paths
You should not need to specify the include paths with Nix, but in case you do, you can use the -i command line flag or add the path to the standard-library.agda-lib file in ~/.agda/libraries.
I'm making a haskell program and I'm setting buildInput like this to include pkgs.ffmpeg-full:
(myHaskellPackages.callCabal2nix "App" (./.) {}).overrideAttrs (oldAttrs: {
buildInputs = (oldAttrs.buildInputs or []) ++ [ pkgs.ffmpeg-full ];
})
However this seems to make the ffmpeg package accessible during build time only rather than runtime of the application.
What attribute do I need to set for ffmpeg-full to be available during runtime - being able to invoke the ffmpeg executable?
There is a section about runtime dependencies in nix pills but I don't understand that section, it doesn't make sense how it can always determine runtime dependencies by hashes alone? I mean if I reference an executable in a shell script - surely nix does not parse the shell script to determine the executable I reference. https://nixos.org/guides/nix-pills/automatic-runtime-dependencies.html#idm140737320205792
Something is different for runtime dependencies however. Build
dependencies are automatically recognized by Nix once they are used in
any derivation call, but we never specify what are the runtime
dependencies for a derivation.
There's really black magic involved. It's something that at first
glance makes you think "no, this can't work in the long term", but at
the same time it works so well that a whole operating system is built
on top of this magic.
In other words, Nix automatically computes all the runtime
dependencies of a derivation, and it's possible thanks to the hash of
the store paths.
default.nix:
{
ghc ? "ghc8106",
pkgs ? import <nixpkgs> {}
}:
with pkgs.haskell.lib;
let
haskellPkgs = pkgs.haskell.packages.${ghc};
inherit (pkgs) lib;
mySourceRegexes = [
"^app.*$"
"^.*\\.cabal$"
"package.yaml"
];
myApp = (haskellPkgs.callCabal2nix "my-hello"
(lib.sourceByRegex ./. mySourceRegexes) { });
in myApp
.overrideAttrs(
oa: {
nativeBuildInputs = oa.nativeBuildInputs ++ [pkgs.hello pkgs.makeWrapper];
installPhase = oa.installPhase + ''
ln -s ${pkgs.hello.out}/bin/hello $out/bin/hello
'';
postFixup = ''
wrapProgram $out/bin/x-exe --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.hello ]}
'';
})
src/Main.hs:
module Main where
import System.Process (callCommand)
main :: IO ()
main = do
putStrLn "HELLO"
callCommand "hello"
putStrLn "BYE"
Seems this is not directly supported with an explicitly stated list of dependencies. However we can indirectly achieve this with "wrapping".
I found more information about wrapping here: https://nixos.wiki/wiki/Nix_Cookbook#Wrapping_packages
So I can do a ls that references the package.
...
appPkg = (myHaskellPackages.callCabal2nix "HaskellNixCabalStarter" (./.) {}).overrideAttrs (oldAttrs: {
buildInputs = (oldAttrs.buildInputs or []) ++ [ pkgs.ffmpeg-full ];
});
in
(pkgs.writeScriptBin "finderapp" ''
#!${pkgs.stdenv.shell}
ls ${pkgs.ffmpeg-full}/bin/ffmpeg
exec ${appPkg}/bin/app
''
)
We can verify the built package(?) correctly depends on the appropriate with:
nix-store -q --references result
/nix/store/0cq84xic2absp75ciajv4lfx5ah1fb59-ffmpeg-full-4.2.2
/nix/store/rm1hz1lybxangc8sdl7xvzs5dcvigvf7-bash-4.4-p23
/nix/store/wlvnjx53xfangaa4m5rmabknjbgpvq3d-HaskellNixCabalStarter-0.1.0.0
I've got a default.nix file that builds a derivation (at least my understanding of it).
{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc864" } :
nixpkgs.pkgs.haskell.packages.${compiler}.callCabal2nix "bhoogle" (./.) {}
I can successfully nix-build this. Is there a way I can install this into my user profile with nix-env directly? For example something like nix-env -i -f default.nix.
Otherwise I need to define a package in my system profile with something like:
example = pkgs.callPackage /home/chris/example/default.nix {};
Quite literally my initial guess (thanks #Robert):
nix-env -i -f default.nix
Documented in nix --help:
--file / -f path
Specifies the Nix expression (designated below as the active Nix expression) used by the --install, --upgrade, and --query
--available operations to obtain
derivations. The default is ~/.nix-defexpr.
If the argument starts with http:// or https://, it is interpreted as the URL of a tarball that will be downloaded and
unpacked to a temporary location. The tarball
must include a single top-level directory containing at least a file named default.nix.
I have set up bazel to build a number of CLI tools that perform various database maintenance tasks. Each one is a py_binary or cc_binary target that is called from the command line with the path to some data file: it processes that file and stores the results in a database.
Now, I need to create a dependent package that contains data files and shell scripts that call these CLI tools to perform application-specific database operations.
However, there doesn't seem to be a way to depend on the existing py_binary or cc_binary targets from a new package that only contains sh_binary targets and data files. Trying to do so results in an error like:
ERROR: /workspace/shbin/BUILD.bazel:5:12: in deps attribute of sh_binary rule //shbin:run: py_binary rule '//pybin:counter' is misplaced here (expected sh_library)
Is there a way to call/depend on an existing bazel binary target from a shell script using sh_binary?
I have implemented a full example here:
https://github.com/psigen/bazel-mixed-binaries
Notes:
I cannot use py_library and cc_library instead of py_binary and cc_binary. This is because (a) I need to call mixes of the two languages to process my data files and (b) these tools are from an upstream repository where they are already designed as CLI tools.
I also cannot put all the data files into the CLI tool packages -- there are multiple application-specific packages and they cannot be mixed.
You can either create a genrule to run these tools as part of the build, or create a sh_binary that depends on the tools via the data attribute and runs them them.
The genrule approach
This is the easier way and lets you run the tools as part of the build.
genrule(
name = "foo",
tools = [
"//tool_a:py",
"//tool_b:cc",
],
srcs = [
"//source:file1",
":file2",
],
outs = [
"output_file1",
"output_file2",
],
cmd = "$(location //tool_a:py) --input=$(location //source:file1) --output=$(location output_file1) && $(location //tool_b:cc) < $(location :file2) > $(location output_file2)",
)
The sh_binary approach
This is more complicated, but lets you run the sh_binary either as part of the build (if it is in a genrule.tools, similar to the previous approach) or after the build (from under bazel-bin).
In the sh_binary you have to data-depend on the tools:
sh_binary(
name = "foo",
srcs = ["my_shbin.sh"],
data = [
"//tool_a:py",
"//tool_b:cc",
],
)
Then, in the sh_binary you have to use the so-called "Bash runfiles library" built into Bazel to look up the runtime-path of the binaries. This library's documentation is in its source file.
The idea is:
the sh_binary has to depend on a specific target
you have to copy-paste some boilerplate code to the top of the sh_binary (reason is described here)
then you can use the rlocation function to look up the runtime-path of the binaries
For example your my_shbin.sh may look like this:
#!/bin/bash
# --- begin runfiles.bash initialization ---
...
# --- end runfiles.bash initialization ---
path=$(rlocation "__main__/tool_a/py")
if [[ ! -f "${path:-}" ]]; then
echo >&2 "ERROR: could not look up the Python tool path"
exit 1
fi
$path --input=$1 --output=$2
The __main__ in the rlocation path argument is the name of the workspace. Since your WORKSPACE file does not have a "workspace" rule in, which would define the workspace's name, Bazel will use the default workspace name, which is __main__.
An easier approach for me is to add the cc_binary as a dependency in the data section. In prefix/BUILD
cc_binary(name = "foo", ...)
sh_test(name = "foo_test", srcs = ["foo_test.sh"], data = [":foo"])
Inside foo_test.sh, the working directory is different, so you need to find the right prefix for the binary
#! /usr/bin/env bash
executable=prefix/foo
$executable ...
A clean way to do this is to use args and $(location):
Contents of BUILD:
py_binary(
name = "counter",
srcs = ["counter.py"],
main = "counter.py",
)
sh_binary(
name = "run",
srcs = ["run.sh"],
data = [":counter"],
args = ["$(location :counter)"],
)
Contents of counter.py (your tool):
print("This is the counter tool.")
Contents of run.sh (your bash script):
#!/bin/bash
set -eEuo pipefail
counter="$1"
shift
echo "This is the bash script, about to call the counter tool."
"$counter"
And here's a demo showing the bash script calling the Python tool:
$ bazel run //example:run 2>/dev/null
This is the bash script, about to call the counter tool.
This is the counter tool.
It's also worth mentioning this note (from the docs):
The arguments are not passed when you run the target outside of bazel (for example, by manually executing the binary in bazel-bin/).
So, finally starting to get a stable nix environment that I can basically do all of my development in. Hooray!
Now I want to make it reproducible, as in yarn.lock (for those familiar with npm/yarn in javascript land) or Pipfile.lock (very similar for Python).
Basically the idea is that I would have a way to generate a similar lock file whenever I run nix-env -if my-env.nix, or after running this command, if that is how it would work. From this lock file, I could then exactly restore my nix profile, down to the exact versions of dependencies and sub-dependencies of the installed profile. This could be checked into git or whatever after testing out new improvements, and so a record of the environment would be maintained.
It seems to me this would be one of the most obvious use cases for Nix, and one of the major advantages over just using Docker (though the two aren't mutually exclusive), so I apologize if I've missed some relevant documentation.
What you're probably looking for is a shell.nix file like this:
let
pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/696c6bed4e8e2d9fd9b956dea7e5d49531e9d13f.tar.gz";
sha256 = "1v3yrpj542niyxp0h3kffsdjwlrkvj0mg4ljb85d142gyn3sdzd4";
}) {};
in pkgs.mkShell {
buildInputs = with pkgs; [
git
hello
];
}
Upon calling nix-shell (which by default uses the shell.nix file in the current directory), you'll be in an environment with git and hello from the specific given nixpkgs revision. This can be reproduced among all nix users (okay almost*). I can really recommend using shell.nix files for all development.
Alternatively, there's also the lesser known -r flag to nix-env, which states that
--remove-all, -r
Remove all previously installed packages first. This is equivalent to running nix-env -e '.*' first, except that everything happens in a single transaction.
You can effectively use it to replace the stateful ~/.nix-profile/manifest.nix. Create a file env.nix containing:
let
pkgs = import <nixpkgs> {};
in {
inherit (pkgs) git hello;
}
Now running nix-env -ir env.nix will install exactly git and hello and remove everything else, so you can reproduce your nix-env installations with this single file. To install additional things: Add it to the file and run the command again. You can also pin nixpkgs to a specific version as in the file above to not care about your nix-channel setup (which is also stateful).
Edit: It's also possible to get some packages from your channel and some of a specific nixpkgs revision:
let
pkgs = import <nixpkgs> {};
fixed = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/696c6bed4e8e2d9fd9b956dea7e5d49531e9d13f.tar.gz";
sha256 = "1v3yrpj542niyxp0h3kffsdjwlrkvj0mg4ljb85d142gyn3sdzd4";
}) {};
in {
inherit (pkgs) git;
inherit (fixed) hello;
}
*: The nix config, overlays and your system (Linux/Mac) can still influence this. It's a good idea to use import <nixpkgs> { config = {}; overlays = []; } for development to avoid this.
After many suggestions from folks on IRC, I was able to put together the following script, which takes a nix expression as its only argument, and copies the derivation file and records the nixpkgs versions locally while installing the specified environment:
#!/bin/bash
if [ -z "$1" ] || [ "${1: -4}" != ".nix" ] || [ ! -f "$1" ]
then
echo "No .nix file supplied"
exit -1
fi
ENV_DRV=$(nix-instantiate "$1")
cp "$ENV_DRV" ./env_backup.drv
chmod u+rw ./env_backup.drv
nix-env --set "$ENV_DRV"
NIXPKGS_VERSION=$(nix-instantiate --eval '<nixpkgs/lib>' -A version)
NIXOS_VERSION=$(nix-instantiate --eval '<nixos/lib>' -A version)
printf "nixpkgs: %s\\nnixos: %s" "$NIXPKGS_VERSION" "${NIXOS_VERSION}" > .nix_versions
Caveat: your nix expression should have the nix package in it, since we are using nix-env --set.
I haven't tried using a copied .drv file yet, but it somehow needs to be restored into the nix store; I've mainly included it as a last resort and for debugging. The output into .nix_versions should be more helpful, as these contain git commit hashes (after the last ".") that can be used to employ the correct revision of nixpkgs (thanks to infinisil on IRC):
pkgs = import "${(import <nixpkgs> {}).fetchFromGitHub { owner = "NixOS"; repo = "nixpkgs"; rev = "<your revision hash>"; sha256 = "<the hash of the output>"; }}" {}
To fill in the hash, either supply an incorrect hash to get the correct hash back, or just use the following: nix-prefetch-url --unpack github.com/nixos/nixpkgs/archive/<revision>.tar.gz.
Or, if manually checking out nixpkgs, you can just do e.g.:
with import ((builtins.getEnv "HOME") + "/workspace/nixpkgs") { }; # or:
with import "../nixpkgs" { }; # or similar
I haven't tested this sort of thing with nix-shell yet, but hope to do so soon.