I've just started using the Nix package manager on OSX and I'm attempting to create my first package for the pass binary (https://www.passwordstore.org) - which is not available in the Nixpkgs repository.
I'm attempting to specify a runtime dependency (getopt), however this doesn't appear to be available when the binary is used.
This is my packages's default.nix:
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
let
version = "1.7.1";
in {
pass = stdenv.mkDerivation rec {
name = "pass-${version}";
src = fetchurl {
url = "https://git.zx2c4.com/password-store/snapshot/password-store-1.7.1.tar.xz";
sha256 = "0scqkpll2q8jhzcgcsh9kqz0gwdpvynivqjmmbzax2irjfaiklpn";
};
buildInputs = [ stdenv makeWrapper];
installPhase = ''
make install PREFIX=$out/artifact
makeWrapper $out/artifact/bin/pass $out/bin/pass \
--set PATH ${stdenv.lib.makeBinPath [ getopt ]}
'';
meta = {
homepage = "https://www.passwordstore.org";
description = "The standard unix password manager";
license = stdenv.lib.licenses.gpl2Plus;
};
};
}
I can successfully build this package (nix-build --show-trace) and install it (nix-env -i ./result).
Listing the runtime dependencies for my package shows getopt listed:
nix-store -qR $(which pass)
...
/nix/store/c5swmygzc0kmvpq6cfkvwm2yz1k57kqy-getopt-1.1.4
However when I come to use the binary (pass init my-key) I get the following error:
/nix/store/...-pass-1.7.1/artifact/bin/pass: line 302:
/usr/local/bin/getopt: No such file or directory
Can anyone advise what I'm doing wrong?
Thanks
It looks like getopt gets a special treatment. The darwin.sh script looks for it using brew and port and falls back to /usr/local. That's why the (correct) wrapper has no effect.
So the solution seems to be, to make it look for getopt in PATH, which is provided by the wrapper script. You can probably make it as simple as GETOPT=getopt (which is similar to openbsd.sh)
For patching source code, see the NixPkgs documentation
After running nix-build, you should run cat result/bin/pass to look at your wrapper script and make sure it looks OK. It should be a shell script that sets the PATH to include getopt and then calls result/artifact/bin/pass.
Then try running the wrapper script. Note that the wrapper should be in result/bin, not result/artifact/bin.
Related
I'm making a haskell program and I'm setting buildInput like this to include pkgs.ffmpeg-full:
(myHaskellPackages.callCabal2nix "App" (./.) {}).overrideAttrs (oldAttrs: {
buildInputs = (oldAttrs.buildInputs or []) ++ [ pkgs.ffmpeg-full ];
})
However this seems to make the ffmpeg package accessible during build time only rather than runtime of the application.
What attribute do I need to set for ffmpeg-full to be available during runtime - being able to invoke the ffmpeg executable?
There is a section about runtime dependencies in nix pills but I don't understand that section, it doesn't make sense how it can always determine runtime dependencies by hashes alone? I mean if I reference an executable in a shell script - surely nix does not parse the shell script to determine the executable I reference. https://nixos.org/guides/nix-pills/automatic-runtime-dependencies.html#idm140737320205792
Something is different for runtime dependencies however. Build
dependencies are automatically recognized by Nix once they are used in
any derivation call, but we never specify what are the runtime
dependencies for a derivation.
There's really black magic involved. It's something that at first
glance makes you think "no, this can't work in the long term", but at
the same time it works so well that a whole operating system is built
on top of this magic.
In other words, Nix automatically computes all the runtime
dependencies of a derivation, and it's possible thanks to the hash of
the store paths.
default.nix:
{
ghc ? "ghc8106",
pkgs ? import <nixpkgs> {}
}:
with pkgs.haskell.lib;
let
haskellPkgs = pkgs.haskell.packages.${ghc};
inherit (pkgs) lib;
mySourceRegexes = [
"^app.*$"
"^.*\\.cabal$"
"package.yaml"
];
myApp = (haskellPkgs.callCabal2nix "my-hello"
(lib.sourceByRegex ./. mySourceRegexes) { });
in myApp
.overrideAttrs(
oa: {
nativeBuildInputs = oa.nativeBuildInputs ++ [pkgs.hello pkgs.makeWrapper];
installPhase = oa.installPhase + ''
ln -s ${pkgs.hello.out}/bin/hello $out/bin/hello
'';
postFixup = ''
wrapProgram $out/bin/x-exe --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.hello ]}
'';
})
src/Main.hs:
module Main where
import System.Process (callCommand)
main :: IO ()
main = do
putStrLn "HELLO"
callCommand "hello"
putStrLn "BYE"
Seems this is not directly supported with an explicitly stated list of dependencies. However we can indirectly achieve this with "wrapping".
I found more information about wrapping here: https://nixos.wiki/wiki/Nix_Cookbook#Wrapping_packages
So I can do a ls that references the package.
...
appPkg = (myHaskellPackages.callCabal2nix "HaskellNixCabalStarter" (./.) {}).overrideAttrs (oldAttrs: {
buildInputs = (oldAttrs.buildInputs or []) ++ [ pkgs.ffmpeg-full ];
});
in
(pkgs.writeScriptBin "finderapp" ''
#!${pkgs.stdenv.shell}
ls ${pkgs.ffmpeg-full}/bin/ffmpeg
exec ${appPkg}/bin/app
''
)
We can verify the built package(?) correctly depends on the appropriate with:
nix-store -q --references result
/nix/store/0cq84xic2absp75ciajv4lfx5ah1fb59-ffmpeg-full-4.2.2
/nix/store/rm1hz1lybxangc8sdl7xvzs5dcvigvf7-bash-4.4-p23
/nix/store/wlvnjx53xfangaa4m5rmabknjbgpvq3d-HaskellNixCabalStarter-0.1.0.0
So, finally starting to get a stable nix environment that I can basically do all of my development in. Hooray!
Now I want to make it reproducible, as in yarn.lock (for those familiar with npm/yarn in javascript land) or Pipfile.lock (very similar for Python).
Basically the idea is that I would have a way to generate a similar lock file whenever I run nix-env -if my-env.nix, or after running this command, if that is how it would work. From this lock file, I could then exactly restore my nix profile, down to the exact versions of dependencies and sub-dependencies of the installed profile. This could be checked into git or whatever after testing out new improvements, and so a record of the environment would be maintained.
It seems to me this would be one of the most obvious use cases for Nix, and one of the major advantages over just using Docker (though the two aren't mutually exclusive), so I apologize if I've missed some relevant documentation.
What you're probably looking for is a shell.nix file like this:
let
pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/696c6bed4e8e2d9fd9b956dea7e5d49531e9d13f.tar.gz";
sha256 = "1v3yrpj542niyxp0h3kffsdjwlrkvj0mg4ljb85d142gyn3sdzd4";
}) {};
in pkgs.mkShell {
buildInputs = with pkgs; [
git
hello
];
}
Upon calling nix-shell (which by default uses the shell.nix file in the current directory), you'll be in an environment with git and hello from the specific given nixpkgs revision. This can be reproduced among all nix users (okay almost*). I can really recommend using shell.nix files for all development.
Alternatively, there's also the lesser known -r flag to nix-env, which states that
--remove-all, -r
Remove all previously installed packages first. This is equivalent to running nix-env -e '.*' first, except that everything happens in a single transaction.
You can effectively use it to replace the stateful ~/.nix-profile/manifest.nix. Create a file env.nix containing:
let
pkgs = import <nixpkgs> {};
in {
inherit (pkgs) git hello;
}
Now running nix-env -ir env.nix will install exactly git and hello and remove everything else, so you can reproduce your nix-env installations with this single file. To install additional things: Add it to the file and run the command again. You can also pin nixpkgs to a specific version as in the file above to not care about your nix-channel setup (which is also stateful).
Edit: It's also possible to get some packages from your channel and some of a specific nixpkgs revision:
let
pkgs = import <nixpkgs> {};
fixed = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/696c6bed4e8e2d9fd9b956dea7e5d49531e9d13f.tar.gz";
sha256 = "1v3yrpj542niyxp0h3kffsdjwlrkvj0mg4ljb85d142gyn3sdzd4";
}) {};
in {
inherit (pkgs) git;
inherit (fixed) hello;
}
*: The nix config, overlays and your system (Linux/Mac) can still influence this. It's a good idea to use import <nixpkgs> { config = {}; overlays = []; } for development to avoid this.
After many suggestions from folks on IRC, I was able to put together the following script, which takes a nix expression as its only argument, and copies the derivation file and records the nixpkgs versions locally while installing the specified environment:
#!/bin/bash
if [ -z "$1" ] || [ "${1: -4}" != ".nix" ] || [ ! -f "$1" ]
then
echo "No .nix file supplied"
exit -1
fi
ENV_DRV=$(nix-instantiate "$1")
cp "$ENV_DRV" ./env_backup.drv
chmod u+rw ./env_backup.drv
nix-env --set "$ENV_DRV"
NIXPKGS_VERSION=$(nix-instantiate --eval '<nixpkgs/lib>' -A version)
NIXOS_VERSION=$(nix-instantiate --eval '<nixos/lib>' -A version)
printf "nixpkgs: %s\\nnixos: %s" "$NIXPKGS_VERSION" "${NIXOS_VERSION}" > .nix_versions
Caveat: your nix expression should have the nix package in it, since we are using nix-env --set.
I haven't tried using a copied .drv file yet, but it somehow needs to be restored into the nix store; I've mainly included it as a last resort and for debugging. The output into .nix_versions should be more helpful, as these contain git commit hashes (after the last ".") that can be used to employ the correct revision of nixpkgs (thanks to infinisil on IRC):
pkgs = import "${(import <nixpkgs> {}).fetchFromGitHub { owner = "NixOS"; repo = "nixpkgs"; rev = "<your revision hash>"; sha256 = "<the hash of the output>"; }}" {}
To fill in the hash, either supply an incorrect hash to get the correct hash back, or just use the following: nix-prefetch-url --unpack github.com/nixos/nixpkgs/archive/<revision>.tar.gz.
Or, if manually checking out nixpkgs, you can just do e.g.:
with import ((builtins.getEnv "HOME") + "/workspace/nixpkgs") { }; # or:
with import "../nixpkgs" { }; # or similar
I haven't tested this sort of thing with nix-shell yet, but hope to do so soon.
I want to run a nix-shell with the following packages installed:
aspell
aspellDicts.en
hello
I can't simply do: nix-shell -p aspell aspellDicts.en hello --pure as this will not correctly install the aspell dictionaries. Nix provides a aspellWithDict function that can be used to build aspell with dictionaries:
nix-build -E 'with import <nixpkgs> {}; aspellWithDicts (d: [d.en])'
I want to use the result of this build as a dependency in another local package (foo). This is how I'm currently achieving this:
./pkgs/aspell-with-dicts/default.nix:
with import <nixpkgs> {};
aspellWithDicts (d: [d.en])
./pkgs/foo/default.nix:
{stdenv, aspellWithDicts, hello}:
stdenv.mkDerivation rec {
name = "foo";
buildInputs = [ aspellWithDicts hello ];
}
./custom-packages.nix:
{ system ? builtins.currentSystem }:
let
pkgs = import <nixpkgs> { inherit system; };
in
rec {
aspellWithDicts = import ./pkgs/aspell-with-dicts;
foo = import ./pkgs/foo {
aspellWithDicts = aspellWithDicts;
hello = pkgs.hello;
stdenv = pkgs.stdenv;
};
}
Running the shell works as expected: nix-shell ./custom-packages.nix -A foo --pure
So my solution works, but could this outcome be achieved in a more succinct idiomatic way?
Do you need to build foo? What in foo you will use?
Assume you only want to use the shell via nix-shell and not want to build/install anything using nix-build or nix-env -i, this should works.
The following shell.nix
with import <nixpkgs> {};
with pkgs;
let
myAspell = aspellWithDicts (d: [d.en]);
in
stdenv.mkDerivation {
name = "myShell";
buildInputs = [myAspell hello];
shellHooks = ''
echo Im in $name.
echo aspell is locate at ${myAspell}
echo hello is locate at ${hello}
'';
}
will give you a shell with aspell and hello
$ nix-shell
Im in myShell.
aspell is locate at /nix/store/zcclppbibcg4nfkis6zqml8cnrlnx00b-aspell-env
hello is locate at /nix/store/gas2p68jqbzgb7zr96y5nc8j7nk61kkk-hello-2.10
If it is the case that foo have some code to build and install.
mkDerivation in foo/default.nix must have src field which could be src = ./.; or something like fetchurl or fetchFromGithub (see document for examples).
Then you can use callPackages or import (depends on how the nix expression was written) with foo/default.nix as argument to bring what foo provided to use in this shell.
If you try to build this shell.nix (or foo/default.nix) it will failed with missing src
$ nix-build shell.nix
these derivations will be built:
/nix/store/20h8cva19irq8vn39i72j8iz40ivijhr-myShell.drv
building path(s) ‘/nix/store/r1f6qpxz91h5jkj7hzrmaymmzi9h1yml-myShell’
unpacking sources
variable $src or $srcs should point to the source
builder for ‘/nix/store/20h8cva19irq8vn39i72j8iz40ivijhr-myShell.drv’ failed with exit code 1
error: build of ‘/nix/store/20h8cva19irq8vn39i72j8iz40ivijhr-myShell.drv’ failed
To make this code more idiomatic, I have the following suggestions:
callPackage
Use the pkgs.callPackage function. It will take care of passing the arguments that your derivation needs. This is why many files in NixPkgs look like { dependency, ...}: something. The first argument is the function you want to inject dependencies into and the second argument is an attribute set that you can use to pass some dependencies manually.
By using callPackage you do not need to import <nixpkgs> {}, so your code will be easier to use in new contexts <nixpkgs> can't be used and it will evaluate a bit faster because it has to evaluate the NixPkgs fix-point only once.
(Of course you have to import <nixpkgs> once to get started, but after that, there should be no need.)
with
In pkgs/aspell-with-dicts/default.nix you use a with keyword, which is ok, but in this case it does not really add value. I prefer to refer to variables explicitly, so prefer to read pkgs.something when it's used once or twice, or inherit (pkgs) something if it's use more often. This way the reader can easily determine where a variable comes from.
I do use it when experimenting with unfamiliar packages or functions, because maintenance is not an issue then.
pkgs/aspell-with-dicts/default.nix
Unless you expect that your instantiation of aspell is something you want to reuse, it's probably easier to just construct it where you use it.
If you do expect to reuse a specific configuration of a package, you might want to make it a first class package by constructing it in an overlay.
That's it. I think the most important point is avoiding <nixpkgs> and apart from that it's already pretty idiomatic.
I don't know what your mysterious foo is, but if it's open source, please consider upstreaming it into NixPkgs. Nix has a very welcoming community in my experience.
I'm writing a .nix expression to be used primarily by nix-shell. I'm not sure how to do that. Note this is not on NixOS, but I don't think that is very relevant.
The particular example I'm looking at is that I want to get this version-dependent name that looks like:
idea-ultimate = buildIdea rec {
name = "idea-ultimate-${version}";
version = "2017.2.2"; /* updated by script */
description = "Integrated Development Environment (IDE) by Jetbrains, requires paid license";
license = stdenv.lib.licenses.unfree;
src = fetchurl {
url = "https://download.jetbrains.com/idea/ideaIU-${version}-no-jdk.tar.gz";
sha256 = "b8eb9d612800cc896eb6b6fbefbf9f49d92d2350ae1c3c4598e5e12bf93be401"; /* updated by script */
};
wmClass = "jetbrains-idea";
update-channel = "IDEA_Release";
};
My nix expression is the following:
let
pkgs = import <nixpkgs> {};
stdenv = pkgs.stdenv;
# idea_name = assert pkgs.jetbrains.idea-ultimate.name != ""; pkgs.jetbrains.idea-ultimate.name;
in rec {
scalaEnv = stdenv.mkDerivation rec {
name = "scala-env";
builder = "./scala-build.sh";
shellHook = ''
alias cls=clear
'';
CLANG_PATH = pkgs.clang + "/bin/clang";
CLANGPP_PATH = pkgs.clang + "/bin/clang++";
# A bug in the nixpkgs openjdk (#29151) makes us resort to Zulu OpenJDK for IDEA:
# IDEA_JDK = pkgs.openjdk + "/lib/openjdk";
# PATH = "${pkgs.jetbrains.idea-ultimate}/${idea_name}/bin:$PATH";
IDEA_JDK = /usr/lib/jvm/zulu-8-amd64;
# IDEA_JDK = /opt/zulu8.23.0.3-jdk8.0.144-linux_x64;
# IDEA_JDK = /usr/lib/jvm/java-8-openjdk-amd64;
buildInputs = with pkgs; [
ammonite
boehmgc
clang
emacs
jetbrains.idea-ultimate
less
libunwind
openjdk
re2
sbt
stdenv
unzip
zlib
];
};
}
I have commented out setting PATH as it depends on getting idea_name in the let-clause. As an interesting side note, as is, this does not fail if I leave it uncommented but causes a very bizarre error when executing nix-shell about not being able to execute bash. I've also tried the more simple case of let idea_name = pkgs.jetbrains.idea-ultimate.name; but this fails later on when idea_name is used in setting PATH since idea_name ends up being undefined.
Update:
I began exploring with nix-instantiate, but the derivation of interest seems empty:
[nix-shell:/nix/store]$ nix-instantiate --eval --xml -E "((import <nixpkgs> {}).callPackage ./3hk87pqgl2qdqmskxbhy23cyr24q8g6s-nixpkgs-18.03pre114739.d0d905668c/nixpkgs/pkgs/applications/editors/jetbrains { }).idea-ultimate";
<?xml version='1.0' encoding='utf-8'?>
<expr>
<derivation>
<repeated />
</derivation>
</expr>
If your intent is to get idea-ultimate into nix-shell environment, then just include that package to buildInputs. I see it's already included, so it should already be present in your PATH.
BTW, you can extend your shellHook and export PATH and other variables rather from there, where you have full bash. Why would you do it from bash? Less copying. When you specify
IDEA_JDK = /usr/lib/jvm/zulu-8-amd64;
in Nix, the file /usr/lib/jvm/zulu-8-amd64 get's copied to nix store and IDEA_JDK is set to point to file in /nix/store. Was that your intent?
Regarding nix-instantiate:
$ nix-instantiate --eval -E 'with import <nixpkgs>{}; idea.pycharm-community.outPath'
"/nix/store/71jk0spr30rm4wsihjwbb1hcwwvzqr4k-pycharm-community-2017.1"
but you still have to remove doublequotes (https://gist.github.com/danbst/a9fc068ff26e31d88de9709965daa2bd)
Also, nitpick, assert pkgs.jetbrains.idea-ultimate.name != ""; can be dropped as it's impossible to have empty derivation name in Nix.
And another nitpick. You'll soon find very incovenient to launch IDE from shell every time. It seems a good idea to specify, that some package is used for development, but nix-shell doesn't work well for non-cli applications. Not to mention occasional problems with Nix GC and nix-shell. You'd better install IDE globally or per-user, it is better long-term solution.
[ADDENDUM]
You are looking for this (dev-environment.nix):
with import <nixpkgs> { };
buildEnv {
name = "my-super-dev-env";
paths = [
#emacs
nano
idea.pycharm-community
];
buildInputs = [ makeWrapper ];
postBuild = ''
for f in $(ls -d $out/bin/*); do
wrapProgram $f \
--set IDEA_JDK "/path/to/zulu-jdk" \
--set CLANG_PATH ... \
--set CLANCPP_PATH ...
done
'';
}
which you install using nix-env -if ./dev-environment.nix. It will wrap your programs with those env vars, without polluting your workspace (you can pollute it further using nix-shell with shell hook, if you want).
I have a single Python script called myscript.py and would like to package it up as a nix derivation with mkDerivation.
The only requirement is that my Python script has a run-time dependency, say, for the consul Python library (which itself depends on the requests and six Python libraries).
For example for myscript.py:
#!/usr/bin/env python3
import consul
print('hi')
How to do that?
I can't figure out how to pass mkDerivation a single script (its src seems to always want a directory, or fetchgit or similar), and also can't figure out how to make the dependency libraries available at runtime.
When you have a single Python file as your script, you don't need src in your mkDerivation and you also don't need to unpack any source code.
The default mkDerivation will try to unpack your source code; to prevent that, simply set dontUnpack = true.
myscript-package = pkgs.stdenv.mkDerivation {
name = "myscript";
propagatedBuildInputs = [
(pkgs.python36.withPackages (pythonPackages: with pythonPackages; [
consul
six
requests2
]))
];
dontUnpack = true;
installPhase = "install -Dm755 ${./myscript.py} $out/bin/myscript";
};
If your script is executable (which we ensure with install -m above) Nix will automatically replace your #!/usr/bin/env python3 line with one which invokes the right specific python interpreter (the one for python36 in the example above), and which does so in an environment that has the Python packages you've specifified in propagatedBuildInputs available.
If you use NixOS, you can then also put your package into environment.systemPackages, and myscript will be available in shells on that NixOS.
This helper function is really nice:
pkgs.writers.writePython3Bin "github-owner-repos" { libraries = [ pkgs.python3Packages.PyGithub ]; } ''
import os
import sys
from github import Github
if __name__ == '__main__':
gh = Github(os.environ['GITHUB_TOKEN'])
for repo in gh.get_user(login=sys.argv[1]).get_repos():
print(repo.ssh_url)
''
https://github.com/nixos/nixpkgs/blob/master/pkgs/build-support/writers/default.nix#L319