According to https://nixos.org/nixpkgs/manual/#ssec-unpack-phase I can use a zip file as a 'source'.
with import <nixpkgs> {};
stdenv.mkDerivation {
name = "test-environment";
nativeBuildInputs = [ unzip ];
buildInputs = [ unzip ];
src = "./x.zip";
}
nix-build
these derivations will be built:
/nix/store/431i1riasgh2hjs5rd9azfh0ssjlg7qj-test-environment.drv
building '/nix/store/431i1riasgh2hjs5rd9azfh0ssjlg7qj-test-environment.drv'...
unpacking sources
unpacking source archive ./x.zip
unzip: cannot find or open ./x.zip, ./x.zip.zip or ./x.zip.ZIP.
unzip: cannot find or open ./x.zip, ./x.zip.zip or ./x.zip.ZIP.
do not know how to unpack source archive ./x.zip
builder for '/nix/store/431i1riasgh2hjs5rd9azfh0ssjlg7qj-test-environment.drv' failed with exit code 1
error: build of '/nix/store/431i1riasgh2hjs5rd9azfh0ssjlg7qj-test-environment.drv' failed
There is a x.zip file in my working directory. Why is it failing?
Update:
If I set unpackCmd = "ls -altr"; I see the following output:
total 16
drwxr-x--- 9 nobody nixbld 4096 May 26 12:39 ..
drwx------ 2 nixbld nixbld 4096 May 26 12:39 .
-rw-r--r-- 1 nixbld nixbld 5806 May 26 12:39 env-vars
Why isn't my zip file appearing in that context?
You don't want your Nix derivation to reference local filesystem elements! If anything hasn't been hashed and put into the store, your build is impure, and won't work in sandboxed mode.
Change it to src = ./x.zip, and the file will be hashed and added to the store, and src will be set to a /nix/store/.....-x.zip path.
with import <nixpkgs> {};
stdenv.mkDerivation {
name = "test-environment";
nativeBuildInputs = [ unzip ];
buildInputs = [ unzip ];
src = ./x.zip; ## NO QUOTES!
}
Related
I'm trying to write a derivation for a Go project that uses a Makefile to build its frontend with Yarn. As Yarn downloads files, I'm using a fixed derivation hash for networking.
However, Yarn attempts to write to its global package cache in /usr/local or /homeless-shelter, even though neither path exists.
To work around this issue, I tried to use makeWrapper to pass the --global-folder /tmp argument, but while the derivation runs it doesn't seem to be using it, as even --help is ignored.
How can I correctly wrap Yarn, or what other tool can I use to fix this?
cd frontend && yarn install --frozen-lockfile
yarn install v1.22.19
warning Skipping preferred cache folder "/homeless-shelter/.cache/yarn" because it is not writable.
warning Selected the next writable cache folder in the list, will be "/build/.yarn-cache-1000".
[1/4] Resolving packages...
[2/4] Fetching packages...
warning Cannot find a suitable global folder. Tried these: "/usr/local, /homeless-shelter/.yarn"
warning Cannot find a suitable global folder. Tried these: "/usr/local, /homeless-shelter/.yarn"
warning Cannot find a suitable global folder. Tried these: "/usr/local, /homeless-shelter/.yarn"
warning Cannot find a suitable global folder. Tried these: "/usr/local, /homeless-shelter/.yarn"
warning Cannot find a suitable global folder. Tried these: "/usr/local, /homeless-shelter/.yarn"
warning Cannot find a suitable global folder. Tried these: "/usr/local, /homeless-shelter/.yarn"
Done in 24.32s.
Error: ENOENT: no such file or directory, open '/homeless-shelter/.yarnrc'
make: *** [Makefile:218: frontend] Error 1
My config:
{ pkgs ? (
let
inherit (builtins) fetchTree fromJSON readFile;
inherit ((fromJSON (readFile ./flake.lock)).nodes) nixpkgs;
in
import (fetchTree nixpkgs.locked) {
overlays = [];
}
)
}:
with pkgs;
stdenv.mkDerivation rec {
pname = "someapp";
version = "v1.0";
src = fetchFromGitHub {
owner = "someowner";
repo = "someapp";
rev = "ffffffffffffffffffffffffffffffff";
sha256 = "sha256-fffffffffffffffffffffffffffffffff";
};
wrappedYarn = yarn.overrideAttrs (oldAttrs: {
buildInputs = oldAttrs.buildInputs or [] ++ [ makeBinaryWrapper ];
postInstall = oldAttrs.postInstall or "" + ''
wrapProgram $out/bin/yarn \
/*--append-flags "--global-folder /tmp/yarn --global-link /tmp/yarn2"*/
--append-flags "--help"
'';
});
buildInputs = [ go git wrappedYarn gcc gnumake ];
outputHashMode = "flat";
outputHashAlgo = "sha256";
outputHash = lib.fakeHash;
meta = with lib; {
description = "SomeApp";
homepage = "https://github.com/someowner/someapp";
platforms = platforms.linux;
};
}
Well, I'm trying to figure out how srcs works for genrule, I have the following:
genrule(
name = "flutter_build_android",
srcs = [
"//:genfiles"
],
outs = ["android/mobile.apk"],
cmd_bash = "ls -ltr && flutter build apk > $#",
tags = ["local"]
)
//:genfiles is a filegroup using glob:
filegroup(
name = "genfiles",
srcs = glob(["lib/**","assets/**", "pubspec.yaml"])
)
When executing my genrule what I expect is only files under ://genfiles label should be returned, but it is returning all folders under my root project:
What's happening is that the genrule is tagged with local, and local means
precludes the action or test from being remotely cached, remotely executed, or run inside the sandbox
https://bazel.build/reference/be/common-definitions#common.tags
Sandboxing is what prevents an action from seeing files that aren't declared as dependencies (i.e. in srcs here). Without the sandbox, the action sees everything.
$ tree
.
├── BUILD
├── file1
├── file2
├── file3
└── WORKSPACE
0 directories, 5 files
$ cat BUILD
genrule(
name = "gen_foo",
outs = ["foo"],
srcs = ["file1"],
cmd = "echo ----- ; ls ; echo ----- ; wc -l $< > $#",
# tags = ["local"],
)
$ bazel build foo
INFO: Analyzed target //:foo (5 packages loaded, 9 targets configured).
INFO: Found 1 target...
INFO: From Executing genrule //:gen_foo:
-----
bazel-out
external
file1
-----
Target //:foo up-to-date:
bazel-bin/foo
INFO: Elapsed time: 0.379s, Critical Path: 0.02s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
# edit BUILD to restore the local tag
$ cat BUILD
genrule(
name = "gen_foo",
outs = ["foo"],
srcs = ["file1"],
cmd = "echo ----- ; ls ; echo ----- ; wc -l $< > $#",
tags = ["local"],
)
$ bazel build foo
INFO: Analyzed target //:foo (5 packages loaded, 9 targets configured).
INFO: Found 1 target...
INFO: From Executing genrule //:gen_foo:
-----
BUILD
WORKSPACE
bazel-out
external
file1
file2
file3
local-spawn-runner.8714966150718292736
-----
Target //:foo up-to-date:
bazel-bin/foo
INFO: Elapsed time: 0.350s, Critical Path: 0.02s
INFO: 2 processes: 1 internal, 1 local.
INFO: Build completed successfully, 2 total actions
I haven't found any info about handling temporary files in Nix derivations.
I found $TMP and $TMPDIR env vars, but they both point just to /tmp, which is system global.
{
pkgs ? import <nixpkgs> {}
}:
pkgs.stdenv.mkDerivation {
pname = "show-tmp"
version = "0.1.0";
src = ./.;
configurePhase = ''
echo "tmp = $tmp; TMP = $TMP; TMPDIR = $TMPDIR"
'';
buildPhase = '':'';
installPhase = '':'';
}
Variable $tmp is not defined inside mkDerivation. I would expect such thing, because other derivation scope vars follow low case style such as $out.
The problem with /tmp is obvious - it is global directory.
I need to worry about collisions and cleaning.
My derivation-hook archives a big folder tree.
If you're on Linux, don't worry. The Nix sandbox will give your build its own empty /tmp. It is removed when your derivation is done.
On macOS, $TMP and $TMPDIR are taken care of but /tmp is a potential problem.
How Nix creates a private /tmp on Linux
macOS Darwin, where Nix was installed in early 2020:
nix-build --expr 'with import <nixpkgs> {}; runCommand "hi" {} "echo a > /tmp/a; ls -al /tmp; sleep 1;"'
ls -al /private/tmp/
...
-rw-r--r-- 1 nixbld1 wheel 2 May 19 12:49 a
...
with derivation
let
pkgs = import <nixpkgs> {};
in
with pkgs;
stdenv.mkDerivation {
name = "asdfasdf";
version = "0.1";
src = /home/srghma/opt/foxitsoftware/foxitreader/FoxitReader; # this is executeable file
dontUnpack = true; # not fu**** working
installPhase = ''
echo "not even executed"
'';
}
I have an error
nix-build tmp.nix
these derivations will be built:
/nix/store/x75gi70i1i57y8d3k4hhx0r3z5kjn6h6-asdfasdf.drv
building '/nix/store/x75gi70i1i57y8d3k4hhx0r3z5kjn6h6-asdfasdf.drv'...
unpacking sources
unpacking source archive /nix/store/3hnf69pky6mqaxv4jxly9fyywqpq6iml-FoxitReader
do not know how to unpack source archive /nix/store/3hnf69pky6mqaxv4jxly9fyywqpq6iml-FoxitReader
builder for '/nix/store/x75gi70i1i57y8d3k4hhx0r3z5kjn6h6-asdfasdf.drv' failed with exit code 1
error: build of '/nix/store/x75gi70i1i57y8d3k4hhx0r3z5kjn6h6-asdfasdf.drv' failed
why dontUnpack not working?
Update: created bug issue at nixpkgs https://github.com/NixOS/nixpkgs/issues/65434
Try this:
let
pkgs = import <nixpkgs> {};
in
with pkgs;
stdenv.mkDerivation {
name = "asdfasdf";
version = "0.1";
# Renamed to imply that 'src' functionality is not being used.
executable = /home/srghma/opt/foxitsoftware/foxitreader/FoxitReader; # this is executeable file
phases = [ "installPhase" ]; # Removes all phases except installPhase
installPhase = ''
mkdir -p $out/bin
cp ${executable} $out/bin
'';
}
#FRidh wrote
Many of the dont attributes were not yet implemented in the
current and past stable releases. So, if you intend to use it with the
current stable or older stables, use instead unpackPhase = ":".
For example in the following (which I assume is a nix expression):
(import <nixpkgs> {}).haskellPackages.ghcWithPackages (hpkgs: with hpkgs; [
lens
aeson
turtle
])
What does <nixpkgs> reference? I also see it used in other contexts for example:
nix-shell '<nixpkgs>' -A linuxPackages.kernel
<nixpkgs> is a Nix expression that is evaluated by looking at the Nix search path in the NIX_PATH environment variable and/or -I option.
It is described in more detail in the Nix manual.
Note that the Nix search path is impractical in many situations. You can only pass it from the outside, and it easily creates impurity. In my experience, problems are better solved with explicit argument passing or the functions relating to fix-points like callPackage and the overlay system.
As an example, NixOS has only one extra search path parameter, and it is only read once in nixos/default.nix if no explicit configuration is given. This way, you have the flexibility to provide your own configuration, which is why you (nix-build) and hydra can confidently build the NixOS VM tests, bootable images, docker images, etc.
From the Nix manual, 5.2.1. Values, section "Simple Values":
Paths can also be specified between angle brackets, e.g. .
This means that the directories listed in the environment variable
NIX_PATH will be searched for the given file or directory name.
From the NixOS manual, 7.2. Common Environment Variables, section NIX_PATH:
A colon-separated list of directories used to look up Nix expressions
enclosed in angle brackets (i.e., ). For instance, the value
/home/eelco/Dev:/etc/nixos
will cause Nix to look for paths relative to /home/eelco/Dev and
/etc/nixos, in that order. It is also possible to match paths
against a prefix. For example, the value
nixpkgs=/home/eelco/Dev/nixpkgs-branch:/etc/nixos
will cause Nix to search for <nixpkgs/path> in
/home/eelco/Dev/nixpkgs-branch/path and /etc/nixos/nixpkgs/path.
If a path in the Nix search path starts with http:// or https://,
it is interpreted as the URL of a tarball that will be downloaded and
unpacked to a temporary location. The tarball must consist of a single
top-level directory. For example, setting NIX_PATH to
nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/nixos-15.09.tar.gz
tells Nix to download the latest revision in the Nixpkgs/NixOS 15.09
channel.
A following shorthand can be used to refer to the official channels:
nixpkgs=channel:nixos-15.09
The search path can be extended using the -I option, which takes
precedence over NIX_PATH.
Examples
1. with import <nixpkgs> {}; /* rest of the expression */
In my case, <nixpkgs> is /nix/var/nix/profiles/per-user/root/channels/nixos:
$ echo $NIX_PATH
# VVVVVVV
/home/a_user/.nix-defexpr/channels:nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos:nixos-config=/etc/nixos/configuration.nix:/nix/var/nix/profiles/per-user/root/channels
# ^^^^^^^
Because <nixpkgs> evaluates to "a directory, the file default.nix in that directory is loaded" by import. (Nix manual, 15.4.1. Advanced Attributes, section import path, builtins.import path)
$ ll /nix/var/nix/profiles/per-user/root/channels/nixos
lrwxrwxrwx 1 root root 80 Dec 31 1969 /nix/var/nix/profiles/per-user/root/channels/nixos -> /nix/store/ywlfq2ns4
a3fzb2ap74lvahmrg1p0lmk-nixos-19.03.172231.7b36963e7a7/nixos/
$ ll $(readlink -f /nix/var/nix/profiles/per-user/root/channels/nixos)
total 3308
dr-xr-xr-x 8 root root 4096 Dec 31 1969 ./
dr-xr-xr-x 4 root root 4096 Dec 31 1969 ../
# (...)
dr-xr-xr-x 7 root root 4096 Dec 31 1969 nixos/
dr-xr-xr-x 17 root root 4096 Dec 31 1969 pkgs/
-r--r--r-- 1 root root 1097 Dec 31 1969 COPYING
-r--r--r-- 1 root root 968 Dec 31 ---> default.nix <---
# (...)
If my understanding is correct, after the import the provided Nix expression is evaluated with an empty attribute set ({}). The result is an attribute list, and the with expression will include all its containing attributes in the local lexical scope.
2. nix repl '<nixpkgs/nixos>'
Example taken from the NixOS manual, 5.3. Modularity, showing the active NixOS configuration settings in the repl.
# On NixOS 19.03
$ nix repl
Welcome to Nix version 2.2.2. Type :? for help.
nix-repl> <nixpkgs>
/nix/var/nix/profiles/per-user/root/channels/nixos
nix-repl> <nixpkgs/nixos>
/nix/var/nix/profiles/per-user/root/channels/nixos/nixos
Load the systems NixOS configuration on the repl:
nix-repl> :l <nixpkgs/nixos>
Added 6 variables.
Load all Nix expressions from Nixpkgs on the repl:
nix-repl> :l <nixpkgs>
Added 10089 variables.
Or loading them directly into the repl:
$ nix repl '<nixpkgs>'
Welcome to Nix version 2.2.2. Type :? for help.
Loading '<nixpkgs>'...
Added 10089 variables.
$ nix repl '<nixpkgs/nixos>'
Welcome to Nix version 2.2.2. Type :? for help.
Loading '<nixpkgs/nixos>'...
Added 6 variables.
Cheatsheet:
nix-repl> :help
The following commands are available:
<expr> Evaluate and print expression
<x> = <expr> Bind expression to variable
:a <expr> Add attributes from resulting set to scope
:b <expr> Build derivation
:i <expr> Build derivation, then install result into current profile
:l <path> Load Nix expression and add it to scope
:p <expr> Evaluate and print expression recursively
:q Exit nix-repl
:r Reload all files
:s <expr> Build dependencies of derivation, then start nix-shell
:t <expr> Describe result of evaluation
:u <expr> Build derivation, then start nix-shell
Because of the <nixpkgs/path> convention (where path equals nixos), the angle expression will evaluate to /nix/var/nix/profiles/per-user/root/channels/nixos/nixos. The above ll output also shows a nixos folder above default.nix, and inside there is indeed another default.nix that will get evaluated by nix repl:
$ ll $(readlink -f /nix/var/nix/profiles/per-user/root/channels/nixos/nixos)
total 72
dr-xr-xr-x 7 root root 4096 Dec 31 1969 ./
dr-xr-xr-x 8 root root 4096 Dec 31 1969 ../
-r--r--r-- 1 root root 886 Dec 31 ---> default.nix <---
-r--r--r-- 1 root root 197 Dec 31 1969 README
-r--r--r-- 1 root root 6074 Dec 31 1969 release-combined.nix
-r--r--r-- 1 root root 9251 Dec 31 1969 release.nix
-r--r--r-- 1 root root 2038 Dec 31 1969 release-small.nix
Miscellaneous
Issue #338 is still open to improve the Nix manual, documenting the angle syntax in their own section (UPDATE: Fixed and closed, 10/23/2019).
Nix Pills sections (e.g., 15.5. Conclusion and Chapter 16. Nixpkgs Parameters) consistently call the angle syntax "angular brackets" syntax.
In his answer, Robert Hensing warns against using <nixpkgs> everywhere, and an issue #1161 is still open on just that topic. (Excerpt: "We have an unofficial guideline that nobody should use angle brackets in nixpkgs, but why not make it into a hard requirement?")
One can evaluate the value using nix repl:
nix repl
Welcome to Nix version 2.1.3. Type :? for help.
nix-repl> <nixpkgs>
/nix/var/nix/profiles/per-user/root/channels/nixos