I am new to nixos, this is my understanding about configurations
Configuration files created by installer
/etc/nixos/configuration.nix :: The central point of system description used by nixos-rebuild
/etc/nixos/hardware-configuration.nix :: to be include in above configuration.nix
Configuration files for packages
<package>.nix on nixpkgs github :: configuration for each module (options are searchable on nixos package page)
These are what I do not fully understand
defatult.nix (any where in filesystem) :: for nix-shell like .bashrc
~/.nixpkgs/config.nix :: nix-env overrided configuration for each users
~/.config/<various>.nix :: ?? no idea
Am I understand it right?
Where can I find more information on these configuration files?
You don't call all of these files configuration files. E.g. the <package>.nix files are rather called derivations. What all these files share is the language in which they are written.
/etc/nixos/configuration.nix is indeed where you configure your system and ~/.nixpkgs/config.nix where you configure nix-env.
default.nix doesn't mean anything in particular except that this is the default file that it selected by the commands nix-build and nix-shell when you give them a directory as argument instead of a specific file. Note e.g. that the nixpkgs collection (on GitHub like you noticed already) contains a lot of such default.nix files.
To understand all of this better I advise you to read Nix-pills (that's a long series but it's worth it) and of course the NixOS, Nix and nixpkgs manuals.
Related
By "install packages" I mean to evaluate Nix build expressions (using nix-env, nix-shell -p, etc.) to build from source instead of using a substitute.
Also cross-posted to Unix& Linux because, as Charles Duffy pointed out, it is more on topic if it is about command-line tools or configuration. Still leaving this here because I assume forcing a package to always compile from source is possible by only using the Nix language, I just don't yet know how. (Or if it is in fact not possible, someone will point it out, and then this question does belong here.)
Either set the substitute option to false in nix.conf (the default is true) or use --option substitute false when invoking a Nix command.
nix-env --options substitute false -i hello
nix-shell --options substitute false -p hello
Might not be the droids you are looking for
As Robert Hensing (comment, chat), Henri Menke (comment), and Vladimír Čunát (comment) pointed out, this may not be the thing that you are really after.
To elaborate: I have been using the most basic Nix features confidently, but got to a point where I need to maintain and deploy a custom fork of a large application written in C, which is quite intimidating at the outset.
Tried to attack the problem the simplest way to just fetch my fork and re-build it with the new source, so I boiled it down to this question. Although, I suspect that the right direction for me is something along the lines of Nixpkgs/Create and debug packages in the NixOS Wiki.
Only re-build the package itself
Vladimír Čunát commented that "disabling substitutes makes you rebuild everything that's missing locally, even though I suspect that people asking such a question often only want to rebuild the specified package itself."
(This is probably achieved with nix-build or "just" overriding the original package but could be wrong. The latter is mention (maybe demonstrated even?) in the NixOS wiki article Development environment with nix-shell but haven't been able to read it thoroughly yet.)
Test for reproducibility
One might arrive to formulating this same question if they want to make sure that subsequent builds are deterministic. As Henri Menke comments, one should use nix-build --check for that.
The --check option is easy to miss; it's not documented in man nix-build or at nix-build in the Nix manual, but at nix-store --realize because (as man nix-build explains it):
nix-build is essentially a wrapper around nix-instantiate (to
translate a high-level Nix expression to a low-level store derivation)
and nix-store --realise (to build the store derivation) [and so] all
options not listed here are passed to nix-store --realise, except
for --arg and --attr / -A which are passed to nix-instantiate.
See detailed examples in the Nix manual at 18.1. Spot-Checking Build Determinism and the next section right after it.
The relevant parts for the substitute configuration option under the nix.conf section from the Nix manual:
Name
nix.conf — Nix configuration file
Description
Nix reads settings from two configuration files:
The system-wide configuration file sysconfdir/nix/nix.conf (i.e. /etc/nix/nix.conf on most systems), or $NIX_CONF_DIR/nix.conf if NIX_CONF_DIR is set.
The user configuration file $XDG_CONFIG_HOME/nix/nix.conf, or ~/.config/nix/nix.conf if XDG_CONFIG_HOME is not set.
You can override settings on the command line using the --option flag,
e.g. --option keep-outputs false.
The following settings are currently available:
[..]
substitute
If set to true (default), Nix will use binary substitutes if available. This option can be disabled to force building from source.
(Formerly known as use-binary-caches.)
Notes
Setting substitute to false (either with --options or in nix.conf) won't recompile the package if the command issue multiple times. That is, hello above would be compiled from source the first time, and then it will access the already present store path if the command issued again.
This is where it gets fuzzy: it is clear that no recompilation takes place because unless the package's Nix build expression doesn't change, the store output hash won't change either, making the next compilation output equivalent to the previous one, hence the action would be superfluous.
So if one would do some light hacking on a package, and just wanted to try it out locally (e.g., with nix-shell) then one would have to use -I nixpkgs=a/local/nixpkgs/dir to pick up those changes - and eventually do a recompilation? Or should one use nix-build?
See also question How to nix-build again a built store path?
I am attempting to use Bazel to compile a dhall program based on dhall-kubernetes to generate a Kubernetes YAML file.
The basic dhall compile without dhall-kubernetes using a simple bazel macro works ok.
I have made an example using dhall's dependency resolution to download dhall-kubernetes - see here. This also works but is very slow (I think because dhall downloads each remote file separately), and introduces a network dependency to the bazel rule execution, which I would prefer to avoid.
My preferred approach is to use Bazel to download an archive release version of dhall-kubernetes, then have the rule access it locally (see here). My solution requires a relative path in Prelude.dhall and package.dhall for the examples/k8s package to reference dhall-kubernetes. While it works, I am concerned that this is subverting the Bazel sandbox by requiring special knowledge of the folder structure used internally by Bazel. Is there a better way?
Prelude.dhall:
../../external/dhall-kubernetes/1.17/Prelude.dhall
WORKSPACE:
workspace(name = "dhall")
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
DHALL_KUBERNETES_VERSION = "4.0.0"
http_archive(
name = "dhall-kubernetes",
sha256 = "0bc2b5d2735ca60ae26d388640a4790bd945abf326da52f7f28a66159e56220d",
url = "https://github.com/dhall-lang/dhall-kubernetes/archive/v%s.zip" % DHALL_KUBERNETES_VERSION,
strip_prefix = "dhall-kubernetes-4.0.0",
build_file = "#//:BUILD.dhall-kubernetes",
)
BUILD.dhall-kubernetes:
package(default_visibility=['//visibility:public'])
filegroup(
name = "dhall-k8s-1.17",
srcs = glob([
"1.17/**/*",
]),
)
examples/k8s/BUILD:
package(default_visibility = ["//visibility:public"])
genrule(
name = "special_ingress",
srcs = ["ingress.dhall",
"Prelude.dhall",
"package.dhall",
"#dhall-kubernetes//:dhall-k8s-1.17"
],
outs = ["ingress.yaml"],
cmd = "dhall-to-yaml --file $(location ingress.dhall) > $#",
visibility = [
"//visibility:public"
]
)
There is a way to instrument dhall to do "offline" builds, meaning that the package manager fetches all Dhall dependencies instead of Dhall fetching them.
In fact, I implemented something exactly this for Nixpkgs, which you may be able to translate to Bazel:
Add Nixpkgs support for Dhall
High-level explanation
The basic trick is to take advantage of a feature of Dhall's import system, which is that if a package protected by a semantic integrity check (i.e. a "semantic hash") is cached then Dhall will use the cache instead of fetching the package. You can build upon this trick to have the package manager bypass Dhall's remote imports by injecting dependencies in this way.
You can find the Nix-related logic for this here:
Nix function for building a Dhall package
... but I will try to explain how it works in a package-manager-independent way.
Package structure
First, the final product of a Dhall "package" built using Nix is a directory with the following structure:
$ nix-build --attr 'dhallPackages.Prelude'
…
$ tree -a ./result
./result
├── .cache
│ └── dhall
│ └── 122026b0ef498663d269e4dc6a82b0ee289ec565d683ef4c00d0ebdd25333a5a3c98
└── binary.dhall
2 directories, 2 files
The contents of this directory are:
./cache/dhall/1220XXX…XXX
A valid cache directory for Dhall containing a single build product: the binary encoding of the interpreted Dhall expression.
You can create such a binary file using dhall encode and you can compute the file name by replacing the XXX…XXX above with the sha256 encoding of the expression, which you can obtain using the dhall hash command.
./binary.dhall
A convenient Dhall file containing the expression missing sha256:XXX…XXX. Interpreting this expression only succeeds if the expression we built matching the hash sha256:XXX…XXX is already cached.
The file is called binary.dhall because this is the Dhall equivalent of a "binary" package distribution, meaning that the import can only be obtained from a binary cache and cannot be fetched and interpreted from source.
Optional: ./source.dhall
This is a file containing a fully αβ-normalized expression equivalent to the expression that was cached. By default, this should be omitted for all packages except perhaps the top-level package, since it contains the same expression that is stored inside of ./cache/1220XXX…XXX, albeit less efficiently (since the binary encoding is more compact)
This file is called ./source.dhall because this is the Dhall equivalent of a "source" package distribution, which contains valid source code to produce the same result.
User interface
The function for building a package takes four arguments:
The package name
This is not material to the build. It's just to name things since every Nix package has to have a human-readable name.
The dependencies for the build
Each of these dependencies is a build product that produces a directory tree just like the one I described above (i.e. a ./cache directory, a ./binary.dhall file, and an optional ./source.dhall file)
A Dhall expression
This is can be arbitrary Dhall source code, with only one caveat: all remote imports transitively referenced by the expression must be protected by integrity checks AND those imports must match one of the dependencies to this Dhall package (so that the import can be satisfied via the cache instead of the Dhall runtime fetching the URL)
A boolean option specifying whether to keep the ./source.dhall file, which is False by default
Implementation
The way that the Dhall package builder works is:
First, build the Haskell Dhall package with the -f-with-http flag
This flag compiles out support for HTTP remote imports, that way if the user forgets to supply a dependency for a remote import they will get an error message saying Import resolution is disabled
We'll be using this executable for all of the subsequent steps
Create a cache directory within the current working directory named .cache/dhall
... and populate the cache directory with the binary files stored inside each dependency's ./cache/ directory
Configure the interpreter to use the cache directory we created
... by setting XDG_CACHE_HOME to point to the .cache directory we just created in our current working directory
Interpret and α-normalize the Dhall source code for our package
... using the dhall --alpha command. Save the result to $out/source.dhall where $out is the directory that will store the final build product
Obtain the expression's hash
... using the dhall hash command. We will need this hash for the following two steps.
Create the corresponding binary cache file
... using the dhall encode command and save the file to $out/cache/dhall/1220${HASH}
Create the ./binary.dhall file
... by just writing out a text file to $out/binary.dhall containing missing sha256:${HASH}
Optional: Delete the ./source.dhall file
... if the user did not request to keep the file. Omitting this file by default helps conserve space within the package store by not storing the same expression twice (as both a binary file and source code).
Packaging conventions
Once you have this function, there are a couple of conventions that can help simplify doing things "in the large"
By default, a package should build a project's ./package.dhall file
Make it easy to override the package version
Make it easy to override the file built within the package
In other words, if a user prefers to import individual files like https://prelude.dhall-lang.org/List/map instead of the top-level ./package.dhall file there should be a way for them to specify a dependency like Prelude.override { file = "./List/map"; } to obtain a package that builds and caches that individual file.
Conclusion
I hope that helps! If you have more questions about how to do this you can either ask them here or you can also discuss more on our Discourse forum, especially on the thread where this idiom first originated:
Dhall Discourse - Offline use of Prelude
Many CI providers give you a directory whose contents are retained across builds and you can use that as a cache. Everything that is stored elsewhere is lost. This means that any artefacts that are created during a nix-build that are placed in the nix store (/nix/store) are lost. I'm trying to figure out how to convince nix to prefer that other cache directory over the global /nix/store. However the documentation is a bit lacking.
What I've tried so far:
Add file:///the/path to substituters and then nix copy --to that path. However I discovered that nix only creates some metadata files in that directory and copies the actual derivation into /nix/store. That's not what I want.
Use local?root=/the/path instead of the file:// url (btw, this syntax is not documented anywhere, I only found it in a single github issue!). That made nix copy the whole derivation to that folder, but I couldn't figure out how to convince nix-build to actually consult that store during build.
Would it be possible to use something along the lines of nix run --store ~/my-nix nixpkgs.hello -c hello --greeting 'Hi everybody!'? The installation guide points to uses of --store for such a use case, as well as some sections in the manual: 1, 2.
An example of this can be found in nix run's tests.
There are also the environment variables NIX_STORE_DIR which might be of use? It's documented in nix-shell --help.
There's also several issues in the Nix repo, here's an interesting discussion.
One of the first types of Nix expression one encounters when learning how to use the Nix package manager is default.nix; on the wonderful NixOS IRC channel I learned of the existence of shell.nix and release.nix as well.
I got the impression that - roughly -default.nix is to be used with nix-build for simply building the package, shell.nix is used with nix-shell to create an interactive environment with the package and release.nix is used with nixops in deploying the package.
Since this is likely incomplete and partially incorrect, and since this does not seem to be clearly documented, I would like a clear and precise explanation of these sorts of "standard files"; in particular, for each of these file types (as well as any other standard files I am missing), I would like to know:
What are the typical use cases for such a file? What should they not be used for?
How is this file type structured typically? What are the minimal requirements for it?
Could you show a paradigm example of such a file within its use context, i.e. with use instructions and including lines of code needed to use it in the shell or another Nix expression?
As an additional bonus question, I want to know which - if any - of these standard files should be used when installing a package into a NixOS module? How would that be done?
First of all, default.nix and shell.nix have special meanings in Nix tool, but release.nix is a convenience one.
Next, default.nix is used as a default file when running nix-build, and shell.nix is used as a default file when running nix-shell. Just like you said.
Next, default.nix isn't used only for nix-build. For example, <nixpkgs/lib/default.nix> is used as aggregator for functions, and doesn't contain derivations. So not every default.nix is supposed to be "built" (but if default.nix is an attribute set of derivations, it will be buildable, and nix-build will build all of them).
Next, nix-shell will use default.nix if no shell.nix is found.
Next, default.nix is used as default file when importing directory. So if you write x = import ./some/directory;, then ./some/directory/default.nix will be imported. That actually should explain why "nix-build ." uses default.nix.
And finally, there are two common formats for derivations in default.nix: derivation, and callPackage derivation. You can't nix-build the latter. Almost any package in nixpkgs is written in this style, see hello. But you can
nix-build -E 'with import <nixpkgs> { }; callPackage ./path/to/default.nix { }'
as a workaround. nix-shell also supports this -E argument.
As said by #danbst only default.nix and shell.nix have special meanings for the nix tooling, out of that there is no real standard and everybody is free to use what fits most their needs.
That said, it doesn't mean you cannot set your own set of rules, personally for a single derivation project I like to arrange nix files in the following manner:
default.nix: Use callpackage to import derivation.nix.
derivation.nix: nixpkgs style derivation file.
shell.nix: nix-shell file.
module.nix: NixOS module file, import default.nix.
test.nix: NixOS test file.
release.nix: Hydra jobset declaration.
We had a talk about this topic at the Tokyo NixOS meetup, an example of a such code organization can be found here.
in Java/Groovy, afaik, a package has to be defined in the corresponding folder. This results in all class files which are stored in /a/b/c start with the line package a.b.c. Is this still necessary? With regards to convention over configuration, this isn't DRY...
What kind of problems would arise when this package definition would be optional`?
While it is conventional for the directory structure to match the package structure, and certain problems arise if they don't match, it is in fact not a requirement that they match. This is also true of Java (though a lot of folks don't realize that).
Below is an example which demonstrates this.
groovydemo $ mkdir classes
groovydemo $
groovydemo $ cat src/groovy/com/demo/SomeClass.groovy
package com.somethingotherthandemo
class SomeClass {}
groovydemo $
groovydemo $ groovyc -d classes/ src/groovy/com/demo/SomeClass.groovy
groovydemo $ find classes -type f
classes/com/somethingotherthandemo/SomeClass.class
The reasons for using packages in Groovy (and Grails) are some of the same reason why they are used in Java.
Packages serve to organize classes into logical namespaces, typically by grouping collaborating classes together.
It helps avoid naming conflicts with other classes (either Java or Groovy).
In any non-trival system where you have hundreds or thousands of classes, packages provide a very useful mechanism for organization and structure.
I think what you're saying is that the package name is implied by the directory the class is in, so why do you need to state it explicity? This is only true in some cases (like Grails) where there's a convention that establishes the root of the source files (e.g. src/groovy).
But imagine I'm writing a Groovy app and have a file at /a/b/c/D.groovy, how can we tell if the root of the source files is /a and thus the package name is b.c or the root of the source files is /a/b and therefore the package name is just c? As far as I can see, we can't, so the package name needs to be stated in the source file explicitly.