Let's say I need PostgreSQL 9.6.3 and Ruby 2.3.1 and various other tools. I can't find a tutorial that explains what I need to do.
From the Nix manual, I seem to need to write a Nix expression to install the needed dependencies, but I can't make the leap from:
{ stdenv, fetchurl, perl }:
stdenv.mkDerivation {
name = "hello-2.1.1";
builder = ./builder.sh;
src = fetchurl {
url = ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz;
md5 = "70c9ccf9fac07f762c24f2df2290784d";
};
inherit perl;
}
to the expression that will install the proper PostgreSQL and Ruby versions. It is absolutely unclear to me where to even put the file that installs PostgreSQL and Ruby, or how to run a single file in a given directory.
Can someone provide pointers to such tutorials, or point me in the right direction?
You can use nix-shell for this. It drops you into a shell configured to the given nix expression. Initially that expression could simply be along the lines of buildInputs = [ pkgs.ruby ]; and you can develop it from that. There are a number of helpful articles online written by nix users that give more examples of using nix-shell, like this one from garbas.si
You may also find it useful to get a better idea of how nix packages work. There's a separate nixpkgs manual that covers in greater detail using nix to create package expressions. A quick skim of the 3rd section should be useful to give a bit more understanding. There's also a chapter on using nix with ruby bundler that might be useful for you. Again there are articles that give more examples of its use, such as one from stesie.github.io.
If you need postgresql actually running in your environment nix won't manage that for you; its function is solely the building and management of packages, not their activation. You could simply activate postgres manually, use the nix-shell hook, or create some other integration with nix, but I think the most robust option is to make use of the Linux distribution that's built on top of Nix - NixOS. NixOS integrates with nix packages and manages the services provided by the packages. You could create a NixOS configuration with postgres active and your development environment present. This utility from github.com/chrisfarms may also be of interest.
Related
By "install packages" I mean to evaluate Nix build expressions (using nix-env, nix-shell -p, etc.) to build from source instead of using a substitute.
Also cross-posted to Unix& Linux because, as Charles Duffy pointed out, it is more on topic if it is about command-line tools or configuration. Still leaving this here because I assume forcing a package to always compile from source is possible by only using the Nix language, I just don't yet know how. (Or if it is in fact not possible, someone will point it out, and then this question does belong here.)
Either set the substitute option to false in nix.conf (the default is true) or use --option substitute false when invoking a Nix command.
nix-env --options substitute false -i hello
nix-shell --options substitute false -p hello
Might not be the droids you are looking for
As Robert Hensing (comment, chat), Henri Menke (comment), and Vladimír Čunát (comment) pointed out, this may not be the thing that you are really after.
To elaborate: I have been using the most basic Nix features confidently, but got to a point where I need to maintain and deploy a custom fork of a large application written in C, which is quite intimidating at the outset.
Tried to attack the problem the simplest way to just fetch my fork and re-build it with the new source, so I boiled it down to this question. Although, I suspect that the right direction for me is something along the lines of Nixpkgs/Create and debug packages in the NixOS Wiki.
Only re-build the package itself
Vladimír Čunát commented that "disabling substitutes makes you rebuild everything that's missing locally, even though I suspect that people asking such a question often only want to rebuild the specified package itself."
(This is probably achieved with nix-build or "just" overriding the original package but could be wrong. The latter is mention (maybe demonstrated even?) in the NixOS wiki article Development environment with nix-shell but haven't been able to read it thoroughly yet.)
Test for reproducibility
One might arrive to formulating this same question if they want to make sure that subsequent builds are deterministic. As Henri Menke comments, one should use nix-build --check for that.
The --check option is easy to miss; it's not documented in man nix-build or at nix-build in the Nix manual, but at nix-store --realize because (as man nix-build explains it):
nix-build is essentially a wrapper around nix-instantiate (to
translate a high-level Nix expression to a low-level store derivation)
and nix-store --realise (to build the store derivation) [and so] all
options not listed here are passed to nix-store --realise, except
for --arg and --attr / -A which are passed to nix-instantiate.
See detailed examples in the Nix manual at 18.1. Spot-Checking Build Determinism and the next section right after it.
The relevant parts for the substitute configuration option under the nix.conf section from the Nix manual:
Name
nix.conf — Nix configuration file
Description
Nix reads settings from two configuration files:
The system-wide configuration file sysconfdir/nix/nix.conf (i.e. /etc/nix/nix.conf on most systems), or $NIX_CONF_DIR/nix.conf if NIX_CONF_DIR is set.
The user configuration file $XDG_CONFIG_HOME/nix/nix.conf, or ~/.config/nix/nix.conf if XDG_CONFIG_HOME is not set.
You can override settings on the command line using the --option flag,
e.g. --option keep-outputs false.
The following settings are currently available:
[..]
substitute
If set to true (default), Nix will use binary substitutes if available. This option can be disabled to force building from source.
(Formerly known as use-binary-caches.)
Notes
Setting substitute to false (either with --options or in nix.conf) won't recompile the package if the command issue multiple times. That is, hello above would be compiled from source the first time, and then it will access the already present store path if the command issued again.
This is where it gets fuzzy: it is clear that no recompilation takes place because unless the package's Nix build expression doesn't change, the store output hash won't change either, making the next compilation output equivalent to the previous one, hence the action would be superfluous.
So if one would do some light hacking on a package, and just wanted to try it out locally (e.g., with nix-shell) then one would have to use -I nixpkgs=a/local/nixpkgs/dir to pick up those changes - and eventually do a recompilation? Or should one use nix-build?
See also question How to nix-build again a built store path?
Many CI providers give you a directory whose contents are retained across builds and you can use that as a cache. Everything that is stored elsewhere is lost. This means that any artefacts that are created during a nix-build that are placed in the nix store (/nix/store) are lost. I'm trying to figure out how to convince nix to prefer that other cache directory over the global /nix/store. However the documentation is a bit lacking.
What I've tried so far:
Add file:///the/path to substituters and then nix copy --to that path. However I discovered that nix only creates some metadata files in that directory and copies the actual derivation into /nix/store. That's not what I want.
Use local?root=/the/path instead of the file:// url (btw, this syntax is not documented anywhere, I only found it in a single github issue!). That made nix copy the whole derivation to that folder, but I couldn't figure out how to convince nix-build to actually consult that store during build.
Would it be possible to use something along the lines of nix run --store ~/my-nix nixpkgs.hello -c hello --greeting 'Hi everybody!'? The installation guide points to uses of --store for such a use case, as well as some sections in the manual: 1, 2.
An example of this can be found in nix run's tests.
There are also the environment variables NIX_STORE_DIR which might be of use? It's documented in nix-shell --help.
There's also several issues in the Nix repo, here's an interesting discussion.
Bazel use CROSSTOOL files to figure out how to builds things. This can be used to (for example) switch between GCC and Clang by setting --crosstool_top. The problem is that it's far from trivial to construct those files.
Does anyone know of any tools that can inspect a Linux installation and generate the needed crosstool files for any "common" compiler(s) that happens to be installed? Something that would be able to find and support any installed versions of Clang and GCC would be enought, any other compilers (icc, etc.) would be fantastic.
(Alternatively: are there any repo's with pre-constructed crosstool files for default installations of all the common compilers?)
Note
I've already found #bazel_tools//tools/cpp:cc_configure.bzl et al. but those seem to only generate configs for the default system compiler and I'm specifically looking for support for the non default compiler(s).
It's only a variation on cc_configure, but you can use environment variables to tweak the generation. Maybe using CC will be enough? If not, what else would you need (pull requests welcomed)?
There is no repo with premade crosstools yet, there will eventually be (maybe in the form of docker containers, we'll see) but currently there's not.
I've read the .travis.yml in the agda-stdlib project, while it's very different and complex from a simple library that was written in Agda purely (without those Haskell codes and Shell scripts).
I'm confused with the stdlib's .tarvis.yml. I've installed agda via cabal install, but the stdlib is trying to clone and compile Agda on Travis CI, and there're a lot of commands that seems to be irrealavent to building it.
Also, agda-stdlib seems to be available on Ubuntu's source. This could be the 3rd approach to install it.
Also, the stdlib doesn't have dependencies, but I have. I don't know how to add a dependency either.
Conclusion of my question:
In the 3 choices of installing agda listed above, which one should I choose?
How to add an dependency that let the agda compiler knows I'm actually using it?
The standard library is a bit of a special case: it evolves in lock-step with the development version of Agda. As such it is often the case that it cannot be compiled with a version of Agda readily available in your distribution of choice (e.g. because it uses syntax that was not available beforehand!) and it is forced to pull the latest version from github.
Installing Agda
If your library is compatible with a distributed version then it will be far simpler for you to simply pull it from the repositories via apt-get install agda.
Alternatively Scott Fleischman has a basic example on how to use a docker image to typecheck your development: https://github.com/scott-fleischman/agda-travis
Installing your dependencies
If your project relies on dependencies then you do need to install them. In practice it'll probably mean fetching a bunch of tarballs via wget, and having a ~/.agda/libraries pointing at their library files.
Cf. the manual on library management
So I've installed Nix on Arch linux and I'm able to run nix-env -i example, however how can I define a Nix configuration?
As I don't have any /nixos/configuration.nix file present.
Is this possible?
My goal here is to be able to define a configuration which I could then use something like nixos-rebuild switch to install and provision all the software.
I use NixOS, but I use /etc/nixos/configuration.nix to describe my system; I keep it fairly minimal and prefer not to install "user" software by editing configuration.nix.
So what I do instead is use my ~/.nixpkgs/config.nix (which I believe you also have an equivalent of even in a non-NixOS nix install? I've never actually used nix separately).
The basic structure I use is this:
{
packageOverrides = nixpkgs: with nixpkgs; rec {
mine = with pkgs; buildEnv {
name = "mine";
paths = [
# your packages here
];
};
};
}
buildEnv is a convenience function from nix that makes an "environment" package out of a bunch of others; installing the package mine depends on (and so installs) all of the things listed in paths, and also makes sure they get included in PATH and things like that.
Then I just use nix-env -riA nixos.mine to deploy changes I've made to my environment description (or to rebuild my environment following channel updates). The -r tells it to remove everything else other than mine from the new generation of my profile, which means I can (ab?)use nix-env -i some-package as a way of "temporarily" installing some-package, and if I don't decide I like it enough to actually record it in my config.nix it'll just get removed anyway next time I deploy.
You can certainly create your own configuration. For example, you can do something like this:
let
pkgs = import <nixpkgs> {};
in
{
packages = [
pkgs.vim
pkgs.gimp
];
}
That would produce a set containing an attribute called packages, containing a list of Nix packages. But, you'd need to develop a tool to build environments from this, which is part of what nix-env does. For example, the tool can use nix-env to determine what is already installed, compare that to the selected packages in the configuration, and then install/uninstall packages accordingly.
You don't have a /etc/nixos/configuration.nix because that's NixOS-specific. So while you can do as you asked, you'd have to roll your own solution.