I am trying to use specific versions of two packages:
ruby 3.0.3
postgresql 14.0.4
After hours of googling and experimenting, I came up with such a shell.nix file:
let
# stable-22.05
pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/040c6d8374d090f46ab0e99f1f7c27a4529ecffd.tar.gz") {};
# inludes postgres 14.0.4
pgpkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/d86bcbb415938888e7f606d55c52689aec127f43.tar.gz") {};
# includes ruby 3.0.3
rubypkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/d1c3fea7ecbed758168787fe4e4a3157e52bc808.tar.gz") {};
in
pkgs.mkShell {
buildInputs = [
pgpkgs.postgresql_14
rubypkgs.ruby_3_0
];
}
(used https://lazamar.co.uk/nix-versions to find a commit)
I am wondering if there is a better way for handling my requirement?
You should not need to define or use pkgs. You can just get the mkShell function from pgpkgs (which seems like the better option since it's slightly newer than rubypkgs.
For the long term, I recommend fixing your project to work with the version of Ruby in commit d86bcbb and then just use that commit for everything, which is simpler than using multiple commits.
Another thing you should know is that you can use git to clone nixpkgs to somewhere on your computer, then use git checkout d86bcbb to check out the commit you are interested in. Then set your NIX_PATH environment variable to make it so that nixpkgs refers to that commit:
export NIX_PATH=nixpkgs=/path/to/your/nixpkgs
Now you can use import <nixpkgs> in your expressions, and if you want to try a different commit you can just use the regular git commands for changing commits, without needing to use a text editor or deal with large hashes. Also, the command nix-shell -p ruby -p postgresql uses NIX_PATH to find its derivations.
Related
The question may not be precise, but this is what I am trying to achieve:
The puppeteer-core NPM package requires a headless browser at runtime, and I found that I could use it with the latest Chromium in Nixpkgs. My current workflow:
I have overridden the node2nix-created default.nix with override.nix by adding Chromium to buildInputs:
{pkgs ? import <nixpkgs> {
inherit system;
}, system ? builtins.currentSystem}:
let
nodePackages = import ./default.nix {
inherit pkgs system;
};
in
nodePackages // {
shell = nodePackages.shell.override {
buildInputs = [ pkgs.chromium ];
};
}
Issue nix-shell override.nix -A shell.
export CHROMIUM_PATH=$(which chromium)
Setting up Chromium in the javascript files via process.env.CHROMIUM_PATH.
This works well, and the project can be shared easily with others by creating a tarball, but they would have to set up the environment manually as well, and I also don't like Chromium being included in the build dependencies. I could have just installed it globally with Nix, but the rest of the manual steps would still be there.
I read the manuals, and getting comfortable with the language, but haven't built a derivation or a package yet, therefore I'm sure I missing something obvious.
Some of the discussions I found on the topic:
https://nixos.org/nixos/nix-pills/automatic-runtime-dependencies.html
Runtime-only dependencies #1080
https://discourse.nixos.org/t/handling-dynamic-runtime-dependencies/323
I'm in the process of migrating entire CloudFormation stacks to Troposphere, including Lambda and Lambda-reliant CFN Custom Resources.
One of my goals is to circumvent the creation of template files altogether, making the Python code the sole "source of truth" (i.e without template files that are created and therefore can be edited, causing config drift).
This requires the ability to:
Pass a file-like object to the SAM builder (instead of a file-name)
Calling the AWS SAM builder from Python and not the CLI
My first naive idea was that I would be able to import a few modules from aws-sam-cli put a wrapper for io.StringIO around it (to hold the template as file-like object) and presto! Then I looked at the source code for sam build and all hope left me:
I may not be able to use Docker/containers for building, as I it will map the build environment, including template files.
AWS SAM CLI is not designed to have a purely callable set of library functions, similar to boto3. Close, but not quite.
Here is the core of the Python source
with BuildContext(template,
base_dir,
build_dir,
clean=clean,
manifest_path=manifest_path,
use_container=use_container,
parameter_overrides=parameter_overrides,
docker_network=docker_network,
skip_pull_image=skip_pull_image,
mode=mode) as ctx:
builder = ApplicationBuilder(ctx.function_provider,
ctx.build_dir,
ctx.base_dir,
manifest_path_override=ctx.manifest_path_override,
container_manager=ctx.container_manager,
mode=ctx.mode
)
try:
artifacts = builder.build()
modified_template = builder.update_template(ctx.template_dict,
ctx.original_template_path,
artifacts)
move_template(ctx.original_template_path,
ctx.output_template_path,
modified_template)
click.secho("\nBuild Succeeded", fg="green")
msg = gen_success_msg(os.path.relpath(ctx.build_dir),
os.path.relpath(ctx.output_template_path),
os.path.abspath(ctx.build_dir) == os.path.abspath(DEFAULT_BUILD_DIR))
click.secho(msg, fg="yellow")
This relies on a number of imports from a aws-sam-cli internal library with the build focused ones being
from samcli.commands.build.build_context import BuildContext
from samcli.lib.build.app_builder import ApplicationBuilder, BuildError, UnsupportedBuilderLibraryVersionError, ContainerBuildNotSupported
from samcli.lib.build.workflow_config import UnsupportedRuntimeException
It's clear that this means it's not as simple as creating something like a boto3 client and away I go! It looks more like I'd have to fork the whole thing and throw out nearly everything to be left with the build command, context and environment.
Interestingly enough, sam package and sam deploy, according to the docs, are merely aliases for aws cloudformation package and aws cloudformation deploy, meaning those can be used in boto3!
Has somebody possibly already solved this issue? I've googled and searched here, but haven't found anything.
I use PyCharm and the AWS Toolkit which if great for development and debugging and from there I can run SAM builds, but it's "hidden" in the PyCharm plugins - which are written in Kotlin!
My current work-around is to create the CFN templates as temp files and pass them to the CLI commands which are called from Python - an approach I've always disliked.
I may put in a feature request with the aws-sam-cli team and see what they say, unless one of them reads this.
I've managed to launch sam local start-api from a python3 script.
Firstly, pip3 install aws-sam-cli
Then the individual command can be imported and run.
import sys
from samcli.commands.local.start_api.cli import cli
sys.exit(cli())
... provided there's a template.yaml in the current directory.
What I haven't (yet) managed to do is influence the command-line arguments that cli() would receive, so that I could tell it which -t template to use.
Edit
Looking at the way aws-sam-cli integration tests work it seems that they actually kick off a process to run the CLI. So they don't actually pass a parameter to the cli() call at all :-(
For example:
class TestSamPython36HelloWorldIntegration(InvokeIntegBase):
template = Path("template.yml")
def test_invoke_returncode_is_zero(self):
command_list = self.get_command_list(
"HelloWorldServerlessFunction", template_path=self.template_path, event_path=self.event_path
)
process = Popen(command_list, stdout=PIPE)
return_code = process.wait()
self.assertEquals(return_code, 0)
.... etc
from https://github.com/awslabs/aws-sam-cli/blob/a83aa9e620ff679ca740496a3f1ff4872b88894a/tests/integration/local/invoke/test_integrations_cli.py
See also start_api_integ_base.py in the same repo.
I think on the whole this is to be expected because the whole thing is implemented in terms of the click command-line application framework. Unfortunately.
See for example http://click.palletsprojects.com/en/7.x/testing/ which says "The CliRunner.invoke() method runs the command line script in isolation ..." -- my emphasis.
I am using following python script to run sam cli commands. This should work for you too.
import json
import sys
import os
try:
LAMBDA_S3_BUCKET="s3-bucket-name-in-same-region"
AWS_REGION="us-east-1"
API_NAME = "YourAPIName"
BASE_PATH="/path/to/your/project/code/dir"
STACK_NAME="YourCloudFormationStackName"
BUILD_DIR="%s/%s" % (BASE_PATH, "build_artifact")
if not os.path.exists(BUILD_DIR):
os.mkdir(BUILD_DIR)
os.system("cd %s && sam build --template template.yaml --build-dir %s" % (BASE_PATH, BUILD_DIR))
os.system("cd %s && sam package --template-file %s/template.yaml --output-template-file packaged.yaml --s3-bucket %s" %(BASE_PATH, BUILD_DIR, LAMBDA_S3_BUCKET))
os.system("cd %s && sam deploy --template-file packaged.yaml --stack-name %s --capabilities CAPABILITY_IAM --region %s" %(BASE_PATH, STACK_NAME, AWS_REGION))
except Exception as e:
print(e.message)
exit(1)
So I've installed Nix on Arch linux and I'm able to run nix-env -i example, however how can I define a Nix configuration?
As I don't have any /nixos/configuration.nix file present.
Is this possible?
My goal here is to be able to define a configuration which I could then use something like nixos-rebuild switch to install and provision all the software.
I use NixOS, but I use /etc/nixos/configuration.nix to describe my system; I keep it fairly minimal and prefer not to install "user" software by editing configuration.nix.
So what I do instead is use my ~/.nixpkgs/config.nix (which I believe you also have an equivalent of even in a non-NixOS nix install? I've never actually used nix separately).
The basic structure I use is this:
{
packageOverrides = nixpkgs: with nixpkgs; rec {
mine = with pkgs; buildEnv {
name = "mine";
paths = [
# your packages here
];
};
};
}
buildEnv is a convenience function from nix that makes an "environment" package out of a bunch of others; installing the package mine depends on (and so installs) all of the things listed in paths, and also makes sure they get included in PATH and things like that.
Then I just use nix-env -riA nixos.mine to deploy changes I've made to my environment description (or to rebuild my environment following channel updates). The -r tells it to remove everything else other than mine from the new generation of my profile, which means I can (ab?)use nix-env -i some-package as a way of "temporarily" installing some-package, and if I don't decide I like it enough to actually record it in my config.nix it'll just get removed anyway next time I deploy.
You can certainly create your own configuration. For example, you can do something like this:
let
pkgs = import <nixpkgs> {};
in
{
packages = [
pkgs.vim
pkgs.gimp
];
}
That would produce a set containing an attribute called packages, containing a list of Nix packages. But, you'd need to develop a tool to build environments from this, which is part of what nix-env does. For example, the tool can use nix-env to determine what is already installed, compare that to the selected packages in the configuration, and then install/uninstall packages accordingly.
You don't have a /etc/nixos/configuration.nix because that's NixOS-specific. So while you can do as you asked, you'd have to roll your own solution.
Let's say I need PostgreSQL 9.6.3 and Ruby 2.3.1 and various other tools. I can't find a tutorial that explains what I need to do.
From the Nix manual, I seem to need to write a Nix expression to install the needed dependencies, but I can't make the leap from:
{ stdenv, fetchurl, perl }:
stdenv.mkDerivation {
name = "hello-2.1.1";
builder = ./builder.sh;
src = fetchurl {
url = ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz;
md5 = "70c9ccf9fac07f762c24f2df2290784d";
};
inherit perl;
}
to the expression that will install the proper PostgreSQL and Ruby versions. It is absolutely unclear to me where to even put the file that installs PostgreSQL and Ruby, or how to run a single file in a given directory.
Can someone provide pointers to such tutorials, or point me in the right direction?
You can use nix-shell for this. It drops you into a shell configured to the given nix expression. Initially that expression could simply be along the lines of buildInputs = [ pkgs.ruby ]; and you can develop it from that. There are a number of helpful articles online written by nix users that give more examples of using nix-shell, like this one from garbas.si
You may also find it useful to get a better idea of how nix packages work. There's a separate nixpkgs manual that covers in greater detail using nix to create package expressions. A quick skim of the 3rd section should be useful to give a bit more understanding. There's also a chapter on using nix with ruby bundler that might be useful for you. Again there are articles that give more examples of its use, such as one from stesie.github.io.
If you need postgresql actually running in your environment nix won't manage that for you; its function is solely the building and management of packages, not their activation. You could simply activate postgres manually, use the nix-shell hook, or create some other integration with nix, but I think the most robust option is to make use of the Linux distribution that's built on top of Nix - NixOS. NixOS integrates with nix packages and manages the services provided by the packages. You could create a NixOS configuration with postgres active and your development environment present. This utility from github.com/chrisfarms may also be of interest.
One of the first types of Nix expression one encounters when learning how to use the Nix package manager is default.nix; on the wonderful NixOS IRC channel I learned of the existence of shell.nix and release.nix as well.
I got the impression that - roughly -default.nix is to be used with nix-build for simply building the package, shell.nix is used with nix-shell to create an interactive environment with the package and release.nix is used with nixops in deploying the package.
Since this is likely incomplete and partially incorrect, and since this does not seem to be clearly documented, I would like a clear and precise explanation of these sorts of "standard files"; in particular, for each of these file types (as well as any other standard files I am missing), I would like to know:
What are the typical use cases for such a file? What should they not be used for?
How is this file type structured typically? What are the minimal requirements for it?
Could you show a paradigm example of such a file within its use context, i.e. with use instructions and including lines of code needed to use it in the shell or another Nix expression?
As an additional bonus question, I want to know which - if any - of these standard files should be used when installing a package into a NixOS module? How would that be done?
First of all, default.nix and shell.nix have special meanings in Nix tool, but release.nix is a convenience one.
Next, default.nix is used as a default file when running nix-build, and shell.nix is used as a default file when running nix-shell. Just like you said.
Next, default.nix isn't used only for nix-build. For example, <nixpkgs/lib/default.nix> is used as aggregator for functions, and doesn't contain derivations. So not every default.nix is supposed to be "built" (but if default.nix is an attribute set of derivations, it will be buildable, and nix-build will build all of them).
Next, nix-shell will use default.nix if no shell.nix is found.
Next, default.nix is used as default file when importing directory. So if you write x = import ./some/directory;, then ./some/directory/default.nix will be imported. That actually should explain why "nix-build ." uses default.nix.
And finally, there are two common formats for derivations in default.nix: derivation, and callPackage derivation. You can't nix-build the latter. Almost any package in nixpkgs is written in this style, see hello. But you can
nix-build -E 'with import <nixpkgs> { }; callPackage ./path/to/default.nix { }'
as a workaround. nix-shell also supports this -E argument.
As said by #danbst only default.nix and shell.nix have special meanings for the nix tooling, out of that there is no real standard and everybody is free to use what fits most their needs.
That said, it doesn't mean you cannot set your own set of rules, personally for a single derivation project I like to arrange nix files in the following manner:
default.nix: Use callpackage to import derivation.nix.
derivation.nix: nixpkgs style derivation file.
shell.nix: nix-shell file.
module.nix: NixOS module file, import default.nix.
test.nix: NixOS test file.
release.nix: Hydra jobset declaration.
We had a talk about this topic at the Tokyo NixOS meetup, an example of a such code organization can be found here.