How to install systemd service on nixos - nix

If I do this:
#!/usr/bin/env bash
set -e;
cd "$(dirname "$BASH_SOURCE")"
ln -sf "$(pwd)/interos-es-mdb.service" '/etc/systemd/system/interos-es-mdb.service'
systemctl enable interos-es-mdb.service
systemctl start interos-es-mdb.service
then I get this error:
ln: failed to create symbolic link '/etc/systemd/system/interos-es-mdb.service': Read-only file system
anyone know the right way to install a service on nixos machine? (I am the root user)...here is the service for reference:
[Unit]
Description=Interos MongoDB+ES log capture
After=network.target
[Service]
Environment=interos_emit_only_json=yes
EnvironmentFile=/root/interos/env/es-service.env
StartLimitIntervalSec=0
Type=simple
Restart=always
RestartSec=1
ExecStart=/root/interos/repos/elastic-search-app/syslog-exec.sh
[Install]
WantedBy=multi-user.target
update:
perhaps what I am looking for is "per-user" service, not something run as root etcetera.

The reason its broken
NixOS is a declarative operating system. This means that directories like /etc live inside the read-only /nix/store directory. Only the nix-daemon is allowed to mount the nix-store as writable. Therefore, you must create a systemd.services.<yourservice> entry in your configuration.nix to interact with the underlying system; alternatively you can patch nixpkgs directly and point your configuration to your fork.
All running services not declared explicitly by the user can be assumed to live inside nixpkgs/nixos/modules.
Fix
configuration.nix:
{
systemd.services.foo = {
enable = true;
description = "bar";
unitConfig = {
Type = "simple";
# ...
};
serviceConfig = {
ExecStart = "${foo}/bin/foo";
# ...
};
wantedBy = [ "multi-user.target" ];
# ...
};
}
user services
almost identical except they begin with systemd.user.services. In addition, user home directories are not managed declartively, so you can also place a regular systemd unit file under $XDG_CONFIG_DIR/systemd as usual.
relevant:
Full list of valid attributes for systemd.services.<name>, From: NixOS Manual
Module basics, From: Wiki

An appropriate entry in your /etc/nixos/configuration.nix might look like:
let
# assumes you build a derivation for your software and put it in
# /etc/nixos/pkgs/interosEsMdb/default.nix
interosEsMdb = import ./pkgs/interosEsMdb {};
in config.systemd.services.interosEsMdb = {
description = "Interos MongoDB+ES log capture";
after = ["network.target"];
wantedBy = ["multi-user.target"];
serviceConfig = {
# change this to refer to your actual derivation
ExecStart = "${interosEsMdb}/bin/syslog-exec.sh";
EnvironmentFile = "${interosEsMdb}/lib/es-service.env";
Restart = "always";
RestartSec = 1;
}
}
...assuming you actually build a derivation for interosEsMdb (which is the only sane and proper way to package software on NixOS).

Related

NixOS service systemd unit's $PATH does not contain expected dependency

I have the following definition:
hello123 =
(pkgs.writeScriptBin "finderapp" ''
#!${pkgs.stdenv.shell}
# Call hello with a traditional greeting
ls ${pkgs.ffmpeg-full}/bin/ffmpeg
ffmpeg --help
echo hello
''
);
And the service:
systemd.services = {
abcxyz = {
enable = true;
description = "abcxyz";
serviceConfig = {
WorkingDirectory = "%h/temp/";
Type = "simple";
ExecStart = "${hello123}/bin/finderapp";
Restart = "always";
RestartSec = 60;
};
wantedBy = [ "default.target" ];
};
};
However, this seems to fail being able to execute ffmpeg:
Jul 10 19:47:54 XenonKiloCranberry systemd[1]: Started abcxyz.
Jul 10 19:47:54 XenonKiloCranberry finderapp[10042]: /nix/store/9yx9s5yjc6ywafadplblzdfaxqimz95w-ffmpeg-full-4.2.3/bin/ffmpeg
Jul 10 19:47:54 XenonKiloCranberry finderapp[10042]: /nix/store/bxfwljbpvl21wsba00z5dm9dmshsk3bx-finderapp/bin/finderapp: line 5: ffmpeg: command not found
Jul 10 19:47:54 XenonKiloCranberry finderapp[10042]: hello
Why is this failing? I assume it's correctly getting ffmpeg as a runtime dependency (verified with nix-store -q --references ...) as stated in another question here: https://stackoverflow.com/a/68330101/1663462
If I add a echo $PATH to the script, it outputs the following:
Jul 10 19:53:44 XenonKiloCranberry finderapp[12011]: /nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin:/nix/store/97vambzyvpvrd9wgrrw7i7svi0s8vny5-findutils-4.7.0/bin:/nix/store/srmjkp5pq8c055j0lak2hn0ls0fis8yl-gnugrep-3.4/bin:/nix/store/p34p7ysy84579lndk7rbrz6zsfr03y71-gnused-4.8/bin:/nix/store/vfzp1mavwiz5w3v10hs69962k0gwl26c-systemd-243.7/bin:/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/sbin:/nix/store/97vambzyvpvrd9wgrrw7i7svi0s8vny5-findutils-4.7.0/sbin:/nix/store/srmjkp5pq8c055j0lak2hn0ls0fis8yl-gnugrep-3.4/sbin:/nix/store/p34p7ysy84579lndk7rbrz6zsfr03y71-gnused-4.8/sbin:/nix/store/vfzp1mavwiz5w3v10hs69962k0gwl26c-systemd-243.7/sbin
Or these paths basically:
/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin
/nix/store/97vambzyvpvrd9wgrrw7i7svi0s8vny5-findutils-4.7.0/bin
/nix/store/srmjkp5pq8c055j0lak2hn0ls0fis8yl-gnugrep-3.4/bin
/nix/store/p34p7ysy84579lndk7rbrz6zsfr03y71-gnused-4.8/bin
/nix/store/vfzp1mavwiz5w3v10hs69962k0gwl26c-systemd-243.7/bin
/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/sbin
/nix/store/97vambzyvpvrd9wgrrw7i7svi0s8vny5-findutils-4.7.0/sbin
/nix/store/srmjkp5pq8c055j0lak2hn0ls0fis8yl-gnugrep-3.4/sbin
/nix/store/p34p7ysy84579lndk7rbrz6zsfr03y71-gnused-4.8/sbin
/nix/store/vfzp1mavwiz5w3v10hs69962k0gwl26c-systemd-243.7/sbin
Which shows that ffmpeg is not in there.
I don't think this is the most elegant solution as the dependencies have to be known in the service definition instead of the package/derivation, but it's a solution none the less.
We can add additional paths with path = [ pkgs.ffmpeg-full ];:
abcxyz = {
enable = true;
description = "abcxyz";
path = [ pkgs.ffmpeg-full ];
serviceConfig = {
WorkingDirectory = "%h/temp/";
Type = "simple";
ExecStart = "${hello123}/bin/finderapp";
Restart = "always";
RestartSec = 60;
};
wantedBy = [ "default.target" ];
};
In addition to the previous answers
not using PATH
adding to PATH via systemd config
you can add it to the PATH inside the wrapper script, making the script more self-sufficient and making the extended PATH available to subprocesses, if ffmpeg or any other command needs it (probably not in this case).
The ls command has no effect on subsequent commands, like it shouldn't.
What you want is to add it to PATH:
hello123 =
(pkgs.writeScriptBin "finderapp" ''
#!${pkgs.stdenv.shell}
# Call hello with a traditional greeting
PATH="${pkgs.ffmpeg-full}/bin${PATH:+:${PATH}}"
ffmpeg --help
echo hello
''
);
The part ${PATH:+:${PATH}} takes care of the : and pre-existing PATH, if there is one. The simplistic :${PATH} could effectively add . to PATH when it was empty, although that would be rare.

How to get Task ID from within ECS container?

Hello I am interested in retrieving the Task ID from within inside a running container which lives inside of a EC2 host machine.
AWS ECS documentation states there is an environment variable ECS_CONTAINER_METADATA_FILE with the location of this data but will only be set/available if ECS_ENABLE_CONTAINER_METADATA variable is set to true upon cluster/EC2 instance creation. I don't see where this can be done in the aws console.
Also, the docs state that this can be done by setting this to true inside the host machine but would require to restart the docker agent.
Is there any other way to do this without having to go inside the EC2 to set this and restart the docker agent?
This doesn't work for newer Amazon ECS container versions anymore, and in fact it's now much simpler and also enabled by default. Please refer to this docu, but here's a TL;DR:
If you're using Amazon ECS container agent version 1.39.0 and higher, you can just do this inside the docker container:
curl -s "$ECS_CONTAINER_METADATA_URI_V4/task" \
| jq -r ".TaskARN" \
| cut -d "/" -f 3
Here's a list of container agent releases, but if you're using :latest – you're definitely fine.
The technique I'd use is to set the environment variable in the container definition.
If you're managing your tasks via Cloudformation, the relevant yaml looks like so:
Taskdef:
Type: AWS::ECS::TaskDefinition
Properties:
...
ContainerDefinitions:
- Name: some-name
...
Environment:
- Name: AWS_DEFAULT_REGION
Value: !Ref AWS::Region
- Name: ECS_ENABLE_CONTAINER_METADATA
Value: 'true'
This technique helps you keep everything straightforward and reproducible.
If you need metadata programmatically and don't have access to the metadata file, you can query the agent's metadata endpoint:
curl http://localhost:51678/v1/metadata
Note that if you're getting this information as a running task, you may not be able to connect to the loopback device, but you can connect to the EC2 instance's own IP address.
We set it with the so called user data, which are executed at the start of the machine. There are multiple ways to set it, for example: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-console
It could look like this:
#!/bin/bash
cat <<'EOF' >> /etc/ecs/ecs.config
ECS_CLUSTER=ecs-staging
ECS_ENABLE_CONTAINER_METADATA=true
EOF
Important: Adjust the ECS_CLUSTER above to match your cluster name, otherwise the instance will not connect to that cluster.
Previous answers are correct, here is another way of doing this:
From the ec2 instance where container is running, run this command
curl http://localhost:51678/v1/tasks | python -mjson.tool |less
From the AWS ECS cli Documentation
Command:
aws ecs list-tasks --cluster default
Output:
{
"taskArns": [
"arn:aws:ecs:us-east-1:<aws_account_id>:task/0cc43cdb-3bee-4407-9c26-c0e6ea5bee84",
"arn:aws:ecs:us-east-1:<aws_account_id>:task/6b809ef6-c67e-4467-921f-ee261c15a0a1"
]
}
To list the tasks on a particular container instance
This example command lists the tasks of a specified container instance, using the container instance UUID as a filter.
Command:
aws ecs list-tasks --cluster default --container-instance f6bbb147-5370-4ace-8c73-c7181ded911f
Output:
{
"taskArns": [
"arn:aws:ecs:us-east-1:<aws_account_id>:task/0cc43cdb-3bee-4407-9c26-c0e6ea5bee84"
]
}
My ECS solution as bash and Python snippets. Logging calls are able to print for debug by piping to sys.stderr while print() is used to pass the value back to a shell script
#!/bin/bash
TASK_ID=$(python3.8 get_ecs_task_id.py)
echo "TASK_ID: ${TASK_ID}"
Python script - get_ecs_task_id.py
import json
import logging
import os
import sys
import requests
# logging configuration
# file_handler = logging.FileHandler(filename='tmp.log')
# redirecting to stderr so I can pass back extracted task id in STDOUT
stdout_handler = logging.StreamHandler(stream=sys.stderr)
# handlers = [file_handler, stdout_handler]
handlers = [stdout_handler]
logging.basicConfig(
level=logging.INFO,
format="[%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s",
handlers=handlers,
datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(__name__)
def get_ecs_task_id(host):
path = "/task"
url = host + path
headers = {"Content-Type": "application/json"}
r = requests.get(url, headers=headers)
logger.debug(f"r: {r}")
d_r = json.loads(r.text)
logger.debug(d_r)
ecs_task_arn = d_r["TaskARN"]
ecs_task_id = ecs_task_arn.split("/")[2]
return ecs_task_id
def main():
logger.debug("Extracting task ID from $ECS_CONTAINER_METADATA_URI_V4")
logger.debug("Inside get_ecs_task_id.py, redirecting logs to stderr")
logger.debug("so that I can pass the task id back in STDOUT")
host = os.environ["ECS_CONTAINER_METADATA_URI_V4"]
ecs_task_id = get_ecs_task_id(host)
# This print statement passes the string back to the bash wrapper, don't remove
logger.debug(ecs_task_id)
print(ecs_task_id)
if __name__ == "__main__":
main()

Nix external software built/installed is not being found

I've just started using the Nix package manager on OSX and I'm attempting to create my first package for the pass binary (https://www.passwordstore.org) - which is not available in the Nixpkgs repository.
I'm attempting to specify a runtime dependency (getopt), however this doesn't appear to be available when the binary is used.
This is my packages's default.nix:
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
let
version = "1.7.1";
in {
pass = stdenv.mkDerivation rec {
name = "pass-${version}";
src = fetchurl {
url = "https://git.zx2c4.com/password-store/snapshot/password-store-1.7.1.tar.xz";
sha256 = "0scqkpll2q8jhzcgcsh9kqz0gwdpvynivqjmmbzax2irjfaiklpn";
};
buildInputs = [ stdenv makeWrapper];
installPhase = ''
make install PREFIX=$out/artifact
makeWrapper $out/artifact/bin/pass $out/bin/pass \
--set PATH ${stdenv.lib.makeBinPath [ getopt ]}
'';
meta = {
homepage = "https://www.passwordstore.org";
description = "The standard unix password manager";
license = stdenv.lib.licenses.gpl2Plus;
};
};
}
I can successfully build this package (nix-build --show-trace) and install it (nix-env -i ./result).
Listing the runtime dependencies for my package shows getopt listed:
nix-store -qR $(which pass)
...
/nix/store/c5swmygzc0kmvpq6cfkvwm2yz1k57kqy-getopt-1.1.4
However when I come to use the binary (pass init my-key) I get the following error:
/nix/store/...-pass-1.7.1/artifact/bin/pass: line 302:
/usr/local/bin/getopt: No such file or directory
Can anyone advise what I'm doing wrong?
Thanks
It looks like getopt gets a special treatment. The darwin.sh script looks for it using brew and port and falls back to /usr/local. That's why the (correct) wrapper has no effect.
So the solution seems to be, to make it look for getopt in PATH, which is provided by the wrapper script. You can probably make it as simple as GETOPT=getopt (which is similar to openbsd.sh)
For patching source code, see the NixPkgs documentation
After running nix-build, you should run cat result/bin/pass to look at your wrapper script and make sure it looks OK. It should be a shell script that sets the PATH to include getopt and then calls result/artifact/bin/pass.
Then try running the wrapper script. Note that the wrapper should be in result/bin, not result/artifact/bin.

How to get the name from a nixpkgs derivation in a nix expression to be used by nix-shell?

I'm writing a .nix expression to be used primarily by nix-shell. I'm not sure how to do that. Note this is not on NixOS, but I don't think that is very relevant.
The particular example I'm looking at is that I want to get this version-dependent name that looks like:
idea-ultimate = buildIdea rec {
name = "idea-ultimate-${version}";
version = "2017.2.2"; /* updated by script */
description = "Integrated Development Environment (IDE) by Jetbrains, requires paid license";
license = stdenv.lib.licenses.unfree;
src = fetchurl {
url = "https://download.jetbrains.com/idea/ideaIU-${version}-no-jdk.tar.gz";
sha256 = "b8eb9d612800cc896eb6b6fbefbf9f49d92d2350ae1c3c4598e5e12bf93be401"; /* updated by script */
};
wmClass = "jetbrains-idea";
update-channel = "IDEA_Release";
};
My nix expression is the following:
let
pkgs = import <nixpkgs> {};
stdenv = pkgs.stdenv;
# idea_name = assert pkgs.jetbrains.idea-ultimate.name != ""; pkgs.jetbrains.idea-ultimate.name;
in rec {
scalaEnv = stdenv.mkDerivation rec {
name = "scala-env";
builder = "./scala-build.sh";
shellHook = ''
alias cls=clear
'';
CLANG_PATH = pkgs.clang + "/bin/clang";
CLANGPP_PATH = pkgs.clang + "/bin/clang++";
# A bug in the nixpkgs openjdk (#29151) makes us resort to Zulu OpenJDK for IDEA:
# IDEA_JDK = pkgs.openjdk + "/lib/openjdk";
# PATH = "${pkgs.jetbrains.idea-ultimate}/${idea_name}/bin:$PATH";
IDEA_JDK = /usr/lib/jvm/zulu-8-amd64;
# IDEA_JDK = /opt/zulu8.23.0.3-jdk8.0.144-linux_x64;
# IDEA_JDK = /usr/lib/jvm/java-8-openjdk-amd64;
buildInputs = with pkgs; [
ammonite
boehmgc
clang
emacs
jetbrains.idea-ultimate
less
libunwind
openjdk
re2
sbt
stdenv
unzip
zlib
];
};
}
I have commented out setting PATH as it depends on getting idea_name in the let-clause. As an interesting side note, as is, this does not fail if I leave it uncommented but causes a very bizarre error when executing nix-shell about not being able to execute bash. I've also tried the more simple case of let idea_name = pkgs.jetbrains.idea-ultimate.name; but this fails later on when idea_name is used in setting PATH since idea_name ends up being undefined.
Update:
I began exploring with nix-instantiate, but the derivation of interest seems empty:
[nix-shell:/nix/store]$ nix-instantiate --eval --xml -E "((import <nixpkgs> {}).callPackage ./3hk87pqgl2qdqmskxbhy23cyr24q8g6s-nixpkgs-18.03pre114739.d0d905668c/nixpkgs/pkgs/applications/editors/jetbrains { }).idea-ultimate";
<?xml version='1.0' encoding='utf-8'?>
<expr>
<derivation>
<repeated />
</derivation>
</expr>
If your intent is to get idea-ultimate into nix-shell environment, then just include that package to buildInputs. I see it's already included, so it should already be present in your PATH.
BTW, you can extend your shellHook and export PATH and other variables rather from there, where you have full bash. Why would you do it from bash? Less copying. When you specify
IDEA_JDK = /usr/lib/jvm/zulu-8-amd64;
in Nix, the file /usr/lib/jvm/zulu-8-amd64 get's copied to nix store and IDEA_JDK is set to point to file in /nix/store. Was that your intent?
Regarding nix-instantiate:
$ nix-instantiate --eval -E 'with import <nixpkgs>{}; idea.pycharm-community.outPath'
"/nix/store/71jk0spr30rm4wsihjwbb1hcwwvzqr4k-pycharm-community-2017.1"
but you still have to remove doublequotes (https://gist.github.com/danbst/a9fc068ff26e31d88de9709965daa2bd)
Also, nitpick, assert pkgs.jetbrains.idea-ultimate.name != ""; can be dropped as it's impossible to have empty derivation name in Nix.
And another nitpick. You'll soon find very incovenient to launch IDE from shell every time. It seems a good idea to specify, that some package is used for development, but nix-shell doesn't work well for non-cli applications. Not to mention occasional problems with Nix GC and nix-shell. You'd better install IDE globally or per-user, it is better long-term solution.
[ADDENDUM]
You are looking for this (dev-environment.nix):
with import <nixpkgs> { };
buildEnv {
name = "my-super-dev-env";
paths = [
#emacs
nano
idea.pycharm-community
];
buildInputs = [ makeWrapper ];
postBuild = ''
for f in $(ls -d $out/bin/*); do
wrapProgram $f \
--set IDEA_JDK "/path/to/zulu-jdk" \
--set CLANG_PATH ... \
--set CLANCPP_PATH ...
done
'';
}
which you install using nix-env -if ./dev-environment.nix. It will wrap your programs with those env vars, without polluting your workspace (you can pollute it further using nix-shell with shell hook, if you want).

using the nix typing system with `nix-instantiate`

myservice
i've written a nixos service in myservice.nix and i include it in /etc/nixos/configuration.nix with:
imports [ /path/to/myservice.nix ];
and later i'm using it inside configuration.nix:
services.myservice.enable = true;
question
in one scenario i can't use nixos-rebuild switch but because typing in nix is linked to the options system using foo = mkOption { type = types.int; ...} i'm forced to use the options systems even though i just want to compute a configuration file for nginx using nix.
how to evaluate that nginx.conf only?
#aszlig wrote me this command:
nix-instantiate --eval --strict -E '(import <nixpkgs/nixos> { configuration = { imports = [ nixcloud-reverse-proxy/nixcloud-reverse-proxy.nix ]; services.nixcloud-reverse-proxy.enable = true; }; }).config.system.build.configsFromPath'
execution results in
nix-instantiate --eval --strict -E '(import <nixpkgs/nixos> { configuration = { imports = [ ./nixcloud-reverse-proxy.nix ]; services.nixcloud-reverse-proxy.enable = true; }; }).config.system.build.configsFromPath'
error: attribute ‘configsFromPath’ missing, at (string):1:1
(use ‘--show-trace’ to show detailed location information)
update
nix-build '<nixpkgs/nixos>' -A config.systemd.services.nixcloud-reverse-proxy.runner -I nixos-config=./configuration.nix
...
/nix/store/lp2jbb1wahhlr7qkq81rmfvk84mjk1vk-nixcloud-reverse-proxy-runner
now i can use that to grep the conf file:
cat /nix/store/lp2jbb1wahhlr7qkq81rmfvk84mjk1vk-nixcloud-reverse-proxy-runner | grep -o ' /nix/store/.*nginx-reverse-proxy.conf'
... kind of a workaround but not very precise! i'd rather like a config file in a directory.
I see your file name is nginx-reverse-proxy.conf, so it isn't built with fancy NixOS module system, but with some other ways. In that case (you control how you build your file) you can include it globally:
environment.etc."myconfigs"."nginx-reverse-proxy.conf".text = ...content of that file;
Which your refer then with
$ nix-instantiate --eval -E '
with import <nixpkgs/nixos> {
configuration = { ... };
};
config.environment.etc."myconfigs"."nginx-reverse-proxy.conf".text
'
You probably need to decode that output though using trick described in https://gist.github.com/danbst/a9fc068ff26e31d88de9709965daa2bd
which is itself a convoluted way to do
$ cat $(nix-build -E '
with import <nixpkgs/nixos> {
configuration = { ... };
};
config.environment.etc."myconfigs"."nginx-reverse-proxy.conf".source
')
In case your proxy config is part of general nginx.conf script, then you still can get it using
$ cat $(cat $(nix-build -E '
with import <nixpkgs/nixos> {
configuration = ...;
};
config.system.build.units."nginx.service".unit
')/nginx.service \
| grep nginx.conf \
| sed -r 's/.* (.*nginx.conf).*/\1/g'
)
Because nginx.conf file is private to nginx module, we can't reference it directly, but have to extract it directly from usage site.
Overall, NixOS lacks a good interface to introspect its internals, but it is still possible.

Resources