Using nix-store --restore to install sources - nix

I'm trying to install mythtv on nixOS.
mythtv uses Qt which uses tremor from xiph.org . git.xiph.org is currently unresponsive, so the install fails. Using nix show-derivation on the relevant .drv file, I see
{
"/nix/store/srhlr1v02a1gkp1ld1hrjzda0kfjcjkf-tremor-562307a.drv": {
"outputs": {
"out": {
"path": "/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a",
"hashAlgo": "r:sha256",
"hash": "f80ffaf1dc452061c4a2e7566ccec5f8ec1496ef2894857ed42f3ef7097e0754"
}
},
"inputSrcs": [
"/nix/store/jr7l8xyyanakpqr954rvj58lyqal1vfb-builder.sh",
"/nix/store/m9n5a7g5nb7jglcs9i58ycrkmn998wrp-nix-prefetch-git"
],
...
"env": {
...
"url": "https://git.xiph.org/tremor.git"
}
}
}
So I looked for, and succesfully found, this in cache.nixos.org:
$ curl cache.nixos.org/aq8d6qgp55w0g9xdb81cys9b3l95d7hb.narinfo
StorePath: /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
URL: nar/0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz
Compression: xz
FileHash: sha256:0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa
FileSize: 109380
NarHash: sha256:0m07gq4zfgigsiz8b518xyb19v7qqp76qmp7lb262825vkqzl3zq
NarSize: 588216
References:
Deriver: cr4d6v24rchmkv26mvzlzrr23z7a5ghb-tremor-562307a.drv
Sig: cache.nixos.org-1:rb9+9oe3LMRlh8TbLuZ7nhMaPIc5fwrAhG4tXtBLAgd7EGBoS2jk6BgYzSRI49MgwdPgAKTtzM3W0Wj0jGojDQ==
Then, I downloaded the narchive: curl -O cache.nixos.org/nar/0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz and attempted to restore it: nix-store --restore /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a < <(xzdec 0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz).
This fails initially because the directory exists:
$ nix-store --restore /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a < <(xzdec 0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz)
error: creating directory '/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a': File exists
$ ls -ld /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
drwxr-xr-x 3 nixbld10 nixbld 18 Jun 8 20:57 /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
$ ls -l /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
total 0
So I delete it, and try again:
$ nix-store --delete /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
finding garbage collector roots...
removing stale link from '/nix/var/nix/gcroots/auto/f9ldmvya8mix2j52cwsl8h0rqvjycjjy' to '/tmp/martyn/nixos-rebuild.didhrl/nix.drv'
removing stale link from '/nix/var/nix/gcroots/auto/qs606yn37b7irq7jmkvj0ld9wjv36fma' to '/tmp/martyn/nixos-rebuild.didhrl/nix'
deleting '/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a'
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves 0.00 MiB
1 store paths deleted, 0.07 MiB freed
$ nix-store --restore /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a < <(xzdec 0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz)
error: creating directory '/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a': Permission denied
I find that "permission denied" to be a bit odd - I have normal rights to install, indeed to delete (above)...
But it's okay, I can do this with sudo: ```$ sudo nix-store --restore /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a < <(xzdec 0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz)
$ ls -ld /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
drwxr-xr-x 5 root root 4096 Jun 9 07:27 /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
$ ls -l !$ | head
$ ls -l /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a | head
total 496
-rw-r--r-- 1 root root 6534 Jun 9 07:27 asm_arm.h
-rwxr-xr-x 1 root root 3657 Jun 9 07:27 autogen.sh
-rw-r--r-- 1 root root 4639 Jun 9 07:27 backends.h
-rw-r--r-- 1 root root 14547 Jun 9 07:27 block.c
-rw-r--r-- 1 root root 1109 Jun 9 07:27 block.h
-rw-r--r-- 1 root root 467 Jun 9 07:27 CHANGELOG
-rw-r--r-- 1 root root 10765 Jun 9 07:27 codebook.c
-rw-r--r-- 1 root root 4078 Jun 9 07:27 codebook.h
-rw-r--r-- 1 root root 3084 Jun 9 07:27 codec_internal.h
Better still, the hash looks good:
$ nix-hash /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a --type sha256
f80ffaf1dc452061c4a2e7566ccec5f8ec1496ef2894857ed42f3ef7097e0754
I notice that a lock file remains, so I delete that:
Along the way, I noticed that after the restore, the lock file remains:
$ ls -l /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a.lock
-rw------- 1 root root 0 May 27 08:15 /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a.lock
$ nix-store --delete !$
$ nix-store --delete /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a.lock
finding garbage collector roots...
deleting '/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a.lock'
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves 0.00 MiB
1 store paths deleted, 0.00 MiB freed
So I should be all set? It seems not:
$ nix-env -i /nix/store/srhlr1v02a1gkp1ld1hrjzda0kfjcjkf-tremor-562307a.drv
installing 'tremor-562307a.drv'
these derivations will be built:
/nix/store/srhlr1v02a1gkp1ld1hrjzda0kfjcjkf-tremor-562307a.drv
building '/nix/store/srhlr1v02a1gkp1ld1hrjzda0kfjcjkf-tremor-562307a.drv'...
exporting https://git.xiph.org/tremor.git (rev 562307a4a7082e24553f3d2c55dab397a17c4b4f) into /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
Initialized empty Git repository in /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a/.git/
I am baffled why nix is rebuilding a store path that is extant and has the requested hash.
I tried repairing the path, to see if maybe the database needed updating in some way:
$ nix-store --repair-path /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
error: you don't have sufficient rights to use this command
$ sudo nix-store --repair-path /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
error: path '/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a' is not valid
$ nix-hash /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a --type sha256
f80ffaf1dc452061c4a2e7566ccec5f8ec1496ef2894857ed42f3ef7097e0754
So that 'path ... is not valid' is alarming, but still, it's present with a good hash, so I don't understand why it's supposedly not valid, nor why nix insists on rebuilding it.

Maybe you have a negative narinfo cache entry or maybe the derivation prefers a local build. You could try
nix-store --delete /nix/store/srhlr1v02a1gkp1ld1hrjzda0kfjcjkf-tremor-562307a.drv
nix-store --realise --option narinfo-cache-negative-ttl 0 -vvvvv /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a

Related

Docker: Installing mysql-devel - files are symlink instead of binary

I'm currently trying to deploy a Ruby application to AWS Lambda. When I build the container docker build -t lambda-ruby2.5-mysqldep . and I navigate to the usr/lib64/mysql/ directory, the files are symlinks and not the actual binaries. I tried using unlink and rebuilding but they still come up as symlinks. Why is that? is that correct?
bash-4.2# cd usr/lib64/mysql/
bash-4.2# ls
libmysqlclient_r.so libmysqlclient.so libmysqlclient.so.18 libmysqlclient.so.18.0.0 mysql_config mysql_config55
bash-4.2# ls -l
total 2924
lrwxrwxrwx 1 root root 17 Jun 4 18:16 libmysqlclient_r.so -> libmysqlclient.so
lrwxrwxrwx 1 root root 20 Jun 4 18:16 libmysqlclient.so -> libmysqlclient.so.18
lrwxrwxrwx 1 root root 24 Jun 4 18:16 libmysqlclient.so.18 -> libmysqlclient.so.18.0.0
-rwxr-xr-x 1 root root 2983624 Dec 3 2018 libmysqlclient.so.18.0.0
-rwxr-xr-x 1 root root 7035 Dec 3 2018 mysql_config
lrwxrwxrwx 1 root root 12 Jun 4 18:16 mysql_config55 -> mysql_config
I think this is why I'm getting issues when testing the Lambda function. How can I reinstall and get the actual executable files instead of the symlinks? here is my Dockerfile
# Start with AWS lambda ruby environment
FROM lambci/lambda:build-ruby2.5
RUN yum -y install mysql-devel
RUN gem update bundler
WORKDIR /var/task
CMD "/bin/bash"

/bin/sh: app: not found

/ # which chasquid-util
/usr/local/bin/chasquid-util
/ # chasquid-util
/bin/sh: chasquid-util: not found
/ # /usr/local/bin/chasquid-util
/bin/sh: /usr/local/bin/chasquid-util: not found
/ # ls -al /usr/local/bin/
total 27432
drwxr-xr-x 1 root root 4096 Jul 26 16:18 .
drwxr-xr-x 1 root root 4096 Jul 26 16:18 ..
-rwxr-xr-x 1 root root 11721005 Jul 26 16:18 chasquid
-rwxr-xr-x 1 root root 5510494 Jul 26 16:18 chasquid-util
-rwxr-xr-x 1 root root 2910713 Jul 26 16:18 mda-lmtp
-rwxr-xr-x 1 root root 4767277 Jul 26 16:18 smtp-check
-rwxr-xr-x 1 root root 3164845 Jul 26 16:18 spf-check
/ #
Given your context, this typically means you are missing a shared library. With alpine, it's typically glibc since they ship with libmusl. You can check this with:
ldd chasquid-util
I've got several other reasons for this listed in my DC 2018 slidedeck:
Did you run the intended command? (e.g. docker run --rm my_image -it echo hello world will run the command -it)
Is docker trying to run a json string? (any json paring errors will show up as executing the json as a string)
Does the file exist... in the path and inside the container? (can't run stuff from the host inside a container without building it into the image or mounting a volume)
If it is a shell script, check the first line (e.g. #!/bin/bash)
Check for windows linefeeds on linux shell scripts (look for ^M or \r with different editors)
If it is a binary, there is likely a missing library (use ldd to check)

Can't access mounted volume in docker

UPDATE: This wasn't an issue when I setup my project on a Mac. I was using Fedora 24 in this problem.
I am trying to access my app in my docker instance. When I try and ls the mounted directory, I get a permission error:
root#591d02d0d6d2:/app#
ls: cannot open directory .: Permission denied
This is what my docker file looks like:
FROM pvlltvk/ubuntu-trusty-php-fpm-5.6
RUN apt-get install -y\
php5-curl \
php5-sybase \
freetds-dev \
libxml2-dev
ADD freetds.conf /etc/freetds/freetds.conf
USER root
RUN echo 'alias sf="php /app/app/console"' >> ~/.bashrc
WORKDIR /app
I know it's a permissions issue but I couldn't get any solutions to work. If I run ls -lh I can see that the owner id is 1000 and not root.
root#591d02d0d6d2:/# ls -lh
total 24K
drwxrwxr-x. 11 1000 1000 4.0K Dec 12 17:42 app //my project
drwxr-xr-x. 2 root root 4.0K Dec 13 08:25 bin
drwxr-xr-x. 2 root root 6 Apr 10 2014 boot
drwxr-xr-x. 5 root root 360 Dec 15 02:58 dev

bin mounted to alpine container fails to run with /gogs not found

I'm building gogs from source and using
COPY gogs /home/git/go/src/github.com/gogits/gogs
to add the folder containing the built gogs bin and it's files to my container.
If I cd to /home/git/go/src/github.com/gogits/gogs I can see the file, I can chmod +x it , ls -l it to see permissions and ownership etc etc. However I can't seem to run it. Every way I try I get:
/home/git/go/src/github.com/gogits/gogs # cat /var/log/gogs/stderr.log
/bin/ash: /home/git/go/src/github.com/gogits/gogs/gogs: not found
/bin/ash: /home/git/go/src/github.com/gogits/gogs/gogs: not found
/bin/ash: /home/git/go/src/github.com/gogits/gogs/gogs: not found
Running ls -l . within the gogs directory:
/home/git/go/src/github.com/gogits/gogs # ls -l .
total 33148
-rw-r--r-- 1 git nogroup 613 Aug 27 09:22 Dockerfile
-rw-r--r-- 1 git nogroup 913 Aug 27 09:22 Dockerfile.rpi
-rw-r--r-- 1 git nogroup 1054 Aug 27 09:22 LICENSE
-rw-r--r-- 1 git nogroup 1792 Aug 27 09:22 Makefile
-rw-r--r-- 1 git nogroup 7852 Aug 27 09:22 README.md
-rw-r--r-- 1 git nogroup 5331 Aug 27 09:22 README_ZH.md
drwxr-xr-x 2 git nogroup 4096 Aug 28 08:33 cmd
drwxr-xr-x 10 git nogroup 4096 Aug 28 08:33 conf
drwxr-xr-x 5 git nogroup 4096 Aug 28 08:35 custom
drwxr-xr-x 2 git nogroup 4096 Aug 28 08:33 data
drwxr-xr-x 4 git nogroup 4096 Aug 28 08:33 docker
-rw-r--r-- 1 git nogroup 5359 Aug 27 09:22 glide.lock
-rw-r--r-- 1 git nogroup 1756 Aug 27 09:22 glide.yaml
-rwxr-xr-x 1 git nogroup 33831440 Aug 27 10:12 gogs
-rw-r--r-- 1 git nogroup 839 Aug 27 09:22 gogs.go
drwxr-xr-x 2 git nogroup 4096 Aug 28 08:33 log
drwxr-xr-x 4 git nogroup 4096 Aug 28 08:33 models
drwxr-xr-x 34 git nogroup 4096 Aug 28 08:33 modules
drwxr-xr-x 4 git nogroup 4096 Aug 28 08:33 packager
drwxr-xr-x 14 git nogroup 4096 Aug 28 08:33 public
drwxr-xr-x 14 git nogroup 4096 Aug 28 08:33 routers
drwxr-xr-x 12 git nogroup 4096 Aug 28 08:33 scripts
-rw-r--r-- 1 root root 2750 Aug 28 08:43 supervisord.log
-rw-r--r-- 1 root root 2 Aug 28 08:43 supervisord.pid
drwxr-xr-x 18 git nogroup 4096 Aug 28 08:33 templates
Other stuff I've tried:
/home/git/go/src/github.com/gogits/gogs # cp gogs /usr/bin
/home/git/go/src/github.com/gogits/gogs # /usr/bin/gogs
ash: /usr/bin/gogs: not found
/home/git/go/src/github.com/gogits/gogs # ls /usr/bin | grep gogs
gogs
/home/git/go/src/github.com/gogits/gogs # /usr/bin/gogs
ash: /usr/bin/gogs: not found
/home/git/go/src/github.com/gogits/gogs # ./usr/bin/gogs
ash: ./usr/bin/gogs: not found
/home/git/go/src/github.com/gogits/gogs # ls /usr/bin/gogs
/usr/bin/gogs
Exec'ng into the docker container with the command sh I start to get weird errors after sometime, it may be tend to happen after I've added to PATH the gogs bin, can't remember for sure, but this has happened before
/home/git/go/src/github.com/gogits/gogs # gogs
sh: gogs: not found
/home/git/go/src/github.com/gogits/gogs # ./gogs
sh: ./gogs: not found
/home/git/go/src/github.com/gogits/gogs # export PATH=PATH:/home/git/go/src/github.com/gogits/gogs
/home/git/go/src/github.com/gogits/gogs # gogs
sh: gogs: not found
/home/git/go/src/github.com/gogits/gogs # cp gogs /usr/bin
sh: cp: not found
/home/git/go/src/github.com/gogits/gogs # cp gogs /usr/bin
sh: cp: not found
/home/git/go/src/github.com/gogits/gogs # cp
sh: cp: not found
/home/git/go/src/github.com/gogits/gogs # ls
sh: ls: not found
Also when I did try to build gogs from source within the container I got a Segmentation Fault (core dump) sth like that. I tested it with valgrind and got sth like Bad permissions for mapped region at address 0x1340F80
Requested info:
FROM alpine:edge
MAINTAINER jonathan#saharacluster.com
ENV DEBIAN_FRONTEND noninteractive
COPY gogs /home/git/go/src/github.com/gogits/gogs
WORKDIR /home/git/go/src/github.com/gogits/gogs
RUN apk update && apk add --update \
openssh \
curl\
git\
supervisor ;\
rm -rf /var/lib/apt/lists/* ;\
adduser -S -g 'Gogs' git;\
mkdir -p custom/conf \
log\
data\
/var/log/gogs\
/home/git/.ssh\
/home/git/data\
/var/run/sshd\
/var/log/install\
/var/log/sshd;\
touch /home/git/.ssh/authorized_keys\
/home/git/setup.sh \
/home/git/data/gogs.db;\
chmod 700 /home/git/.ssh && chmod 600 /home/git/.ssh/authorized_keys;\
chown -R git: /var/log/gogs /var/log/sshd /home/git
COPY app.ini custom/conf/app.ini
COPY supervisord.conf /home/git/supervisord.conf
COPY docker-entrypoint.sh /home/git/docker-entrypoint.sh
COPY setup.sh /home/git/setup.sh
RUN chmod +x /home/git/docker-entrypoint.sh /home/git/setup.sh
EXPOSE 22 3000
ENV DEBIAN_FRONTEND dialog
ENTRYPOINT ["/home/git/docker-entrypoint.sh"]
env
GOGS_USER_PASSWORD=mypassword
HOSTNAME=e11c1f1b65cd
SHLVL=1
HOME=/root
DOMAIN=abc.com
GOGS_USER_EMAIL=gogsuser#abc.com
APP_NAME=sc
API_USER_PASSWORD=mypassword
TERM=xterm
GOGS_USER=gogsuser
API_USER_EMAIL=apiuser#abc.com
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBIAN_FRONTEND=dialog
PWD=/home/git/go/src/github.com/gogits/gogs
API_USER=apiuser
The main problem is that you're using an image which doesn't have the Go libraries needed. I suspect that you're also not compiling your Go application for your target OS.
Cross compile
If you're compiling your app in Mac or Windows, you have to cross
compile it. To target 64 bit linux (Debian/Ubuntu), it's like this.
GOOS=linux GOARCH=amd64 go build -o myapp_linux-amd64 .
The flags for other OS's are here https://golang.org/doc/install/source#environment.
It's a good idea to name your output binary with the name of the target platform (_linux-amd64).
Use the official golang:1.7 image
That's the simplest way to do it. The image will already include all the libraries needed.
Later on, you can look into optimising your image size. That's when static linking and Alpine becomes useful but they have their own sets of gotchas (no libc so DNS might not resolve correctly depending on your app). They can be worked around but it might not be worth your time now.

Docker non mounting volume

Here is the situation on a clean docker-machine environment:
➜ momestock docker-machine start momestock
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.
➜ momestock eval "$(docker-machine env momestock)"
➜ momestock echo 'Helo world!' > hello.txt
➜ momestock ls -la
total 12
drwxr-xr-x 2 andrea andrea 4096 Oct 30 11:33 .
drwxr-xr-x 16 andrea andrea 4096 Oct 29 19:49 ..
-rw-r--r-- 1 andrea andrea 12 Oct 30 11:39 hello.txt
➜ momestock pwd
/home/andrea/projects/momestock
➜ momestock docker run -v /home/andrea/projects/momestock:/tmp busybox ls -la /tmp
total 4
drwxr-xr-x 2 root root 40 Oct 30 10:39 .
drwxr-xr-x 18 root root 4096 Oct 30 10:39 ..
➜ momestock
Is that a permissions problem? Or uids?
The problem appears with busybox but also every other container.
EDIT [SOLVED]:
looks like it's docker-machine's normal behavior to not bind mount
What OS do you use? Try to add 777 permissions just for check.
I just found out that docker-machine doesn't bind mount. It's the expected behavior.

Resources