Hi can somebody please tell me what does this expression mean
ls -lrt *.txt|egrep -v " (*.?) (*.?) (*.?) 0 (*.?) (*.?) (*.?) "
any insight is very much appreciated.
First, please note that if someone is grepping the output of ls, it's almost certainly the wrong way to achieve the task. ls is meant for human consumption, and find is probably the better tool.
It looks to me that the person is trying to find any files that have a size greater than zero. If so, then the way that it should be done is:
find . -type f -size +0
ls -lrt
list files (ls), detailed view (-l), sorted backwards (-r) by modification time (-t)
ls -lrt *.txt
list all text files sorted by time, the output looks like this
$ ls -ltr
drwxr-xr-x 2 root root 4.0K Mar 5 2012 selinux
drwxr-xr-x 2 root root 4.0K Apr 25 2012 srv
drwx------ 2 root root 16K Feb 7 2013 lost+found
drwxr-xr-x 2 root root 4.0K Feb 7 2013 cdrom
now you're grepping for all lines except for (-v) those with a zero in the 4th column, which I don't fully understand.
anyway, (*.?) would mean just any sequence of characters (zero or more)
Related
I'm trying to install mythtv on nixOS.
mythtv uses Qt which uses tremor from xiph.org . git.xiph.org is currently unresponsive, so the install fails. Using nix show-derivation on the relevant .drv file, I see
{
"/nix/store/srhlr1v02a1gkp1ld1hrjzda0kfjcjkf-tremor-562307a.drv": {
"outputs": {
"out": {
"path": "/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a",
"hashAlgo": "r:sha256",
"hash": "f80ffaf1dc452061c4a2e7566ccec5f8ec1496ef2894857ed42f3ef7097e0754"
}
},
"inputSrcs": [
"/nix/store/jr7l8xyyanakpqr954rvj58lyqal1vfb-builder.sh",
"/nix/store/m9n5a7g5nb7jglcs9i58ycrkmn998wrp-nix-prefetch-git"
],
...
"env": {
...
"url": "https://git.xiph.org/tremor.git"
}
}
}
So I looked for, and succesfully found, this in cache.nixos.org:
$ curl cache.nixos.org/aq8d6qgp55w0g9xdb81cys9b3l95d7hb.narinfo
StorePath: /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
URL: nar/0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz
Compression: xz
FileHash: sha256:0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa
FileSize: 109380
NarHash: sha256:0m07gq4zfgigsiz8b518xyb19v7qqp76qmp7lb262825vkqzl3zq
NarSize: 588216
References:
Deriver: cr4d6v24rchmkv26mvzlzrr23z7a5ghb-tremor-562307a.drv
Sig: cache.nixos.org-1:rb9+9oe3LMRlh8TbLuZ7nhMaPIc5fwrAhG4tXtBLAgd7EGBoS2jk6BgYzSRI49MgwdPgAKTtzM3W0Wj0jGojDQ==
Then, I downloaded the narchive: curl -O cache.nixos.org/nar/0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz and attempted to restore it: nix-store --restore /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a < <(xzdec 0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz).
This fails initially because the directory exists:
$ nix-store --restore /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a < <(xzdec 0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz)
error: creating directory '/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a': File exists
$ ls -ld /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
drwxr-xr-x 3 nixbld10 nixbld 18 Jun 8 20:57 /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
$ ls -l /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
total 0
So I delete it, and try again:
$ nix-store --delete /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
finding garbage collector roots...
removing stale link from '/nix/var/nix/gcroots/auto/f9ldmvya8mix2j52cwsl8h0rqvjycjjy' to '/tmp/martyn/nixos-rebuild.didhrl/nix.drv'
removing stale link from '/nix/var/nix/gcroots/auto/qs606yn37b7irq7jmkvj0ld9wjv36fma' to '/tmp/martyn/nixos-rebuild.didhrl/nix'
deleting '/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a'
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves 0.00 MiB
1 store paths deleted, 0.07 MiB freed
$ nix-store --restore /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a < <(xzdec 0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz)
error: creating directory '/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a': Permission denied
I find that "permission denied" to be a bit odd - I have normal rights to install, indeed to delete (above)...
But it's okay, I can do this with sudo: ```$ sudo nix-store --restore /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a < <(xzdec 0jm68cws51814dm7gks62vj7a8zrdhwv2x364kq78wbkkqh58maa.nar.xz)
$ ls -ld /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
drwxr-xr-x 5 root root 4096 Jun 9 07:27 /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
$ ls -l !$ | head
$ ls -l /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a | head
total 496
-rw-r--r-- 1 root root 6534 Jun 9 07:27 asm_arm.h
-rwxr-xr-x 1 root root 3657 Jun 9 07:27 autogen.sh
-rw-r--r-- 1 root root 4639 Jun 9 07:27 backends.h
-rw-r--r-- 1 root root 14547 Jun 9 07:27 block.c
-rw-r--r-- 1 root root 1109 Jun 9 07:27 block.h
-rw-r--r-- 1 root root 467 Jun 9 07:27 CHANGELOG
-rw-r--r-- 1 root root 10765 Jun 9 07:27 codebook.c
-rw-r--r-- 1 root root 4078 Jun 9 07:27 codebook.h
-rw-r--r-- 1 root root 3084 Jun 9 07:27 codec_internal.h
Better still, the hash looks good:
$ nix-hash /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a --type sha256
f80ffaf1dc452061c4a2e7566ccec5f8ec1496ef2894857ed42f3ef7097e0754
I notice that a lock file remains, so I delete that:
Along the way, I noticed that after the restore, the lock file remains:
$ ls -l /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a.lock
-rw------- 1 root root 0 May 27 08:15 /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a.lock
$ nix-store --delete !$
$ nix-store --delete /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a.lock
finding garbage collector roots...
deleting '/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a.lock'
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves 0.00 MiB
1 store paths deleted, 0.00 MiB freed
So I should be all set? It seems not:
$ nix-env -i /nix/store/srhlr1v02a1gkp1ld1hrjzda0kfjcjkf-tremor-562307a.drv
installing 'tremor-562307a.drv'
these derivations will be built:
/nix/store/srhlr1v02a1gkp1ld1hrjzda0kfjcjkf-tremor-562307a.drv
building '/nix/store/srhlr1v02a1gkp1ld1hrjzda0kfjcjkf-tremor-562307a.drv'...
exporting https://git.xiph.org/tremor.git (rev 562307a4a7082e24553f3d2c55dab397a17c4b4f) into /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
Initialized empty Git repository in /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a/.git/
I am baffled why nix is rebuilding a store path that is extant and has the requested hash.
I tried repairing the path, to see if maybe the database needed updating in some way:
$ nix-store --repair-path /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
error: you don't have sufficient rights to use this command
$ sudo nix-store --repair-path /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
error: path '/nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a' is not valid
$ nix-hash /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a --type sha256
f80ffaf1dc452061c4a2e7566ccec5f8ec1496ef2894857ed42f3ef7097e0754
So that 'path ... is not valid' is alarming, but still, it's present with a good hash, so I don't understand why it's supposedly not valid, nor why nix insists on rebuilding it.
Maybe you have a negative narinfo cache entry or maybe the derivation prefers a local build. You could try
nix-store --delete /nix/store/srhlr1v02a1gkp1ld1hrjzda0kfjcjkf-tremor-562307a.drv
nix-store --realise --option narinfo-cache-negative-ttl 0 -vvvvv /nix/store/aq8d6qgp55w0g9xdb81cys9b3l95d7hb-tremor-562307a
I have a Docker image which contains an analysis pipeline. To run this pipeline, I need to provide input data and I want to keep the outputs. This pipeline must be able to be run by other users than myself, on their own laptops.
Briefly, my root (/) folder structure is as follows:
total 72
drwxr-xr-x 1 root root 4096 May 29 15:38 bin
drwxr-xr-x 2 root root 4096 Feb 1 17:09 boot
drwxr-xr-x 5 root root 360 Jun 1 15:31 dev
drwxr-xr-x 1 root root 4096 Jun 1 15:31 etc
drwxr-xr-x 2 root root 4096 Feb 1 17:09 home
drwxr-xr-x 1 root root 4096 May 29 15:49 lib
drwxr-xr-x 2 root root 4096 Feb 24 00:00 lib64
drwxr-xr-x 2 root root 4096 Feb 24 00:00 media
drwxr-xr-x 2 root root 4096 Feb 24 00:00 mnt
drwxr-xr-x 1 root root 4096 Mar 12 19:38 opt
drwxr-xr-x 1 root root 4096 Jun 1 15:24 pipeline
dr-xr-xr-x 615 root root 0 Jun 1 15:31 proc
drwx------ 1 root root 4096 Mar 12 19:38 root
drwxr-xr-x 3 root root 4096 Feb 24 00:00 run
drwxr-xr-x 1 root root 4096 May 29 15:38 sbin
drwxr-xr-x 2 root root 4096 Feb 24 00:00 srv
dr-xr-xr-x 13 root root 0 Apr 29 10:14 sys
drwxrwxrwt 1 root root 4096 Jun 1 15:25 tmp
drwxr-xr-x 1 root root 4096 Feb 24 00:00 usr
drwxr-xr-x 1 root root 4096 Feb 24 00:00 var
The pipeline scripts are in /pipeline and are packaged into the image with a "COPY. /pipeline" instruction in my Dockerfile.
For various reasons, this pipeline (which is a legacy pipeline) is set up so that the input data must be in a folder such /pipeline/project. To run my pipeline, I use:
docker run --rm --mount type=bind,source=$(pwd),target=/pipeline/project --user "$(id -u):$(id -g)" pipelineimage:v1
In other words, I mount a folder with the data to /pipeline/project. I found I needed to use the --user to insure the output files would have the correct permissions - i.e. I would have read/write/exec access on my host computer after the container exits.
The pipeline runs but I have one issue: one particular software used by the pipeline automatically tries to produce (and I can't change that) 1 folder in $HOME (so / - which I showed above) and 1 folder in my WORKDIR (which I have set up in my Dockerfile to be /pipeline). These attempts fails, and I'm guessing it's because I am not running the pipeline as root. But I need to use --user to make sure my outputs have the correct permissions - i.e. that I don't require sudo rights to read these outputs etc.
My question is: how am I meant to handle this? It seems that by using --user, I have the correct permissions set for the mounted folder (/pipeline/projects) where many output files are successfully made, no problems there. But how can I ensure the other 2 folders are correctly made outside of that mount?
I have tried the following but not success:
Doing "COPY -chown myhostuid:mygroupid" . pipeline/". This works but I have to hardcode my uid and gid so that won't work if another colleague tries to run the image.
Adding a new user with sudo rights and making it run the image: "RUN useradd -r newuser -g sudo" (I also tried using the "root" group but no success). This just gives me outputs which require sudo rights to read/write/exec. Which is not what I want.
Am I missing something? I don't understand why it's "easy" to handle permissions for a mounted folder but so much harder for the other folders in a container. Thanks.
If your software doesn't rely on relative paths (~/, ./), you can just set $HOME and WORKDIR to a directory that any user can write:
ENV HOME=/tmp
WORKDIR /tmp
If you can't do that, you can pass the uid/gid via the environment to an entrypoint script running as root, chown/chmod as necessary, then drop privileges to run the pipeline (runuser, su, sudo, setuidgid).
For example (untested):
entrypoint.sh
#!/bin/bash
[[ -v "RUN_UID" ]] || { echo "unset RUN_UID" >&2; exit 1; }
[[ -v "RUN_GID" ]] || { echo "unset RUN_GID" >&2; exit 1; }
# chown, chmod, set env, etc.
chown $RUN_UID:$RUN_GID "/path/that/requires/write/permissions"
export HOME=/tmp
# Run the pipeline as a non-root user.
sudo -E -u "#$RUN_UID" -g "#$RUN_GID" /path/to/pipeline
Dockerfile
...
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
Finally, pass the user and group IDs via the environment when running:
docker run --rm --mount type=bind,source=$(pwd),target=/pipeline/project -e RUN_UID=$(id -u) -e RUN_GID=$(id -g) pipelineimage:v1
/ # which chasquid-util
/usr/local/bin/chasquid-util
/ # chasquid-util
/bin/sh: chasquid-util: not found
/ # /usr/local/bin/chasquid-util
/bin/sh: /usr/local/bin/chasquid-util: not found
/ # ls -al /usr/local/bin/
total 27432
drwxr-xr-x 1 root root 4096 Jul 26 16:18 .
drwxr-xr-x 1 root root 4096 Jul 26 16:18 ..
-rwxr-xr-x 1 root root 11721005 Jul 26 16:18 chasquid
-rwxr-xr-x 1 root root 5510494 Jul 26 16:18 chasquid-util
-rwxr-xr-x 1 root root 2910713 Jul 26 16:18 mda-lmtp
-rwxr-xr-x 1 root root 4767277 Jul 26 16:18 smtp-check
-rwxr-xr-x 1 root root 3164845 Jul 26 16:18 spf-check
/ #
Given your context, this typically means you are missing a shared library. With alpine, it's typically glibc since they ship with libmusl. You can check this with:
ldd chasquid-util
I've got several other reasons for this listed in my DC 2018 slidedeck:
Did you run the intended command? (e.g. docker run --rm my_image -it echo hello world will run the command -it)
Is docker trying to run a json string? (any json paring errors will show up as executing the json as a string)
Does the file exist... in the path and inside the container? (can't run stuff from the host inside a container without building it into the image or mounting a volume)
If it is a shell script, check the first line (e.g. #!/bin/bash)
Check for windows linefeeds on linux shell scripts (look for ^M or \r with different editors)
If it is a binary, there is likely a missing library (use ldd to check)
Here is the situation on a clean docker-machine environment:
➜ momestock docker-machine start momestock
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.
➜ momestock eval "$(docker-machine env momestock)"
➜ momestock echo 'Helo world!' > hello.txt
➜ momestock ls -la
total 12
drwxr-xr-x 2 andrea andrea 4096 Oct 30 11:33 .
drwxr-xr-x 16 andrea andrea 4096 Oct 29 19:49 ..
-rw-r--r-- 1 andrea andrea 12 Oct 30 11:39 hello.txt
➜ momestock pwd
/home/andrea/projects/momestock
➜ momestock docker run -v /home/andrea/projects/momestock:/tmp busybox ls -la /tmp
total 4
drwxr-xr-x 2 root root 40 Oct 30 10:39 .
drwxr-xr-x 18 root root 4096 Oct 30 10:39 ..
➜ momestock
Is that a permissions problem? Or uids?
The problem appears with busybox but also every other container.
EDIT [SOLVED]:
looks like it's docker-machine's normal behavior to not bind mount
What OS do you use? Try to add 777 permissions just for check.
I just found out that docker-machine doesn't bind mount. It's the expected behavior.
I was start working on the Docker and I got some question regards on the docker images. Since each time when I create the new image and I will rmi the old images from the Docker, but looks like even after I run the commands > docker rmi 'IMAGE ID', and when I cd into the /var/lib/docker-registry/images directory, it still shows lots of images there as below:
drwxr-xr-x. 2 root root 60 May 14 13:22 65de4a13fc7cf28b4376e65efa31c5c3805e18da4eb01ad0c8b8801f4a10bc16
drwxr-xr-x. 2 root root 60 May 14 13:23 4970b995ce6435831c0e700f7c3ce6363235fbd15333af2dfb60a133ee8e9226
drwxr-xr-x. 2 root root 60 May 18 11:24 edaa232fba2214030847b7f6f57b1c05cf92a7c607332495bfab21738ceb61e1
drwxr-xr-x. 2 root root 60 May 18 16:16 09dab15bb87d1273fce7d786b2443a1d68a6cdcd397443fed4b7ac1bac298d30
drwxr-xr-x. 2 root root 60 May 18 17:20 8a622b1faf8c1df7a1f97e8fa4b841803160a800d04f6af539044f6f22ae9a38
drwxr-xr-x. 2 root root 60 May 19 14:15 b823d1a073894fdb70f7a8846f4d14eb8a2f2842f047a3d8ad847343d576c2e2
drwxr-xr-x. 2 root root 60 May 20 09:38 7c3c8e970803c9487f1f8884eb17603efb442a31b6f2082f3b26592311264e92
drwxr-xr-x. 2 root root 60 May 20 12:31 9d36120c6870e16ff7917b05bbbb84061c70e3135e40b8a10d58b1d6f50b23a3
.....
But when I run docker images command it only show:
root# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
Images1 latest f9a1d33ce592 14 hours ago 2.32 GB
Images2 latest 09dab15bb87d 9 days ago 877.2 MB
So here is my question: If something that I went wrong here since it won't delete the images folders under the /var/lib/docker-registry/images each time when I run docker rmi?
And if I can manually delete the images folders under /var/lib/docker-registry/images?
Some additional information:
I had found out under the /var/lib/docker-registry/repositories/library/ directory, there include multiple images and most of them that I already used >docker rmi to deleted, but when I cd into one of the deleted image folder, it included multiple files as:
[/var/lib/docker-registry/repositories/library/deleted-images1]
root# ls -latr
total 20
-rw-r--r--. 1 root root 159 May 27 17:39 taglatest_json
-rw-r--r--. 1 root root 64 May 27 17:39 tag_latest
-rw-r--r--. 1 root root 159 May 27 17:39 json
drwxr-xr-x. 2 root root 75 May 27 17:39 .
-rw-r--r--. 1 root root 304 May 27 17:39 _index_images
drwxr-xr-x. 14 root root 4096 May 28 10:15 ..
And in _index_images file which contain different ids as:
[{"id": "4970b995..."}, {"id": "65de4a13...."}, {"id": "f9a1d33...."}, {"id": "09dab15...."}]
So just wondering if this is caused the images folders didn't deleted from /var/lib/docker-registry/images directory when run docker rmi?
Is it safe for me to manually delete old images folders from both /var/lib/docker-registry/images and /var/lib/docker-registry/repositories/library folder?
When you build an image, each line of the Dockerfile creates an intermediate image or layer that is not deleted until all other images built from that layer are also deleted. You can see all of these intermediate image layers with docker images -a, and these should be the same as the contents of /var/lib/docker-registry/images. (Either way, I would strongly advise against manually deleting entries from this directory.)
#!/bin/bash
echo "clean containers ..."
docker ps --filter status=exited -aq | xargs --no-run-if-empty docker rm -vf
echo "clean images ..."
docker images --quiet --filter=dangling=true | xargs --no-run-if-empty docker rmi -f
You can try using the below command. It cleans most of the visible parts that my make your docker images list too bulky
docker images -a | grep '<none>'| awk '{print $3}' | xargs --no-run-if-empty docker rmi