Situation and Problem
I am trying to follow this guide on "how to make your own linuxkit with docker for mac", where you can add some kernel modules usually not present in docker images.
After a lot of reading and testing I am failing to do the simplest (one would think) test case in the repository:
linuxkit/test/cases/020_kernel/011_kmod_4.9.x/
https://github.com/linuxkit/linuxkit/tree/master/test/cases/020_kernel/011_kmod_4.9.x
checking the container for the linux kernel-version and config:
... host$ docker run -it --rm -v /:/host -v $(pwd):/macos alpine:latest
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
bdf0201b3a05: Pull complete
Digest: sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03576aa5580d6f4182a49c5fe1913
Status: Downloaded newer image for alpine:latest
/ # / # uname -a
/bin/sh: /: Permission denied
/ #
/ #
/ # uname -a
Linux 029b8e5ada75 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 Linux
/ # cp /host/proc/config.gz /macos/
/ # exit
I went back in the github history to find the hash for my local linuxkit kernel version and modified the dockerfile of that example (or basically used the old one).
So far so good. The problem is, that if I try to do anything related to kernel modules (modinfo, modprobe, depmod, insmod), I will get errors like these:
modinfo: can't open '/lib/modules/4.9.125-linuxkit/modules.dep': No such file or directory
This is because that path simply does not exist in the container (there is not even a modules folder). That is also true if I were to check -- as above -- just in alpine:latest. So there doesn't seem to happen any magic in that dockerfile.
Question
Now I am completely puzzled and left stranded on what to do, hence my question ...
How to do the hello_world example from linuxkit/linuxkit ?
additional notes
The docs of the linuxkit-repository do not mention anything about that problem:
https://github.com/linuxkit/linuxkit/blob/master/docs/kernels.md#compiling-external-kernel-modules
For easy testing I am using
docker-compose
# build with
# docker-compose build
version: '3'
services:
linux-builder:
image: my_linux_kit
build:
context: .
dockerfile: my_linux_kit.dockerfile
# args:
# buildno: 1
privileged: true
And I even tricked it (by inserting by hand) into not showing any errors, but also not doing what I suppose the code should do:
... host$: docker exec -it 7a33fad37914 sh
/ # ls
bin dev hello_world.ko lib mnt root sbin sys usr
check.sh etc home media proc run srv tmp var
/ # /bin/busybox insmod hello_world.ko
/ #
Related
I'm running into an issue with MacOS Ventura where by all bind volumes - where I link a directory on my host machine to one on the container - created with docker-compose are empty. I've tested the same scripts on MacOS 12.4 and 12.6 and they work as expected giving me the same directory contents on the container as on the host, so it seems v13 changed some permission.
The docker-compose.yml file:
version: "3"
services:
bash:
image: ubuntu:latest
stdin_open: true
tty: true
volumes:
- ./:/app
command: "/bin/bash"
So this should be creating a directory on the container called /app and linking that to the host directory the compose file is in.
But when I start the container:
❯ docker-compose up --build
[+] Running 1/0
⠿ Container ruby-docker-bash-1 Created 0.0s
Attaching to ruby-docker-bash-1
And login, the /app directory is empty:
❯ docker exec -it ruby-docker-bash-1 /bin/bash
root#9644de175d48:/# cd app/
root#9644de175d48:/app# ls -la
total 4
drwxr-xr-x 2 root root 40 Feb 1 09:59 .
drwxr-xr-x 1 root root 4096 Feb 1 09:39 ..
The total 4 is really weird here as there are 4 files supposed to be there, but not accessible:
root#9644de175d48:/app# cat Gemfile
cat: Gemfile: No such file or directory
This is the directory contents on the host are:
❯ ls -la
.rw-r--r-- 2.7k paul 1 Feb 09:31 Dockerfile
.rw-r--r-- 3.9k paul 1 Feb 09:32 Gemfile
.rw-r--r-- 27k paul 1 Feb 09:32 Gemfile.lock
.rw-r--r-- 149 paul 1 Feb 10:00 docker-compose.yml
If anyone has any experience with what might be going wrong or how I can get past this absolute time-sink of an issue, I'd really appreciate it.
Thank you!
I figured it out. I use Colima on MacOS, as there is no MacOS VM by docker.
Then I found this comment on a Colima repo issue, https://github.com/abiosoft/colima/issues/500#issuecomment-1343103477, where a user had mentioned they weren't able to sync directories.
To fix volumes on MacOS, using Colima, I did the folowing:
colima delete # reset
colima start --mount-type 9p
This doesn't seem to be documented anywhere. I’ve been through the site, the readme.
But I did find this line of code inside of the Colima repo:
validMountTypes := map[string]bool{"9p": true, "sshfs": true}
if util.MacOS13OrNewer() {
validMountTypes["virtiofs"] = true
}
I’m in MacOS 13, so it seems like there are issues with virtiofs and not in the older 9p mount type.
I'm working on a build step that handles common deployment tasks in a Docker Swarm Mode cluster. As this is a common problem for us and for others, we've shared this build step as a BitBucket pipe: https://bitbucket.org/matchory/swarm-secret-pipe/
The pipe needs to use the docker command to work with a remote Docker installation. This doesn't work, however, because the docker executable cannot be found when the pipe runs.
The following holds true for our test repository pipeline:
The docker option is set to true:
options:
docker: true
The docker service is enabled for the build step:
main:
- step:
services:
- docker: true
Docker works fine in the repository pipeline itself, but not within the pipe.
Pipeline log shows the docker path being mounted into the pipe container:
docker container run \
--volume=/opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build \
--volume=/opt/atlassian/pipelines/agent/ssh:/opt/atlassian/pipelines/agent/ssh:ro \
--volume=/usr/local/bin/docker:/usr/local/bin/docker:ro \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/matchory/swarm-secret-pipe:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/matchory/swarm-secret-pipe \
--workdir=$(pwd) \
--label=org.bitbucket.pipelines.system=true \
radiergummi/swarm-secret-pipe:1.3.7#sha256:baf05b25b38f2a59b044e07f4ad07065de90257a000137a0e1eb71cbe1a438e5
The pipe is pretty standard and uses a recent Alpine image; nothing special in that regard. The PATH is never overwritten. Now for the fun part: If I do ls /usr/local/bin/docker inside the pipe, it shows an empty directory:
ls /usr/local/bin
total 16K
drwxr-xr-x 1 root root 4.0K May 13 13:06 .
drwxr-xr-x 1 root root 4.0K Apr 4 16:06 ..
drwxr-xr-x 2 root root 4.0K Apr 29 09:30 docker
ls /usr/local/bin/docker
total 8K
drwxr-xr-x 2 root root 4.0K Apr 29 09:30 .
drwxr-xr-x 1 root root 4.0K May 13 13:06 ..
ls: /usr/local/bin/docker/docker: No such file or directory
As far as I understand pipelines and Docker, /usr/local/bin/docker should be the docker binary file. Instead, it appears to be an empty directory for some reason.
What is going on here?
I've also looked at other, official, pipes. They don't do anything differently, but seem to be using the docker command just fine (eg. the Azure pipe).
After talking to BitBucket support, I solved the issue. As it turns out, if the docker context is changed, any docker command is sent straight to the remote docker binary, which (on our services) lives at a different path than in BitBucket Pipelines!
As we changed the docker context before using the pipe, and the docker instance mounted into the pipe still has the remote context set, but the pipe searches for the docker binary at another place, the No such file or directory error is thrown.
TL;DR: Always restore the default docker host/context before passing control to a pipe, e.g.:
script:
- export DEFAULT_DOCKER_HOST=$DOCKER_HOST
- unset DOCKER_HOST
- docker context create remote --docker "host=ssh://${DEPLOY_SSH_USER}#${DEPLOY_SSH_HOST}"
- docker context use remote
# do your thing
- export DOCKER_HOST=$DEFAULT_DOCKER_HOST # <------ restore the default host
- pipe: matchory/swarm-secret-pipe:1.3.16
I am Dockerising an old project. A feature in the project pulls in user-specified Git repos, and since the size of a repo could cause the filing system to be overwhelmed, I created a local filing system of a fixed size, and then mounted it. This was intended to prevent the web host from having its file system filled up.
The general approach is this:
IMAGE=filesystem/image.img
MOUNT_POINT=filesystem/mount
SIZE=20
PROJECT_ROOT=`pwd`
# Number of M to set aside for this filing system
dd if=/dev/zero of=$IMAGE bs=1M count=$SIZE &> /dev/null
# Format: the -F permits creation even though it's not a "block special device"
mkfs.ext3 -F -q $IMAGE
# Mount if the filing system is not already mounted
$MOUNTCMD | cut -d ' ' -f 3 | grep -q "^${PROJECT_ROOT}/${MOUNT_POINT}$"
if [ $? -ne 0 ]; then
# -p Create all parent dirs as necessary
mkdir -p $MOUNT_POINT
/bin/mount -t ext3 $IMAGE $MOUNT_POINT
fi
This works fine in a Linux local or remote VM. However, I'd like to run this shell code, or something like it, inside a container. Part of the reason I'd like to do that is to contain all fiddly stuff inside a container, so that building a new host machine is as kept as simple as possible (in my view, setting up custom mounts and cron-restart rules on the host works against that).
So, this command does not work inside a container ("filesystem" is an on-host Docker volume)
mount -t ext3 filesystem/image.img filesystem/mount
mount: can't setup loop device: No space left on device
It also does not work on a container folder ("filesystem2" is a container directory):
dd if=/dev/zero of=filesystem2/image.img bs=1M count=20
mount -t ext3 filesystem2/image.img filesystem2/mount
mount: can't setup loop device: No space left on device
I wonder whether containers just don't have the right internal machinery to do mounting, and thus whether I should change course. I'd prefer not to spend too much time on this (I'm just moving a project to a Docker-only server) which is why I would like to get mount working if I can.
Other options
If that's not possible, then a size-limited Docker volume, that works with both Docker and Swarm, may be an alternative I'd need to look into. There are conflicting reports on the web as to whether this actually works (see this question).
There is a suggestion here to say this is supported in Flocker. However, I am hesitant to use that, as it appears to be abandoned, presumably having been affected by ClusterHQ going bust.
This post indicates I can use --storage-opt size=120G with docker run. However, it does not look like it is supported by docker service create (unless perhaps the option has been renamed).
Update
As per the comment convo, I made some progress; I found that adding --privileged to the docker run enables mounting, at the cost of removing security isolation. A helpful commenter says that it is better to use the more fine-grained control of --cap-add SYS_ADMIN, allowing the container to retain some of its isolation.
However, Docker Swarm has not yet implemented either of these flags, so I can't use this solution. This lengthy feature request suggests to me that this feature is not going to be added in a hurry; it's been pending for two years already.
You won't be able to safely do this inside of a container. Docker removes the mount privilege from containers because using this you could mount the host filesystem and escape the container. However, you can do this outside of the container and mount the filesystem into the container as a volume using the default local driver. The size option isn't supported by most filesystems, tmpfs being one of the few exceptions. Most of them use the size of the underlying device which you defined with the image file creation command:
dd if=/dev/zero of=filesystem/image.img bs=1M count=$SIZE
I had trouble getting docker to create the loop device dynamically, so here's the process to create it manually:
$ sudo losetup --find --show ./vol-image.img
/dev/loop0
$ sudo mkfs -t ext3 /dev/loop0
mke2fs 1.43.4 (31-Jan-2017)
Creating filesystem with 10240 1k blocks and 2560 inodes
Filesystem UUID: 25c95fcd-6c78-4b8e-b923-f808517b28df
Superblock backups stored on blocks:
8193
Allocating group tables: done
Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done
When defining the volume mount options are passed almost verbatim from the mount command you run on the command line:
docker volume create --driver local --opt type=ext3 \
--opt device=filesystem/image.img app_vol
docker service create --mount type=volume,src=app_vol,dst=/filesystem/mount ...
or in a single service create command:
docker service create \
--mount type=volume,src=app_vol,dst=/filesystem/mount,volume-driver=local,volume-opt=type=ext3,volume-opt=device=filesystem/image.img ...
With docker run, the command looks like:
$ docker run -it --rm --mount type=volume,dst=/data,src=ext3vol,volume-driver=local,volume-opt=type=ext3,volume-opt=device=/dev/loop0 busybox /bin/sh
/ # ls -al /data
total 17
drwxr-xr-x 3 root root 1024 Sep 19 14:39 .
drwxr-xr-x 1 root root 4096 Sep 19 14:40 ..
drwx------ 2 root root 12288 Sep 19 14:39 lost+found
The only prerequisite is that you create this file and loop device before creating the service, and that this file is accessible wherever the service is scheduled. I would also suggest making all of the paths in these commands fully qualified rather than relative to the current directory. I'm pretty sure there are a few places that relative paths don't work.
I have found a size-limiting solution I am happy with, and it does not use the Linux mount command at all. I've not implemented it yet, but the tests documented below are satisfying enough. Readers may wish to note the minor warnings at the end.
I had not tried mounting Docker volumes prior to asking this question, since part of my research stumbled on a Stack Overflow poster casting doubt on whether Docker volumes can be made to respect a size limitation. My test indicates that they can, but you may wish to test this on your own platform to ensure it works for you.
Size limit on Docker container
The below commands have been cobbled together from various sources on the web.
To start with, I create a volume like so, with a 20m size limit:
docker volume create \
--driver local \
--opt o=size=20m \
--opt type=tmpfs \
--opt device=tmpfs \
hello-volume
I then create an Alpine Swarm service with a mount on this container:
docker service create \
--mount source=hello-volume,target=/myvol \
alpine \
sleep 10000
We can ensure the container is mounted by getting a shell on the single container in this service:
docker exec -it amazing_feynman.1.lpsgoyv0jrju6fvb8skrybqap
/ # ls - /myvol
total 0
OK, great. So, while remaining in this shell, let's try slowly overwhelming this disk, in 5m increments. We can see that it fails on the fifth try, which is what we would expect:
/ # cd /myvol
/myvol # ls
/myvol # dd if=/dev/zero of=image1 bs=1M count=5
5+0 records in
5+0 records out
/myvol # dd if=/dev/zero of=image2 bs=1M count=5
5+0 records in
5+0 records out
/myvol # ls -l
total 10240
-rw-r--r-- 1 root root 5242880 Sep 16 13:11 image1
-rw-r--r-- 1 root root 5242880 Sep 16 13:12 image2
/myvol # dd if=/dev/zero of=image3 bs=1M count=5
5+0 records in
5+0 records out
/myvol # dd if=/dev/zero of=image4 bs=1M count=5
5+0 records in
5+0 records out
/myvol # ls -l
total 20480
-rw-r--r-- 1 root root 5242880 Sep 16 13:11 image1
-rw-r--r-- 1 root root 5242880 Sep 16 13:12 image2
-rw-r--r-- 1 root root 5242880 Sep 16 13:12 image3
-rw-r--r-- 1 root root 5242880 Sep 16 13:12 image4
/myvol # dd if=/dev/zero of=image5 bs=1M count=5
dd: writing 'image5': No space left on device
1+0 records in
0+0 records out
/myvol #
Finally, let's see if we can get an error by overwhelming the disk in one go, in case the limitation only applies to newly opened file handles in a full disk:
/ # cd /myvol
/ # rm *
/myvol # dd if=/dev/zero of=image1 bs=1M count=21
dd: writing 'image1': No space left on device
21+0 records in
20+0 records out
It turns out we can, so that looks pretty robust to me.
Nota bene
The volume is created with a type and a device of "tmpfs", which sounded to me worryingly like a RAM disk. I've successfully checked that the volume remains connected and intact after a system reboot, so it looks good to me, at least for now.
However, I'd say that when it comes to organising your data persistence systems, don't just copy what I have. Make sure the volume is robust enough for your use case before you put it into production, and of course, make sure you include it in your back-up process.
(This is for Docker version 18.06.1-ce, build e68fc7a).
When I start nexus3 in a docker container I get the following error messages.
$ docker run --rm sonatype/nexus3:3.8.0
Warning: Cannot open log file: ../sonatype-work/nexus3/log/jvm.log
Warning: Forcing option -XX:LogFile=/tmp/jvm.log
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file ../sonatype-work/nexus3/log/jvm.log due to Permission denied
Unable to update instance pid: Unable to create directory /nexus-data/instances
/nexus-data/log/karaf.log (Permission denied)
Unable to update instance pid: Unable to create directory /nexus-data/instances
It indicates that there is a file permission issue.
I am using Red Hat Enterprise Linux 7.5 as host machine and the most recent docker version.
On another machine (ubuntu) it works fine.
The issue occurs in the persistent volume (/nexus-data). However, I do not mount a specific volume and let docker use a anonymous one.
If I compare the volumes on both machines I can see the following permissions:
For Red Hat, where it is not working is belongs to root.
$ docker run --rm sonatype/nexus3:3.8.0 ls -l /nexus-data
total 0
drwxr-xr-x. 2 root root 6 Mar 1 00:07 etc
drwxr-xr-x. 2 root root 6 Mar 1 00:07 log
drwxr-xr-x. 2 root root 6 Mar 1 00:07 tmp
On ubuntu, where it is working it belongs to nexus. Nexus is also the default user in the container.
$ docker run --rm sonatype/nexus3:3.8.0 ls -l /nexus-data
total 12
drwxr-xr-x 2 nexus nexus 4096 Mar 1 00:07 etc
drwxr-xr-x 2 nexus nexus 4096 Mar 1 00:07 log
drwxr-xr-x 2 nexus nexus 4096 Mar 1 00:07 tmp
Changing the user with the options -u is not an option.
I could solve it by deleting all local docker images: docker image prune -a
Afterwards it downloaded the image again and it worked.
This is strange because I also compared the fingerprints of the images and they were identical.
An example of docker-compose for Nexus :
version: "3"
services:
#Nexus
nexus:
image: sonatype/nexus3:3.39.0
expose:
- "8081"
- "8082"
- "8083"
ports:
# UI
- "8081:8081"
# repositories http
- "8082:8082"
- "8083:8083"
# repositories https
#- "8182:8182"
#- "8183:8183"
environment:
- VIRTUAL_PORT=8081
volumes:
- "./nexus/data/nexus-data:/nexus-data"
Setup the volume :
mkdir -p ./nexus/data/nexus-data
sudo chown -R 200 nexus/ # 200 because it's the UID of the nexus user inside the container
Start Nexus
sudo docker-compose up -d
hf
You should attribute correct right to the folder where the persistent volume is located.
chmod u+wxr -R <folder of /nexus-data volumes>
Be carefull, if you execute previous command, it would give write, read and execution right to all users. If you want to give more restricted right, you should modify the command.
I have two Docker images for my documentation files: docs:v1 and docs:v2.
They just contain some files in /docs exposed as a VOLUME:
# docker run --rm docs:v1 cat /docs/doc.txt
Version1
# docker run --rm docs:v2 cat /docs/doc.txt
Version2
And I have my app described in this docker-compose.yml (using v1.4):
app:
image: "busybox"
command: /bin/sh -c "cat /docs/doc.txt && echo `date` >> /logs/log.txt"
volumes:
- "/logs"
volumes_from:
- "docs"
docs:
image: "docs:v1"
So basically my app prints the content of the docs and echo the current date in a log file. The log file is also in a VOLUME.
=> What I want it just to be able to update the docs to docs:v2, see that it prints "Version2" as expected and keep the logs intact.
First run:
# docker-compose up
Creating tmp_docs_1...
Creating tmp_app_1...
Attaching to tmp_docs_1, tmp_app_1
app_1 | Version1
...
# docker run --rm --volumes-from tmp_app_1 busybox cat /logs/log.txt
Tue Aug 25 22:09:11 UTC 2015
It works as expected: Prints the Version1 documentation and echo in the logs.
Next I update the yml file with : image: "docs:v2". Then restart my app:
# docker-compose up
Recreating tmp_docs_1...
Recreating tmp_app_1...
Attaching to tmp_docs_1, tmp_app_1
app_1 | Version1
...
# docker run --rm --volumes-from tmp_app_1 busybox cat /logs/log.txt
Tue Aug 25 22:09:11 UTC 2015
Tue Aug 25 22:10:26 UTC 2015
The logs have been updated, that's fine, but my doc is still in Version1 !
It might be surprising, but that's actually the expected behavior. According to the docker docs: "Changes to a data volume will not be included when you update an image."
Right, but I want to be able to see my updated docs, so let's try to delete the docs container and volume:
# docker-compose rm -v docs
Removing tmp_docs_1... done
# docker-compose up
Creating tmp_docs_1...
Starting tmp_app_1...
Attaching to tmp_docs_1, tmp_app_1
app_1 | Version1
...
No luck .. still in Version 1. That's because the app container still points to the old Version1 volume. So let's try to delete the app as well (just the app, not the volumes this time):
# docker-compose rm app
Removing tmp_app_1... done
# docker-compose up
Starting tmp_docs_1...
Creating tmp_app_1...
Attaching to tmp_docs_1, tmp_app_1
app_1 | Version2
Version2: it worked ! Let's check the logs:
# docker run --rm --volumes-from tmp_app_1 busybox cat /logs/log.txt
Tue Aug 25 22:19:21 UTC 2015
Ach ! My old logs are gone.
So here's the question again: how can I update the docs image, see the change in my app and still be able to keep the logs upon restarts ?
You will want to map your volumes to your actual FS. Right now you are creating volumes in the container by using the volume command, these volumes bypass the UFS and persist even if the container is deleted which is why you have to delete the volume to make it act properly. Map the volumes to an external folder on the host O/S and then when you upgrade your apps the logs will be there but the app upgraded.
To be clear about what is occurring:
You create the intial app > volume inits > logs are saved.
Upgrade app > logs are not updated (Changes to a data volume will not be included when you update an image.) (expected behavior).
Delete data volume > upgrade app > new logs are there old logs are gone because they were deleted.
To resolve mount the volumes to the host O/S so they persist but can be written to on upgrade.