As a background, I have a custom Ubuntu LiveUSB that will automatically boot into "Try it" and the OS will have pre-installed apps that I have burned into the ISO itself.
It works great, but I keep running into problems automating the process.
Rather than doing it by hand every time, (because my bash scripts keep getting different results when I try again for the first time in a while) I was thinking of generating a docker image with the unpacked files from the ISO ready for modification, then run a container with a script in a volume (docker run -v $(pwd)/bin:/data myimage /data/myscript.sh) that would modify the contents, pack it back up into an ISO and save the ISO in /data for me to grab and distribute.
FROM ubuntu:16.04
MAINTAINER Myself
ENV ISO_FILE="ubuntu-16.04.3-desktop-amd64.iso" \
OS_VERSION="16.04.3"
RUN apt-get update && apt-get install -y curl
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN apt-get install -y squashfs-tools genisoimage gnupg2 \
nodejs rsync build-essential libc6-dev-i386 \
wget
# Make directories
RUN mkdir /data
RUN mkdir -p /root/workspace
# Download ubuntu iso
WORKDIR /root/workspace
RUN wget http://releases.ubuntu.com/$OS_VERSION/$ISO_FILE
RUN wget http://releases.ubuntu.com/$OS_VERSION/SHA256SUMS
RUN wget http://releases.ubuntu.com/$OS_VERSION/SHA256SUMS.gpg
# Check hash (default /bin/sh errors out)
RUN /bin/bash -c "sha256sum -c <(grep $ISO_FILE SHA256SUMS)"
# Check signatures
RUN gpg2 --keyserver hkp://keyserver.ubuntu.com --recv-keys 0xFBB75451 0xEFE21092
RUN gpg2 --verify SHA256SUMS.gpg SHA256SUMS
# Create mount
RUN mkdir mnt
# Here is where the docker build fails
RUN mount -o loop $ISO_FILE mnt
# Extract official DVD
RUN mkdir extract-cd
RUN rsync --exclude=/casper/filesystem.squashfs -a mnt/ extract-cd
RUN unsquashfs mnt/casper/filesystem.squashfs
RUN mv squashfs-root edit
RUN umount mnt
# Insert buildscript and make it executable
COPY bin/buildscript.sh /root/workspace/edit/buildscript.sh
RUN chmod +x edit/buildscript.sh
# Prepare to chroot into unsquashed ubuntu image, and run buildscript.sh
RUN mount -o bind /run/ edit/run
RUN mount --bind /dev/ edit/dev
RUN chroot edit/ ./buildscript.sh
# unmount the mountpoints and delete the buildscript.
RUN umount edit/dev
RUN umount edit/run
RUN rm edit/buildscript.sh
And the buildscript.sh I run in chroot inside the builder (or fail to run) is:
#!/bin/bash
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devpts none /dev/pts
export HOME=/root
export LC_ALL=C
add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc) universe multiverse"
curl -sL https://deb.nodesource.com/setup_8.x | bash -
apt install -y nodejs
apt upgrade -y
apt install -y chromium-browser git
apt install -y language-pack-ja language-pack-gnome-ja language-pack-ja-base language-pack-gnome-ja-base
localectl set-locale LANG=ja_JP.UTF-8 LANGUAGE="ja_JP:ja"
source /etc/default/locale
mkdir src
apt autoclean
rm -rf /tmp/* ~/.bash_history
umount /proc || umount -lf /proc
umount /sys
umount /dev/pts
exit
Since this didn't work, I found online that build-run-commit method might work... so I changed the end of the dockerfile to the following
# Create mount
RUN mkdir mnt
RUN mkdir extract-cd
COPY bin/buildscript.sh /root/workspace/buildscript.sh
COPY bin/build_run_step2.sh /root/workspace/build_run_step2.sh
RUN chmod +x buildscript.sh
RUN chmod +x build_run_step2.sh
and then the "run" step of build run commit is the build_run_step2.sh which has the following (run with --privileged)
#!/bin/bash
cd /root/workspace
mount -o loop $ISO_FILE mnt
# Extract official DVD
rsync --exclude=/casper/filesystem.squashfs -a mnt/ extract-cd
unsquashfs mnt/casper/filesystem.squashfs
mv squashfs-root edit
umount mnt
mv ./buildscript.sh edit/buildscript.sh
# Prepare to chroot into unsquashed ubuntu image, and run buildscript.sh
mount -o bind /run/ edit/run
mount --bind /dev/ edit/dev
chroot edit/ ./buildscript.sh
# unmount the mountpoints and delete the buildscript.
umount edit/dev
umount edit/run
rm edit/buildscript.sh
Which works... but then I run into a problem:
Running apt-get update gets errors:
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
and checking ping gets me "no host found" while chrooted.
So One major question and one smaller question (if the major question has no answer):
How can I use docker to create an image with an opened up liveCD ready for customizing and then use docker run on that image to chroot, modify, repackage, and extract the new iso? (I know the commands to do that normally, so rather, I am wondering if/why all these things are not working in docker... aka what are the limitations of chrooting in docker?)
How can I get the chroot system within the container to reach dns so it can run the updates via URLS? (I attempted ping 8.8.8.8 from within the chroot in the container and the pings were coming back fine.)
So incase anyone finds this post. The way to resolve the dns issue is to make sure your resolv.conf file in the chroot is actually pointing to a proper dns servers. Some apps like cubic already do this for you.
This is likely to happen if you have not updated the sources list. Try:
sudo apt update
Related
I have a Dockerfile that extends the Apache Airflow 2.5.1 base image. What I want to do is be able to use docker inside my airflow containers (i.e. docker-in-docker) for testing and evaluation purposes.
My docker-compose.yaml has the following mount:
volumes:
- /var/run/docker.sock:/var/run/docker.sock
My Dockerfile looks as follows:
FROM apache/airflow:2.5.1
USER root
RUN apt-get update && apt-get install -y ca-certificates curl gnupg lsb-release nano
RUN mkdir -p /etc/apt/keyrings
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
RUN echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update
RUN apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
RUN groupadd -f docker
RUN usermod -a -G docker airflow
RUN service docker start
USER airflow
Basically:
Install docker.
Add the airflow user to the docker group.
Start the docker service.
Continue as airflow.
Unfortunately, this does not work. During RUN service docker start, I encounter the following error:
Step 11/12 : RUN service docker start
---> Running in 77e9b044bcea
mkdir: cannot create directory ‘cpuset’: Read-only file system
I have another Dockerfile for building a local jenkins image, which looks as follows:
FROM jenkins/jenkins:lts-jdk11
USER root
RUN apt-get update && apt-get install -y ca-certificates curl gnupg lsb-release nano
RUN mkdir -p /etc/apt/keyrings
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
RUN echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update
RUN apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
RUN groupadd -f docker
RUN usermod -a -G docker jenkins
RUN service docker start
USER jenkins
I.e. it is exactly the same, except that I am using the jenkins user. Building this image works.
I have not set any extraneous permission on my /var/run/docker.sock:
$ ls -la /var/run/docker.sock
srw-rw---- 1 root docker 0 Jan 18 17:14 /var/run/docker.sock
My questions are:
Why does RUN service start docker not work when building my airflow image?
Why does the exact same command in my jenkins Dockerfile work?
I've tried most of the answers to similar questions, e.g. here and here, but they have unfortunately not helped.
I'd rather try to avoid the chmod 777 /var/run/docker.sock solution if at all possible, and it should be since my jenkins image can build correctly...
Just delete the RUN service start docker line.
The docker CLI tool needs to connect to a Docker daemon, which it normally does through the /var/run/docker.sock Unix socket file. Bind-mounting the socket into the container is enough to make the host's Docker daemon accessible; you do not need to separately start Docker in the container.
There are several issues with the RUN service ... line specifically. Docker has a kind of complex setup internally, and some of the things it does aren't normally allowed in a container; that's probably related to the "cannot create directory" error. In any case, a Docker image doesn't persist running processes, so if you were able to start Docker inside the build, it wouldn't still be running when the container eventually ran.
More conceptually, a container doesn't "run services", it is a wrapper around only a single process (and its children). Commands like service or systemctl often won't work the way you expect, and I'd generally avoid them in a Docker context.
I'm not sure that I am using the correct terminology here but I thought I would at least try to explain my issue:
Goal:
I want to create a docker image that people can use that will have all the files preloaded and ready to go because they will have to make an edit to a config.json file basically. So I want people to be able to map their docker-compose (or just CLI) to something like "/my/host/path/config:/config" and when they spin up the image, all the files will be available at that location on their host machine in persistent storage.
Issue:
When you spin up the image for the first time, the directories get created on the host but there are no files for end-user to modify. So they are left with manually copying files into this folder to make it work and that is not acceptable in my humble opinion.
Quick overview of the image:
Python script that uses Selenium to perform some actions on a website
Dockerfile:
FROM python:3.9.2
RUN apt-get update
RUN apt-get install -y cron wget apt-transport-https
# install google chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
RUN apt-get -y update
RUN apt-get install -y google-chrome-stable
# install chromedriver
RUN apt-get install -yqq unzip
RUN wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`/chromedriver_linux64.zip
RUN unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/
# Set TZ
RUN apt-get install -y tzdata
ENV TZ=America/New_York
RUN mkdir -p /config
COPY . /config
WORKDIR /config
RUN pip install --no-cache-dir -r requirements.txt
# set display port to avoid crash
ENV DISPLAY=:99
# Custom Env Vars
ENV DOCKER_IMAGE=true
# Setup Volumes
VOLUME [ "/config" ]
# Run the command on container startup
CMD ["python3", "-u", "/config/RunTests.py"]
Any help would be greatly appreciated.
This is not how docker bind mount work. Docker bind mounts use the mount system call under the hood and it hides the content of the folder inside your containers when a host folder in bind mount.
Running this command docker run -v /my/config:/config container will always hide (override) the content inside your container.
On the contrary if you use empty docker volumes(created by docker volume command), Docker will copy the files to the volume before binding it.
So docker run config_volume:/config container will copy you config files into you volume the first time. Then you can use the volume with volume-from or mount it on another container.
To learn more about this take a look at this issue.
Another workaround is to bind you volume to a folder while creating it. More info here.
docker volume create --driver local \
--opt type=none \
--opt device=$configVolumePath \
--opt o=bind \
config_vol
For me the best solution is to copy or symlink your configuration files on container startup.
You can do so by modifying your add the cp or ln command into your entrypoint script.
I have the following folder structure
db
- build.sh
- Dockerfile
- file.txt
build.sh
PGUID=$(id -u postgres)
PGGID=$(id -g postgres)
CS=$(lsb_release -cs)
docker build --build-arg POSTGRES_UID=${PGUID} --build-arg POSTGRES_GID=${PGGID} --build-arg LSB_CS=${CS} -t postgres:1.0 .
docker run -d postgres:1.0 sh -c "cp file.txt ./file.txt"
Dockerfile
FROM ubuntu:19.10
RUN apt-get update
ARG LSB_CS=$LSB_CS
RUN echo "lsb_release: ${LSB_CS}"
RUN apt-get install -y sudo \
&& sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt eoan-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
RUN apt-get install -y wget \
&& apt-get install -y gnupg \
&& wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
sudo apt-key add -
RUN apt-get update
RUN apt-get install tzdata -y
ARG POSTGRES_GID=128
RUN groupadd -g $POSTGRES_GID postgres
ARG POSTGRES_UID=122
RUN useradd -r -g postgres -u $POSTGRES_UID postgres
RUN apt-get update && apt-get install -y postgresql-10
RUN locale-gen en_US.UTF-8
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/10/main/pg_hba.conf
RUN echo "listen_addresses='*'" >> /etc/postgresql/10/main/postgresql.conf
EXPOSE 5432
CMD ["pg_ctlcluster", "--foreground", "10", "main", "start"]
file.txt
"Hello Hello"
Basically i want to be able to build my image, start my container and copy file.txt in my local to the docker container.
I tried doing it like this docker run -d postgres:1.0 sh -c "cp file.txt ./file.txt" but it doesn't work. I have also tried other options as well but also not working.
At the moment when i run my script sh build.sh, it runs everything and even starts a container but doesn't copy over that file to the container.
Any help on this is appreciated
Sounds like what you want is a mounting the file into a location of your docker container.
You can mount a local directory into your container and access it from the inside:
mkdir /some/dirname
copy filet.txt /some/dirname/
# run as demon, mount /some/dirname to /directory/in/container, run sh
docker run -d -v /some/dirname:/directory/in/container postgres:1.0 sh
Minimal working example:
On windows host:
d:\>mkdir d:\temp
d:\>mkdir d:\temp\docker
d:\>mkdir d:\temp\docker\dir
d:\>echo "SomeDataInFile" > d:\temp\docker\dir\file.txt
# mount one local file to root in docker container, renaming it in the process
d:\>docker run -it -v d:\temp\docker\dir\file.txt:/other_file.txt alpine
In docker container:
/ # ls
bin etc lib mnt other_file.txt root sbin sys usr
dev home media opt proc run srv tmp var
/ # cat other_file.txt
"SomeDataInFile"
/ # echo 32 >> other_file.txt
/ # cat other_file.txt
"SomeDataInFile"
32
/ # exit
this will mount the (outside) directory/file as folder/file inside your container. If you specify a directory/file inside your docker that already exists it will be shadowed.
Back on windows host:
d:\>more d:\temp\docker\dir\file.txt
"SomeDataInFile"
32
See f.e Docker volumes vs mount bind - Use cases on Serverfault for more info about ways to bind mount or use volumes.
I'm trying to dockerize an AEM 6.0 installation, and this is the Dockerfile for my author.
from centos:latest
COPY aem6.0-author-p4502.jar /AEM/aem/author/aem6.0-author-p4502.jar
COPY license.properties /AEM/aem/author/license.properties
RUN yum install dnsmasq -y
RUN systemctl enable dnsmasq
RUN yum install initscripts -y
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*;\
rm -f /lib/systemd/system/sockets.target.wants/*initctl*;\
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
WORKDIR /AEM/aem/author
RUN yum install wget -y
RUN wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/jdk-8u151-linux-x64.rpm"
RUN yum localinstall jdk-8u151-linux-x64.rpm -y
RUN java -XX:MaxPermSize=256m -Xmx512M -jar aem6.0-author-p4502.jar -unpack
COPY aem6 /etc/init.d/aem6
RUN chkconfig --add aem6
RUN yum -y install initscripts && yum update -y & yum clean all
RUN chown -R $USER:$(id -G) /etc/init.d
RUN chmod 777 -R /etc/init.d/aem6
RUN systemctl enable aem6.service
RUN service aem6 start
VOLUME /sys/fs/cgroup
CMD /usr/sbin/init
The build fails on starting the service, with the error - failed to get Dbus connection error. I haven't been able to figure out how to fix it.
I've tried these
- https://github.com/CentOS/sig-cloud-instance-images/issues/45
- https://hub.docker.com/_/centos/
Here, the problem is that you're trying to start the aem service during the "build" phase, with this statement:
RUN service aem6 start
This is problematic for a number of reasons. First, you're building an image. Starting a service at this stage is pointless...when the build process completes, nothing is running. An image is just a collection of files. You don't have any processes until you boot a container, at which point your CMD and ENTRYPOINT influence what is running.
Another problem is that at this stage, nothing else is running inside the container environment. The service command in this case is trying to communicate with systemd using the dbus api, but neither of those services are running.
There is a third slightly more subtle problem: the solution you've chosen relies on systemd, the standard CentOS process manager, and as far as things go you've configured things correctly (by both enabling the service with systemctl enable ... and by starting /sbin/init in your CMD statement). However, running systemd in a container can be tricky, although it is possible. In the past, systemd required the container to run with the --privileged flag; I'm not sure if this is necessary any more.
If you weren't running multiple processes (dnsmasq and aem) inside the container, the simplest solution would be to start the aem service directly, rather than relying on a process manager. This would reduce your Dockerfile to something like:
FROM centos:latest
COPY aem6.0-author-p4502.jar /AEM/aem/author/aem6.0-author-p4502.jar
COPY license.properties /AEM/aem/author/license.properties
WORKDIR /AEM/aem/author
RUN yum install wget -y
RUN wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/jdk-8u151-linux-x64.rpm"
RUN yum localinstall jdk-8u151-linux-x64.rpm -y
RUN java -XX:MaxPermSize=256m -Xmx512M -jar aem6.0-author-p4502.jar -unpack
CMD some commandline to start aem
If you actually require dnsmasq, you could run it in a second container (potentially sharing the same network environment as the aem container).
I am trying to mount the current working directory onto Docker container but isn't working. Here is my Dockerfile
FROM ubuntu:14.04.3
MAINTAINER Upendra Devisetty
RUN apt-get update && apt-get install -y g++ \
make \
git \
zlib1g-dev \
python \
wget \
curl \
python-matplotlib
ENV BINPATH /usr/bin
ENV HISAT2GIT https://upendra_35#bitbucket.org/upendra_35/evolinc.git
RUN git clone "$HISAT2GIT"
RUN chmod +x evolinc/evolinc-part-I.sh && cp evolinc/evolinc-part-I.sh $BINPATH
RUN wget -O- http://cole-trapnell-lab.github.io/cufflinks/assets/downloads/cufflinks-2.2.1.Linux_x86_64.tar.gz | tar xzvf -
RUN wget -O- https://github.com/TransDecoder/TransDecoder/archive/2.0.1.tar.gz | tar xzvf -
RUN wget -O- http://seq.cs.iastate.edu/CAP3/cap3.linux.x86_64.tar | tar vfx -
RUN curl ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/ncbi-blast-2.2.31+-x64-linux.tar.gz > ncbi-blast-2.2.31+-x64-linux.tar.gz
RUN tar xvf ncbi-blast-2.2.31+-x64-linux.tar.gz
RUN wget -O- http://ftp.mirrorservice.org/sites/download.sourceforge.net/pub/sourceforge/q/qu/quast/quast-3.0.tar.gz | tar zxvf -
RUN curl -L http://cpanmin.us | perl - App::cpanminus
RUN cpanm URI/Escape.pm
ENV PATH /CAP3/:$PATH
ENV PATH /ncbi-blast-2.2.31+/bin/:$PATH
ENV PATH /quast-3.0/:$PATH
ENV PATH /cufflinks-2.2.1.Linux_x86_64/:$PATH
ENV PATH /TransDecoder-2.0.1/:$PATH
ENTRYPOINT ["/usr/bin/evolinc-part-I.sh"]
CMD ["-h"]
When i run the following to mount the current working directory to make sure everything is doing ok, what i see is that all those dependencies are getting installed in the current working directory.
docker run --rm -v $(pwd):/working-dir -w /working-dir ubuntu/evolinc:2.0 -c cuffcompare_out_annot_no_annot.combined.gtf -g Brassica_rapa_v1.2_genome.fa -r Brassica_rapa_v1.2_cds.fa -b TE_RNA_transcripts.fa
I thought, they should only be installed on the container and only the output is going to generate in the current working directory. Sorry, i am very new to Docker and i would need some help with this....
Mouting a volume in docker (-v) allows a container to share directories/volumes with the host. Therefore when changing the volume you are in fact changing the mounted directory. If you wanted to copy some files, rather than point at them, you may need to build your own container and use the COPY or ADD instructions.
What is the difference between the `COPY` and `ADD` commands in a Dockerfile?