How to install RabbitMQ on Docker? - docker

I'm attempting to install RabbitMQ inside a Docker container using an Ubuntu 18.04 image for running unittests against it.
To install, I'm running the normal sudo apt-get install rabbitmq-server, and it appears to install fine, but when I attempt to start or communicate with the service, I get the error:
Error: unable to connect to node rabbit#b562da1810ce: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit#b562da1810ce]
rabbit#b562da1810ce:
* connected to epmd (port 4369) on b562da1810ce
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* Authentication failed (rejected by the remote node), please check the Erlang cookie
current node details:
- node name: 'rabbitmq-cli-69#b562da1810ce'
- home dir: /var/lib/rabbitmq
- cookie hash: YUZIPS6zyhfUBX5afdKGcw==
Researching the "please check the Erlang cookie" text gets me a ton of similar questions, none of which seem to apply to Docker or my situation.
I've tried deleting the ~/.erlang.cookie then restarting the service, and completely purging the package and reinstalling. Nothing's worked.
How do I run RabbitMQ inside Docker?
Edit: This is my install procedure.
root#b562da1810ce:$ sudo apt-get purge -yq rabbitmq-server
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
erlang-asn1 erlang-base erlang-corba erlang-crypto erlang-diameter erlang-edoc erlang-eldap erlang-erl-docgen erlang-eunit erlang-ic erlang-inets erlang-mnesia erlang-nox erlang-odbc erlang-os-mon erlang-parsetools erlang-public-key erlang-runtime-tools erlang-snmp erlang-ssh
erlang-ssl erlang-syntax-tools erlang-tools erlang-xmerl libodbc1
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
rabbitmq-server*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 5,678 kB disk space will be freed.
(Reading database ... 69832 files and directories currently installed.)
Removing rabbitmq-server (3.6.10-1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of stop.
(Reading database ... 69618 files and directories currently installed.)
Purging configuration files for rabbitmq-server (3.6.10-1) ...
Processing triggers for systemd (237-3ubuntu10.33) ...
root#b562da1810ce:$ rm -Rf /var/log/rabbitmq/*
root#b562da1810ce:$ sudo apt-get install -yq rabbitmq-server
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
rabbitmq-server
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 4,625 kB of archives.
After this operation, 5,678 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 rabbitmq-server all 3.6.10-1 [4,625 kB]
Fetched 4,625 kB in 4s (1,070 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package rabbitmq-server.
(Reading database ... 69613 files and directories currently installed.)
Preparing to unpack .../rabbitmq-server_3.6.10-1_all.deb ...
Unpacking rabbitmq-server (3.6.10-1) ...
Setting up rabbitmq-server (3.6.10-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/rabbitmq-server.service → /lib/systemd/system/rabbitmq-server.service.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for systemd (237-3ubuntu10.33) ...
root#b562da1810ce:$ sudo service rabbitmq-server status
Status of node rabbit#b562da1810ce
Error: unable to connect to node rabbit#b562da1810ce: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit#b562da1810ce]
rabbit#b562da1810ce:
* connected to epmd (port 4369) on b562da1810ce
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* Authentication failed (rejected by the remote node), please check the Erlang cookie
current node details:
- node name: 'rabbitmq-cli-30#b562da1810ce'
- home dir: /var/lib/rabbitmq
- cookie hash: DHe9O00f7sIHn/dTThKVVQ==
root#b562da1810ce:$ sudo service rabbitmq-server start
* Starting RabbitMQ Messaging Server rabbitmq-server * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
[fail]
root#b562da1810ce:$ sudo service rabbitmq-server status
Status of node rabbit#b562da1810ce
Error: unable to connect to node rabbit#b562da1810ce: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit#b562da1810ce]
rabbit#b562da1810ce:
* connected to epmd (port 4369) on b562da1810ce
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* Authentication failed (rejected by the remote node), please check the Erlang cookie
current node details:
- node name: 'rabbitmq-cli-13#b562da1810ce'
- home dir: /var/lib/rabbitmq
- cookie hash: DHe9O00f7sIHn/dTThKVVQ==
root#b562da1810ce:$ cat /var/log/rabbitmq/startup_err
root#b562da1810ce:$ cat /var/log/rabbitmq/startup_log
ERROR: node with name "rabbit" already running on "b562da1810ce"
Based on the last line from the log, I decided to check ps aux|grep -i rabbit, which shows Rabbit is running. Yet neither service nor rabbitmqctl is able to communicate with it. Why is this?

Either use official docker image from https://hub.docker.com//rabbitmq or yo can use the Dockerfile from https://hub.docker.com//rabbitmq
# Alpine Linux is not officially supported by the RabbitMQ team -- use at your own risk!
FROM alpine:3.10
RUN apk add --no-cache \
# grab su-exec for easy step-down from root
'su-exec>=0.2' \
# bash for docker-entrypoint.sh
bash \
# "ps" for "rabbitmqctl wait" (https://github.com/docker-library/rabbitmq/issues/162)
procps
# Default to a PGP keyserver that pgp-happy-eyeballs recognizes, but allow for substitutions locally
ARG PGP_KEYSERVER=ha.pool.sks-keyservers.net
# If you are building this image locally and are getting `gpg: keyserver receive failed: No data` errors,
# run the build with a different PGP_KEYSERVER, e.g. docker build --tag rabbitmq:3.7 --build-arg PGP_KEYSERVER=pgpkeys.eu 3.7/ubuntu
# For context, see https://github.com/docker-library/official-images/issues/4252
# Using the latest OpenSSL LTS release, with support until September 2023 - https://www.openssl.org/source/
ENV OPENSSL_VERSION 1.1.1d
ENV OPENSSL_SOURCE_SHA256="1e3a91bc1f9dfce01af26026f856e064eab4c8ee0a8f457b5ae30b40b8b711f2"
# https://www.openssl.org/community/omc.html
ENV OPENSSL_PGP_KEY_IDS="0x8657ABB260F056B1E5190839D9C4D26D0E604491 0x5B2545DAB21995F4088CEFAA36CEE4DEB00CFE33 0xED230BEC4D4F2518B9D7DF41F0DB4D21C1D35231 0xC1F33DD8CE1D4CC613AF14DA9195C48241FBF7DD 0x7953AC1FBC3DC8B3B292393ED5E9E43F7DF9EE8C 0xE5E52560DD91C556DDBDA5D02064C53641C25E5D"
# Use the latest stable Erlang/OTP release (https://github.com/erlang/otp/tags)
ENV OTP_VERSION 22.1.8
# TODO add PGP checking when the feature will be added to Erlang/OTP's build system
# http://erlang.org/pipermail/erlang-questions/2019-January/097067.html
ENV OTP_SOURCE_SHA256="7302be70cee2c33689bf2c2a3e7cfee597415d0fb3e4e71bd3e86bd1eff9cfdc"
# Install dependencies required to build Erlang/OTP from source
# http://erlang.org/doc/installation_guide/INSTALL.html
# autoconf: Required to configure Erlang/OTP before compiling
# dpkg-dev: Required to set up host & build type when compiling Erlang/OTP
# gnupg: Required to verify OpenSSL artefacts
# libncurses5-dev: Required for Erlang/OTP new shell & observer_cli - https://github.com/zhongwencool/observer_cli
RUN set -eux; \
\
apk add --no-cache --virtual .build-deps \
autoconf \
ca-certificates \
dpkg-dev dpkg \
gcc \
gnupg \
libc-dev \
linux-headers \
make \
ncurses-dev \
; \
\
OPENSSL_SOURCE_URL="https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz"; \
OPENSSL_PATH="/usr/local/src/openssl-$OPENSSL_VERSION"; \
OPENSSL_CONFIG_DIR=/usr/local/etc/ssl; \
\
# /usr/local/src doesn't exist in Alpine by default
mkdir /usr/local/src; \
\
# Required by the crypto & ssl Erlang/OTP applications
wget --output-document "$OPENSSL_PATH.tar.gz.asc" "$OPENSSL_SOURCE_URL.asc"; \
wget --output-document "$OPENSSL_PATH.tar.gz" "$OPENSSL_SOURCE_URL"; \
export GNUPGHOME="$(mktemp -d)"; \
for key in $OPENSSL_PGP_KEY_IDS; do \
gpg --batch --keyserver "$PGP_KEYSERVER" --recv-keys "$key"; \
done; \
gpg --batch --verify "$OPENSSL_PATH.tar.gz.asc" "$OPENSSL_PATH.tar.gz"; \
gpgconf --kill all; \
rm -rf "$GNUPGHOME"; \
echo "$OPENSSL_SOURCE_SHA256 *$OPENSSL_PATH.tar.gz" | sha256sum -c -; \
mkdir -p "$OPENSSL_PATH"; \
tar --extract --file "$OPENSSL_PATH.tar.gz" --directory "$OPENSSL_PATH" --strip-components 1; \
\
# Configure OpenSSL for compilation
cd "$OPENSSL_PATH"; \
# OpenSSL's "config" script uses a lot of "uname"-based target detection...
MACHINE="$(dpkg-architecture --query DEB_BUILD_GNU_CPU)" \
RELEASE="4.x.y-z" \
SYSTEM='Linux' \
BUILD='???' \
./config \
--openssldir="$OPENSSL_CONFIG_DIR" \
# add -rpath to avoid conflicts between our OpenSSL's "libssl.so" and the libssl package by making sure /usr/local/lib is searched first (but only for Erlang/OpenSSL to avoid issues with other tools using libssl; https://github.com/docker-library/rabbitmq/issues/364)
-Wl,-rpath=/usr/local/lib \
; \
# Compile, install OpenSSL, verify that the command-line works & development headers are present
make -j "$(getconf _NPROCESSORS_ONLN)"; \
make install_sw install_ssldirs; \
cd ..; \
rm -rf "$OPENSSL_PATH"*; \
# use Alpine's CA certificates
rmdir "$OPENSSL_CONFIG_DIR/certs" "$OPENSSL_CONFIG_DIR/private"; \
ln -sf /etc/ssl/certs /etc/ssl/private "$OPENSSL_CONFIG_DIR"; \
# smoke test
openssl version; \
\
OTP_SOURCE_URL="https://github.com/erlang/otp/archive/OTP-$OTP_VERSION.tar.gz"; \
OTP_PATH="/usr/local/src/otp-$OTP_VERSION"; \
\
# Download, verify & extract OTP_SOURCE
mkdir -p "$OTP_PATH"; \
wget --output-document "$OTP_PATH.tar.gz" "$OTP_SOURCE_URL"; \
echo "$OTP_SOURCE_SHA256 *$OTP_PATH.tar.gz" | sha256sum -c -; \
tar --extract --file "$OTP_PATH.tar.gz" --directory "$OTP_PATH" --strip-components 1; \
\
# Configure Erlang/OTP for compilation, disable unused features & applications
# http://erlang.org/doc/applications.html
# ERL_TOP is required for Erlang/OTP makefiles to find the absolute path for the installation
cd "$OTP_PATH"; \
export ERL_TOP="$OTP_PATH"; \
./otp_build autoconf; \
export CFLAGS='-g -O2'; \
# add -rpath to avoid conflicts between our OpenSSL's "libssl.so" and the libssl package by making sure /usr/local/lib is searched first (but only for Erlang/OpenSSL to avoid issues with other tools using libssl; https://github.com/docker-library/rabbitmq/issues/364)
export CFLAGS="$CFLAGS -Wl,-rpath=/usr/local/lib"; \
hostArch="$(dpkg-architecture --query DEB_HOST_GNU_TYPE)"; \
buildArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)"; \
dpkgArch="$(dpkg --print-architecture)"; dpkgArch="${dpkgArch##*-}"; \
./configure \
--host="$hostArch" \
--build="$buildArch" \
--disable-dynamic-ssl-lib \
--disable-hipe \
--disable-sctp \
--disable-silent-rules \
--enable-clock-gettime \
--enable-hybrid-heap \
--enable-kernel-poll \
--enable-shared-zlib \
--enable-smp-support \
--enable-threads \
--with-microstate-accounting=extra \
--without-common_test \
--without-debugger \
--without-dialyzer \
--without-diameter \
--without-edoc \
--without-erl_docgen \
--without-erl_interface \
--without-et \
--without-eunit \
--without-ftp \
--without-hipe \
--without-jinterface \
--without-megaco \
--without-observer \
--without-odbc \
--without-reltool \
--without-ssh \
--without-tftp \
--without-wx \
; \
# Compile & install Erlang/OTP
make -j "$(getconf _NPROCESSORS_ONLN)" GEN_OPT_FLGS="-O2 -fno-strict-aliasing"; \
make install; \
cd ..; \
rm -rf \
"$OTP_PATH"* \
/usr/local/lib/erlang/lib/*/examples \
/usr/local/lib/erlang/lib/*/src \
; \
\
runDeps="$( \
scanelf --needed --nobanner --format '%n#p' --recursive /usr/local \
| tr ',' '\n' \
| sort -u \
| awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }' \
)"; \
apk add --no-cache --virtual .otp-run-deps $runDeps; \
apk del --no-network .build-deps; \
\
# Check that OpenSSL still works after purging build dependencies
openssl version; \
# Check that Erlang/OTP crypto & ssl were compiled against OpenSSL correctly
erl -noshell -eval 'io:format("~p~n~n~p~n~n", [crypto:supports(), ssl:versions()]), init:stop().'
ENV RABBITMQ_DATA_DIR=/var/lib/rabbitmq
# Create rabbitmq system user & group, fix permissions & allow root user to connect to the RabbitMQ Erlang VM
RUN set -eux; \
addgroup -g 101 -S rabbitmq; \
adduser -u 100 -S -h "$RABBITMQ_DATA_DIR" -G rabbitmq rabbitmq; \
mkdir -p "$RABBITMQ_DATA_DIR" /etc/rabbitmq /tmp/rabbitmq-ssl /var/log/rabbitmq; \
chown -fR rabbitmq:rabbitmq "$RABBITMQ_DATA_DIR" /etc/rabbitmq /tmp/rabbitmq-ssl /var/log/rabbitmq; \
chmod 777 "$RABBITMQ_DATA_DIR" /etc/rabbitmq /tmp/rabbitmq-ssl /var/log/rabbitmq; \
ln -sf "$RABBITMQ_DATA_DIR/.erlang.cookie" /root/.erlang.cookie
# Use the latest stable RabbitMQ release (https://www.rabbitmq.com/download.html)
ENV RABBITMQ_VERSION 3.7.23-rc.1
# https://www.rabbitmq.com/signatures.html#importing-gpg
ENV RABBITMQ_PGP_KEY_ID="0x0A9AF2115F4687BD29803A206B73A36E6026DFCA"
ENV RABBITMQ_HOME=/opt/rabbitmq
# Add RabbitMQ to PATH, send all logs to TTY
ENV PATH=$RABBITMQ_HOME/sbin:$PATH \
RABBITMQ_LOGS=- RABBITMQ_SASL_LOGS=-
# Install RabbitMQ
RUN set -eux; \
\
apk add --no-cache --virtual .build-deps \
ca-certificates \
gnupg \
xz \
; \
\
RABBITMQ_SOURCE_URL="https://github.com/rabbitmq/rabbitmq-server/releases/download/v$RABBITMQ_VERSION/rabbitmq-server-generic-unix-latest-toolchain-$RABBITMQ_VERSION.tar.xz"; \
RABBITMQ_PATH="/usr/local/src/rabbitmq-$RABBITMQ_VERSION"; \
\
wget --output-document "$RABBITMQ_PATH.tar.xz.asc" "$RABBITMQ_SOURCE_URL.asc"; \
wget --output-document "$RABBITMQ_PATH.tar.xz" "$RABBITMQ_SOURCE_URL"; \
\
export GNUPGHOME="$(mktemp -d)"; \
gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$RABBITMQ_PGP_KEY_ID"; \
gpg --batch --verify "$RABBITMQ_PATH.tar.xz.asc" "$RABBITMQ_PATH.tar.xz"; \
gpgconf --kill all; \
rm -rf "$GNUPGHOME"; \
\
mkdir -p "$RABBITMQ_HOME"; \
tar --extract --file "$RABBITMQ_PATH.tar.xz" --directory "$RABBITMQ_HOME" --strip-components 1; \
rm -rf "$RABBITMQ_PATH"*; \
# Do not default SYS_PREFIX to RABBITMQ_HOME, leave it empty
grep -qE '^SYS_PREFIX=\$\{RABBITMQ_HOME\}$' "$RABBITMQ_HOME/sbin/rabbitmq-defaults"; \
sed -i 's/^SYS_PREFIX=.*$/SYS_PREFIX=/' "$RABBITMQ_HOME/sbin/rabbitmq-defaults"; \
grep -qE '^SYS_PREFIX=$' "$RABBITMQ_HOME/sbin/rabbitmq-defaults"; \
chown -R rabbitmq:rabbitmq "$RABBITMQ_HOME"; \
\
apk del .build-deps; \
\
# verify assumption of no stale cookies
[ ! -e "$RABBITMQ_DATA_DIR/.erlang.cookie" ]; \
# Ensure RabbitMQ was installed correctly by running a few commands that do not depend on a running server, as the rabbitmq user
# If they all succeed, it's safe to assume that things have been set up correctly
su-exec rabbitmq rabbitmqctl help; \
su-exec rabbitmq rabbitmqctl list_ciphers; \
su-exec rabbitmq rabbitmq-plugins list; \
# no stale cookies
rm "$RABBITMQ_DATA_DIR/.erlang.cookie"
# Added for backwards compatibility - users can simply COPY custom plugins to /plugins
RUN ln -sf /opt/rabbitmq/plugins /plugins
# set home so that any `--user` knows where to put the erlang cookie
ENV HOME $RABBITMQ_DATA_DIR
# Hint that the data (a.k.a. home dir) dir should be separate volume
VOLUME $RABBITMQ_DATA_DIR
# warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)
# Setting all environment variables that control language preferences, behaviour differs - https://www.gnu.org/software/gettext/manual/html_node/The-LANGUAGE-variable.html#The-LANGUAGE-variable
# https://docs.docker.com/samples/library/ubuntu/#locales
ENV LANG=C.UTF-8 LANGUAGE=C.UTF-8 LC_ALL=C.UTF-8
COPY docker-entrypoint.sh /usr/local/bin/
ENTRYPOINT ["docker-entrypoint.sh"]
EXPOSE 4369 5671 5672 25672
CMD ["rabbitmq-server"]
Use command to build the container (dot is to locate the Dockerfile in current directory).
docker build .
Once the image is built then you can use following command to start the container
docker container start youtImageName

Related

"docker:19.03-dind" could not select device driver "nvidia" with capabilities: [[gpu]]

I got a K8S+DinD issue:
launch Kubernetes cluster
start a main docker image and a DinD image inside this cluster
when running a job requesting GPU, got error could not select device driver "nvidia" with capabilities: [[gpu]]
Full error
http://localhost:2375/v1.40/containers/long-hash-string/start: Internal Server Error ("could not select device driver "nvidia" with capabilities: [[gpu]]")
exec to the DinD image inside of K8S pod, nvidia-smi is not available.
Some debugging and it seems it's due to the DinD is missing the Nvidia-docker-toolkit, I had the same error when I ran the same job directly on my local laptop docker, I fixed the same error by installing nvidia-docker2 sudo apt-get install -y nvidia-docker2.
I'm thinking maybe I can try to install nvidia-docker2 to the DinD 19.03 (docker:19.03-dind), but not sure how to do it? By multiple stage docker build?
Thank you very much!
update:
pod spec:
spec:
containers:
- name: dind-daemon
image: docker:19.03-dind
I got it working myself.
Referring to
https://github.com/NVIDIA/nvidia-docker/issues/375
https://github.com/Henderake/dind-nvidia-docker
First, I modified the ubuntu-dind image (https://github.com/billyteves/ubuntu-dind) to install nvidia-docker (i.e. added the instructions in the nvidia-docker site to the Dockerfile) and changed it to be based on nvidia/cuda:9.2-runtime-ubuntu16.04.
Then I created a pod with two containers, a frontend ubuntu container and the a privileged docker daemon container as a sidecar. The sidecar's image is the modified one I mentioned above.
But since this post is 3 year ago from now, I did spent quite some time to match up the dependencies versions, repo migration over 3 years, etc.
My modified version of Dockerfile to build it
ARG CUDA_IMAGE=nvidia/cuda:11.0.3-runtime-ubuntu20.04
FROM ${CUDA_IMAGE}
ARG DOCKER_CE_VERSION=5:18.09.1~3-0~ubuntu-xenial
RUN apt-get update -q && \
apt-get install -yq \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common && \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && \
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable" && \
apt-get update -q && apt-get install -yq docker-ce docker-ce-cli containerd.io
# https://github.com/docker/docker/blob/master/project/PACKAGERS.md#runtime-dependencies
RUN set -eux; \
apt-get update -q && \
apt-get install -yq \
btrfs-progs \
e2fsprogs \
iptables \
xfsprogs \
xz-utils \
# pigz: https://github.com/moby/moby/pull/35697 (faster gzip implementation)
pigz \
# zfs \
wget
# set up subuid/subgid so that "--userns-remap=default" works out-of-the-box
RUN set -x \
&& addgroup --system dockremap \
&& adduser --system -ingroup dockremap dockremap \
&& echo 'dockremap:165536:65536' >> /etc/subuid \
&& echo 'dockremap:165536:65536' >> /etc/subgid
# https://github.com/docker/docker/tree/master/hack/dind
ENV DIND_COMMIT 37498f009d8bf25fbb6199e8ccd34bed84f2874b
RUN set -eux; \
wget -O /usr/local/bin/dind "https://raw.githubusercontent.com/docker/docker/${DIND_COMMIT}/hack/dind"; \
chmod +x /usr/local/bin/dind
##### Install nvidia docker #####
# Add the package repositories
RUN curl -fsSL https://nvidia.github.io/nvidia-docker/gpgkey | apt-key add --no-tty -
RUN distribution=$(. /etc/os-release;echo $ID$VERSION_ID) && \
echo $distribution && \
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
tee /etc/apt/sources.list.d/nvidia-docker.list
RUN apt-get update -qq --fix-missing
RUN apt-get install -yq nvidia-docker2
RUN sed -i '2i \ \ \ \ "default-runtime": "nvidia",' /etc/docker/daemon.json
RUN mkdir -p /usr/local/bin/
COPY dockerd-entrypoint.sh /usr/local/bin/
RUN chmod 777 /usr/local/bin/dockerd-entrypoint.sh
RUN ln -s /usr/local/bin/dockerd-entrypoint.sh /
VOLUME /var/lib/docker
EXPOSE 2375
ENTRYPOINT ["dockerd-entrypoint.sh"]
#ENTRYPOINT ["/bin/sh", "/shared/dockerd-entrypoint.sh"]
CMD []
When I use exec to login into the Docker-in-Docker container, I can successfully run nvidia-smi (which previously return not found error then cannot run any GPU resource related docker run)
Welcome to pull my image at brandsight/dind:nvidia-docker

Docker - mount directory's owner and group

I ran a docker container with binding a host directory to a container directory, but the permissions for the container directory and its files are given differently depending on the hosts.
docker run -w /vlc-android -v $(pwd)/vlc-android:/vlc-android --rm vlc-android:latest bash -c "ls -ld /vlc-android"
result on Mac OS 10.14.6 (Docker desktop version 2.1.0.3)
drwxr-xr-x 2 videolan videolan 64 Sep 27 04:34 /vlc-android
result on Ubuntu server 18.04.3
drwxr-xr-x 2 root root 4096 Sep 27 06:11 /vlc-android
I'm trying to build the VLC player android app. from the source code via a docker image of the vlc-android build environment or below...
FROM debian:stretch-20190506
MAINTAINER VideoLAN roots <roots#videolan.org>
ENV IMAGE_DATE=201907171600
ENV ANDROID_NDK="/sdk/android-ndk" \
ANDROID_SDK="/sdk/android-sdk-linux"
# If someone wants to use VideoLAN docker images on a local machine and does
# not want to be disturbed by the videolan user, we should not take an uid/gid
# in the user range of main distributions, which means:
# - Debian based: <1000
# - RPM based: <500 (CentOS, RedHat, etc.)
ARG VIDEOLAN_CI_UID=499
RUN addgroup --quiet --gid ${VIDEOLAN_CI_UID} videolan && \
adduser --quiet --uid ${VIDEOLAN_CI_UID} --ingroup videolan videolan && \
echo "videolan:videolan" | chpasswd && \
apt-get update && \
apt-get install --no-install-suggests --no-install-recommends -y \
openjdk-8-jdk-headless ca-certificates autoconf m4 automake ant autopoint bison \
flex build-essential libtool libtool-bin patch pkg-config ragel subversion \
git rpm2cpio libwebkitgtk-1.0-0 yasm ragel g++ protobuf-compiler gettext \
libgsm1-dev wget expect unzip python python3 locales libltdl-dev curl && \
echo "deb http://ftp.debian.org/debian stretch-backports main" > /etc/apt/sources.list.d/stretch-backports.list && \
apt-get update && apt-get -y -t stretch-backports install cmake && \
rm -f /etc/apt/sources.list.d/stretch-backports.list && \
echo "deb http://deb.debian.org/debian testing main" > /etc/apt/sources.list.d/testing.list && \
apt-get update && apt-get -y -t testing --no-install-suggests --no-install-recommends install automake && \
rm -f /etc/apt/sources.list.d/testing.list && \
apt-get clean -y && rm -rf /var/lib/apt/lists/* && \
localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 && \
echo "export ANDROID_NDK=${ANDROID_NDK}" >> /etc/profile.d/vlc_env.sh && \
echo "export ANDROID_SDK=${ANDROID_SDK}" >> /etc/profile.d/vlc_env.sh && \
mkdir sdk && cd sdk && \
wget -q https://dl.google.com/android/repository/android-ndk-r18b-linux-x86_64.zip && \
ANDROID_NDK_SHA256=4f61cbe4bbf6406aa5ef2ae871def78010eed6271af72de83f8bd0b07a9fd3fd && \
echo $ANDROID_NDK_SHA256 android-ndk-r18b-linux-x86_64.zip | sha256sum -c && \
unzip android-ndk-r18b-linux-x86_64.zip && \
rm -f android-ndk-r18b-linux-x86_64.zip && \
ln -s android-ndk-r18b android-ndk && \
mkdir android-sdk-linux && \
cd android-sdk-linux && \
mkdir "licenses" && \
echo "24333f8a63b6825ea9c5514f83c2829b004d1fee" > "licenses/android-sdk-license" && \
echo "d56f5187479451eabf01fb78af6dfcb131a6481e" >> "licenses/android-sdk-license" && \
wget -q https://dl.google.com/android/repository/sdk-tools-linux-3859397.zip && \
SDK_TOOLS_SHA256=444e22ce8ca0f67353bda4b85175ed3731cae3ffa695ca18119cbacef1c1bea0 && \
echo $SDK_TOOLS_SHA256 sdk-tools-linux-3859397.zip | sha256sum -c && \
unzip sdk-tools-linux-3859397.zip && \
rm -f sdk-tools-linux-3859397.zip && \
tools/bin/sdkmanager "build-tools;26.0.1" "platform-tools" "platforms;android-26" && \
chown -R videolan /sdk
ENV LANG en_US.UTF-8
USER videolan
RUN git config --global user.name "VLC Android" && \
git config --global user.email buildbot#videolan.org
and built it like below
docker build -t vlc-android .
I want the user id "videolan" is the owner id of the container directory "/vlc-android" and all files under it in the container run on Ubuntu server 18.04.3, like "result on Mac OS 10.14.6 (Docker desktop version 2.1.0.3)".
How can I do?
When you mount a volume on linux, the resulting folder in the docker container will get the same rights as the folder on the host. If the folder on the host is owned by root, then it'll be owned by root also inside the docker container.
To fix your problem, you have to change the owner of the $(pwd)/vlc-android to match the user id used in the container (according to the Dockerfile you attached in your question, the UID is 499).
Try to execute this:
sudo chown 499 -R $(pwd)/vlc-android
then restart the container.
EDIT:
Another solution would be, if you're able to rebuild the docker image on the ubuntu server, to regenerate the image to use the folder owner id instead of 499.
You simply have to fetch the folder owner ID (try to avoid the root user):
id $username
and regenerate the docker image using the following command:
USER_ID=1000
docker build \
-t my_new_vlc_androing_thingy \
--build-arg VIDEOLAN_CI_UID=${USER_ID} \
.
and run it with:
docker run --rm \
-w /vlc-android \
-v $(pwd)/vlc-android:/vlc-android \
my_new_vlc_androing_thingy \
bash -c "ls -ld /vlc-android"

Dockerfile RUN multiple Command are not working

I am new to docker community want webpanel image for my php website when I run docker image building command I facing this error please some help me 4 days had been passed and am still stuck in this.in execution its showing error.
This is error snap Shot Link
#Docker File Code.
FROM lolhens/baseimage:latest
MAINTAINER LolHens <pierrekisters#gmail.com>
RUN apt-get update
RUN apt-get install -y \
curl
RUN cd /tmp
RUN curl http://vestacp.com/pub/vst-install.sh | bash -s \
y no -f \
password admin \
nginx yes \
apache yes \
phpfpm no \
vsftpd no \
proftpd no \
exim yes \
dovecot yes \
spamassassin yes \
clamav yes \
named yes \
iptables no \
fail2ban no \
mysql no \
postgresql yes \
remi yes \
quota yes \
cleanimage
ADD dovecot /etc/init.d/dovecot
RUN chmod +x /etc/init.d/dovecot
RUN cd /usr/local/vesta/data/ips && mv * 127.0.0.1 \
&& cd /etc/apache2/conf.d && sed -i -- 's/172.*.*.*:80/127.0.0.1:80/g' * && sed -i -- 's/172.*.*.*:8443/127.0.0.1:8443/g' * \
&& cd /etc/nginx/conf.d && sed -i -- 's/172.*.*.*:80;/80;/g' * && sed -i -- 's/172.*.*.*:8080/127.0.0.1:8080/g' * \
&& cd /home/admin/conf/web && sed -i -- 's/172.*.*.*:80;/80;/g' * && sed -i -- 's/172.*.*.*:8080/127.0.0.1:8080/g' *
ADD startup.sh /etc/my_init.d/startup.sh
RUN chmod +x /etc/my_init.d/startup.sh
CMD bash
EXPOSE 80 8083 8080 3306 443 25 993 110 53 54
You need to replace the first part of your script with the below snippet (the rest stays the same I guess - I can't test that part). I added inline comments explaining the new lines.
#Docker File Code.
FROM lolhens/baseimage:latest
MAINTAINER LolHens <pierrekisters#gmail.com>
RUN apt-get update
RUN apt-get install -y \
curl
RUN cd /tmp
# First save the script locally
RUN curl -O http://vestacp.com/pub/vst-install.sh
# Make the script runnable
RUN chmod a+x ./vst-install.sh
# Disable the apt-get check for unauthenticated packages
RUN echo "APT::Get::AllowUnauthenticated \"true\";" > /etc/apt/apt.conf.d/99vesta
# Pass 'yes' automatically when asked
RUN yes | ./vst-install.sh -y no -p admin \
nginx yes \
apache yes \
phpfpm no \
vsftpd no \
proftpd no \
exim yes \
dovecot yes \
spamassassin yes \
clamav yes \
named yes \
iptables no \
fail2ban no \
mysql no \
postgresql yes \
remi yes \
quota yes \
cleanimage

Reset a docker image to initial state

I'm new to docker and recently I tried to use setup openstreetmap-tileserver. I tried a manual installation by cloning the project and run docker build -t SampleMap and docker run -v openstreetmap-data:/var/lib/postgresql/10/main SampleMap import and then run the proper command to run the container. I got three images using docker image ls:
ubuntu
none
SampleMap
Everything worked fined. Next, I tried to erase the DB and do the whole process for a new map (a different .osm.pbf file). I removed the image SampleMap (with docker image rm) and tried to do the whole process again but the problem is all the DB tables still exist. It seems that all the changes are written into the Ubuntu image rather than the SampleMap. I'm asking generally is there any way that I can reset the whole Ubuntu image to its initial state? It seems that all the changes are permanent in the Ubuntu image.
Here is the Dockerfile:
FROM ubuntu:18.04
# Based on
# https://switch2osm.org/manually-building-a-tile-server-18-04-lts/
# Set up environment
ENV TZ=UTC
ENV AUTOVACUUM=on
ENV UPDATES=disabled
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
# Install dependencies
RUN echo "deb [ allow-insecure=yes ] http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main" >> /etc/apt/sources.list.d/pgdg.list \
&& apt-get update \
&& apt-get install -y apt-transport-https ca-certificates \
&& apt-get install -y --no-install-recommends --allow-unauthenticated \
apache2 \
apache2-dev \
autoconf \
build-essential \
bzip2 \
cmake \
fonts-noto-cjk \
fonts-noto-hinted \
fonts-noto-unhinted \
clang \
gdal-bin \
git-core \
libagg-dev \
libboost-all-dev \
libbz2-dev \
libcairo-dev \
libcairomm-1.0-dev \
libexpat1-dev \
libfreetype6-dev \
libgdal-dev \
libgeos++-dev \
libgeos-dev \
libgeotiff-epsg \
libicu-dev \
liblua5.3-dev \
libmapnik-dev \
libpq-dev \
libproj-dev \
libprotobuf-c0-dev \
libtiff5-dev \
libtool \
libxml2-dev \
lua5.3 \
make \
mapnik-utils \
nodejs \
npm \
postgis \
postgresql-10 \
postgresql-10-postgis-2.5 \
postgresql-10-postgis-2.5-scripts \
postgresql-contrib-10 \
protobuf-c-compiler \
python-mapnik \
sudo \
tar \
ttf-unifont \
unzip \
wget \
zlib1g-dev \
osmosis \
osmium-tool \
cron \
python3-psycopg2 python3-shapely python3-lxml \
&& apt-get clean autoclean \
&& apt-get autoremove --yes \
&& rm -rf /var/lib/{apt,dpkg,cache,log}/
# Set up renderer user
RUN adduser --disabled-password --gecos "" renderer
USER renderer
# Install latest osm2pgsql
RUN mkdir /home/renderer/src
WORKDIR /home/renderer/src
RUN git clone https://github.com/openstreetmap/osm2pgsql.git
WORKDIR /home/renderer/src/osm2pgsql
RUN mkdir build
WORKDIR /home/renderer/src/osm2pgsql/build
RUN cmake .. \
&& make -j $(nproc)
USER root
RUN make install
USER renderer
# Install and test Mapnik
RUN python -c 'import mapnik'
# Install mod_tile and renderd
WORKDIR /home/renderer/src
RUN git clone -b switch2osm https://github.com/SomeoneElseOSM/mod_tile.git
WORKDIR /home/renderer/src/mod_tile
RUN ./autogen.sh \
&& ./configure \
&& make -j $(nproc)
USER root
RUN make -j $(nproc) install \
&& make -j $(nproc) install-mod_tile \
&& ldconfig
USER renderer
# Configure stylesheet
WORKDIR /home/renderer/src
RUN git clone https://github.com/gravitystorm/openstreetmap-carto.git
WORKDIR /home/renderer/src/openstreetmap-carto
USER root
RUN npm install -g carto
USER renderer
RUN carto project.mml > mapnik.xml
# Load shapefiles
WORKDIR /home/renderer/src/openstreetmap-carto
RUN scripts/get-shapefiles.py
# Configure renderd
USER root
RUN sed -i 's/renderaccount/renderer/g' /usr/local/etc/renderd.conf \
&& sed -i 's/hot/tile/g' /usr/local/etc/renderd.conf
USER renderer
# Configure Apache
USER root
RUN mkdir /var/lib/mod_tile \
&& chown renderer /var/lib/mod_tile \
&& mkdir /var/run/renderd \
&& chown renderer /var/run/renderd
RUN echo "LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so" >> /etc/apache2/conf-available/mod_tile.conf \
&& a2enconf mod_tile
COPY apache.conf /etc/apache2/sites-available/000-default.conf
COPY leaflet-demo.html /var/www/html/index.html
RUN ln -sf /proc/1/fd/1 /var/log/apache2/access.log \
&& ln -sf /proc/1/fd/2 /var/log/apache2/error.log
# Configure PosgtreSQL
COPY postgresql.custom.conf.tmpl /etc/postgresql/10/main/
RUN chown -R postgres:postgres /var/lib/postgresql \
&& chown postgres:postgres /etc/postgresql/10/main/postgresql.custom.conf.tmpl \
&& echo "\ninclude 'postgresql.custom.conf'" >> /etc/postgresql/10/main/postgresql.conf
# copy update scripts
COPY openstreetmap-tiles-update-expire /usr/bin/
RUN chmod +x /usr/bin/openstreetmap-tiles-update-expire \
&& mkdir /var/log/tiles \
&& chmod a+rw /var/log/tiles \
&& ln -s /home/renderer/src/mod_tile/osmosis-db_replag /usr/bin/osmosis-db_replag \
&& echo "* * * * * renderer openstreetmap-tiles-update-expire\n" >> /etc/crontab
# install trim_osc.py helper script
USER renderer
RUN cd ~/src \
&& git clone https://github.com/zverik/regional \
&& cd regional \
&& git checkout 612fe3e040d8bb70d2ab3b133f3b2cfc6c940520 \
&& chmod u+x ~/src/regional/trim_osc.py
# Start running
USER root
COPY run.sh /
COPY indexes.sql /
ENTRYPOINT ["/run.sh"]
CMD []
EXPOSE 80 5432
And here is my run.sh file:
#!/bin/bash
set -x
function CreatePostgressqlConfig()
{
cp /etc/postgresql/10/main/postgresql.custom.conf.tmpl /etc/postgresql/10/main/postgresql.custom.conf
sudo -u postgres echo "autovacuum = $AUTOVACUUM" >> /etc/postgresql/10/main/postgresql.custom.conf
cat /etc/postgresql/10/main/postgresql.custom.conf
}
if [ "$#" -ne 1 ]; then
ls /home/renderer
echo "usage: <import|run>"
echo "commands:"
echo " import: Set up the database and import /data.osm.pbf"
echo " run: Runs Apache and renderd to serve tiles at /tile/{z}/{x}/{y}.png"
echo "environment variables:"
echo " THREADS: defines number of threads used for importing / tile rendering"
echo " UPDATES: consecutive updates (enabled/disabled)"
exit 1
fi
if [ "$1" = "import" ]; then
# Initialize PostgreSQL
CreatePostgressqlConfig
service postgresql start
sudo -u postgres createuser renderer
sudo -u postgres createdb -E UTF8 -O renderer gis
sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis;"
sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore;"
sudo -u postgres psql -d gis -c "ALTER TABLE geometry_columns OWNER TO renderer;"
sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO renderer;"
# Download Luxembourg as sample if no data is provided
if [ ! -f /data.osm.pbf ]; then
echo "WARNING: No import file at /data.osm.pbf, so importing iran-latest as example..."
wget -nv http://download.geofabrik.de/north-america/canada-latest.osm.pbf -O /data.osm.pbf
# wget -nv http://download.geofabrik.de/europe/luxembourg.poly -O /data.poly
fi
# determine and set osmosis_replication_timestamp (for consecutive updates)
osmium fileinfo /data.osm.pbf > /var/lib/mod_tile/data.osm.pbf.info
osmium fileinfo /data.osm.pbf | grep 'osmosis_replication_timestamp=' | cut -b35-44 > /var/lib/mod_tile/replication_timestamp.txt
REPLICATION_TIMESTAMP=$(cat /var/lib/mod_tile/replication_timestamp.txt)
# initial setup of osmosis workspace (for consecutive updates)
sudo -u renderer openstreetmap-tiles-update-expire $REPLICATION_TIMESTAMP
# copy polygon file if available
if [ -f /data.poly ]; then
sudo -u renderer cp /data.poly /var/lib/mod_tile/data.poly
fi
# Import data
sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua -C 2048 --number-processes ${THREADS:-4} -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf
# Create indexes
sudo -u postgres psql -d gis -f indexes.sql
service postgresql stop
exit 0
fi
if [ "$1" = "run" ]; then
# Clean /tmp
rm -rf /tmp/*
# Fix postgres data privileges
chown postgres:postgres /var/lib/postgresql -R
# Initialize PostgreSQL and Apache
CreatePostgressqlConfig
service postgresql start
service apache2 restart
# Configure renderd threads
sed -i -E "s/num_threads=[0-9]+/num_threads=${THREADS:-4}/g" /usr/local/etc/renderd.conf
# start cron job to trigger consecutive updates
if [ "$UPDATES" = "enabled" ]; then
/etc/init.d/cron start
fi
# Run
sudo -u renderer renderd -f -c /usr/local/etc/renderd.conf
service postgresql stop
exit 0
fi
echo "invalid command"
exit 1
When you create a container from your image, you mount a volume, using the -v option:
docker run -v openstreetmap-data:/var/lib/postgresql/10/main SampleMap import
Your persistent data is stored in openstreetmap-data. That file/folder is not in your container (that is created every time), it is mounted from your host's filesystem. That's why it persists

Copy files or clone via ssh in Dockerfile when building docker image

I have been trying for a while to copy files via ssh from a remote server (not gihub) inside the docker image I want to build, but I can't connect to host. Here is the Dockerfile up until the critical point:
FROM r-base:latest
### Install libs
RUN apt-get update && apt-get install -y \
sudo \
gdebi-core \
pandoc \
pandoc-citeproc \
openssh-server \
openssh-client \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
xtail \
wget \
libssl-dev \
libxml2 \
libxml2-dev \
libv8-dev \
curl \
gnupg \
git
COPY ./setup setup
RUN mv setup/.ssh ~/.ssh
RUN touch ~/.ssh/known_hosts
RUN chmod -R 400 ~/.ssh
RUN ssh-agent sh -c 'ssh-add /root/.ssh/id_rsa'
#RUN eval "$(ssh-agent -s)"
#RUN ssh-add -K ~/.ssh/id_rsa #This is commented out as it causes an error
RUN ssh-keyscan hostname > ~/.ssh/known_host
RUN ssh-keygen -R hostname
## THIS IS THE COMMAND WE NEED TO RUN...
RUN scp -r user#hostname:/path/to/folder ./
The owner of the folder is user. The id_rsa.pub was added to the authorized_keys file of the user user on the host, and ssh was restarted there. However I get a Failed authentication error. I tried to use my personal id_rsa which works from the command line, but it also fails inside docker. Is this a docker issuor is it solvable?
I finally managed to do it by generating a key with the command suggested in this post
So to reproduce my case, locally:
cd setup/.ssh/
ssh-keygen -q -t rsa -N '' -f id_rsa
Then on the server add the id_rsa.pub contents to the known hosts for the user user. Can copy the contents to clipboard using xclip: xclip -sel clip < setup/.ssh/id_rsa.pub)
Dockerfile:
I have been trying for a while to copy files via ssh from a remote server (not gihub) inside the docker image I want to build, but I can't connect to host. Here is the Dockerfile up until the critical point:
FROM r-base:latest
### Install libs
RUN apt-get update && apt-get install -y \
sudo \
gdebi-core \
pandoc \
pandoc-citeproc \
openssh-server \
openssh-client \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
xtail \
wget \
libssl-dev \
libxml2 \
libxml2-dev \
libv8-dev \
curl \
gnupg \
git
COPY ./setup setup
RUN chmod -R 600 ~/.ssh
RUN echo "IdentityFile /root/.ssh/id_rsa" >> /etc/ssh/ssh_config
RUN echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config
## THIS IS THE COMMAND WE NEED TO RUN...
RUN scp -r user#hostname:/path/to/folder ./
There’s no specific requirement that you must do everything inside your Dockerfile. Especially things that require remote ssh access are better done outside Docker: consider that anyone who gets your image later on can docker cp a valid ssh key out of it and potentially get access to your internal systems.
For Docker caching reasons, it’s also not a good idea to git clone or otherwise try to remotely retrieve your application from inside the Dockerfile. If you re-run docker build, and nothing else in your Dockerfile has changed, then Docker will skip over the scp step too, even if the remote content has changed.
My general recommendation would be to copy this content from outside the Dockerfile, then build it
# Using whatever credentials are in your local ssh-agent
scp -r user#hostname:/path/to/stuff dist/
# Then your Dockerfile doesn’t need scp or credentials
docker build .
Your Dockerfile then doesn’t need a bunch of extra packages that are only relevant to this path: you should be able to remove sudo openssh-server openssh-client xtail curl gnupg git without actually affecting the single main process you’re trying to run inside your container.

Resources