I'm trying to run a tcpdump command in docker container but I always get this error: "tcpdump: invalid number of output files ="
I'm running docker in Ubuntu.
This is my container run:
sudo docker run --name pcap_log -it --network=host -e "log_int=any" -e "log_rot=30" -e "log_fil=120" --mount source=/home/docker/container_data/pcap_data, target=/pcap_captures, type=bind pcap_log:12345
This is my image:
FROM debian:10.3
RUN apt-get update \
&& apt-get install -y \
tcpdump
RUN mv /usr/sbin/tcpdump /usr/bin/tcpdump
RUN mkdir /pcap_captures
WORKDIR /pcap_captures
ARG log_int
ARG log_rot
ARG log_fil
ENV log_int = $log_int
ENV log_rot = $log_rot
ENV log_fil = $log_fil
CMD tcpdump -i $log_int -G $log_rot -w '%Y-%m-%d_%H:%M:%S.pcap' -W $log_fil -U
"tcpdump: invalid number of output files ="
As you can see, there is an = sign left somewhere that is not interpreted correctly.
The problem is in your ENV declaration with an extra space before and after the = sign. For example
ENV log_int = $log_int
is instructing docker that the environment var log_int should be set to the value = $log_ing. Either you completely drop the equal sign
ENV log_int $log_int
or you remove the spaces arround it
ENV log_int=$log_int
Both are totally equivalent.
I actually suggest your keep the second form, declare all your envs in a single action and secure your vars with quotes:
ENV log_int="$log_int" log_rot="$log_rot" log_fil="$log_fil"
Related
I am following this link to create a spark cluster. I am able to run the spark cluster. However, I have to give an absolute path to start spark-shell. I am trying to set environment variables i.e. PATH and a few others in start-shell.sh. However, it's not setting that inside container. I tried printing it using printenv inside the container. But these variables are never reflected.
Am I trying to set environment variables incorrectly? Spark cluster is running successfully though.
I am using docker-compose.yml to build and recreate an image and container.
docker-compose up --build
Dockerfile
# builder step used to download and configure spark environment
FROM openjdk:11.0.11-jre-slim-buster as builder
# Add Dependencies for PySpark
RUN apt-get update && apt-get install -y curl vim wget software-properties-common ssh net-tools ca-certificates python3 python3-pip python3-numpy python3-matplotlib python3-scipy python3-pandas python3-simpy
# JDBC driver download and install
ADD https://go.microsoft.com/fwlink/?linkid=2168494 /usr/share/java
RUN update-alternatives --install "/usr/bin/python" "python" "$(which python3)" 1
# Fix the value of PYTHONHASHSEED
# Note: this is needed when you use Python 3.3 or greater
ENV SPARK_VERSION=3.1.2 \
HADOOP_VERSION=3.2 \
SPARK_HOME=/opt/spark \
PYTHONHASHSEED=1
# Download and uncompress spark from the apache archive
RUN wget --no-verbose -O apache-spark.tgz "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" \
&& mkdir -p ${SPARK_HOME} \
&& tar -xf apache-spark.tgz -C ${SPARK_HOME} --strip-components=1 \
&& rm apache-spark.tgz
My Dockerfile-spark
When using SPARK_BIN="${SPARK_HOME}/bin/ under ENV in Dockerfile, environment variable get's set. It is visible inside the docker container by using printenv
FROM apache-spark
WORKDIR ${SPARK_HOME}
ENV SPARK_MASTER_PORT=7077 \
SPARK_MASTER_WEBUI_PORT=8080 \
SPARK_LOG_DIR=${SPARK_HOME}/logs \
SPARK_MASTER_LOG=${SPARK_HOME}/logs/spark-master.out \
SPARK_WORKER_LOG=${SPARK_HOME}/logs/spark-worker.out \
SPARK_WORKER_WEBUI_PORT=8080 \
SPARK_MASTER="spark://spark-master:7077" \
SPARK_WORKLOAD="master"
COPY start-spark.sh /
CMD ["/bin/bash", "/start-spark.sh"]
start-spark.sh
#!/bin/bash
. "$SPARK_HOME/bin/load-spark-env.sh"
export SPARK_BIN="${SPARK_HOME}/bin/" # This doesn't work here
export PATH="${SPARK_HOME}/bin/:${PATH}" # This doesn't work here
# When the spark work_load is master run class org.apache.spark.deploy.master.Master
if [ "$SPARK_WORKLOAD" == "master" ];
then
export SPARK_MASTER_HOST=`hostname` # This works here
cd $SPARK_BIN && ./spark-class org.apache.spark.deploy.master.Master --ip $SPARK_MASTER_HOST --port $SPARK_MASTER_PORT --webui-port $SPARK_MASTER_WEBUI_PORT >> $SPARK_MASTER_LOG.
My File structure is
dockerfile
dockerfile-spark # this uses pre-built image created by dockerfile
start-spark.sh # invoked buy dockerfile-spark
docker-compose.yml # uses build parameter to build an image from dockerfile-spark
From inside the master container
root#3abbd4508121:/opt/spark# export
declare -x HADOOP_VERSION="3.2"
declare -x HOME="/root"
declare -x HOSTNAME="3abbd4508121"
declare -x JAVA_HOME="/usr/local/openjdk-11"
declare -x JAVA_VERSION="11.0.11+9"
declare -x LANG="C.UTF-8"
declare -x OLDPWD
declare -x PATH="/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
declare -x PWD="/opt/spark"
declare -x PYTHONHASHSEED="1"
declare -x SHLVL="1"
declare -x SPARK_HOME="/opt/spark"
declare -x SPARK_LOCAL_IP="spark-master"
declare -x SPARK_LOG_DIR="/opt/spark/logs"
declare -x SPARK_MASTER="spark://spark-master:7077"
declare -x SPARK_MASTER_LOG="/opt/spark/logs/spark-master.out"
declare -x SPARK_MASTER_PORT="7077"
declare -x SPARK_MASTER_WEBUI_PORT="8080"
declare -x SPARK_VERSION="3.1.2"
declare -x SPARK_WORKER_LOG="/opt/spark/logs/spark-worker.out"
declare -x SPARK_WORKER_WEBUI_PORT="8080"
declare -x SPARK_WORKLOAD="master"
declare -x TERM="xterm"
root#3abbd4508121:/opt/spark#
There are a couple of different ways to set environment variables in Docker, and a couple of different ways to run processes. A container normally runs one process, which is controlled by the image's ENTRYPOINT and CMD settings. If you docker exec a second process in the container, that does not run as a child process of the main process, and will not see environment variables that are set by that main process.
In the setup you show here, the start-spark.sh script is the main container process (it is the image's CMD). If you docker exec your-container printenv, it will see things set in the Dockerfile but not things set in this script.
Things like filesystem paths will generally be fixed every time you run the container, no matter what command you're running there, so you can specify these in the Dockerfile
ENV SPARK_BIN=${SPARK_HOME}/bin PATH=${SPARK_BIN}:${PATH}
You can specify both an ENTRYPOINT and a CMD in your Dockerfile; if you do, the CMD is passed as arguments to the ENTRYPOINT. This leads to a useful pattern where the CMD is a standard shell command, and the ENTRYPOINT is a wrapper that does first-time setup and then runs it. You can split your script into two:
#!/bin/sh
# spark-env.sh
. "${SPARK_BIN}/load-spark-env.snh"
exec "$#"
#!/bin/sh
# start-spark.sh
spark-class org.apache.spark.deploy.master.Master \
--ip "$SPARK_MASTER_HOST" \
--port "$SPARK_MASTER_PORT" \
--webui-port "$SPARK_MASTER_WEBUI_PORT"
Then in your Dockerfile specify both parts
COPY spark-env.sh start-spark.sh /
ENTRYPOINT ["/spark-env.sh"] # must be JSON-array syntax
CMD ["/start-spark.sh"] # or any other valid CMD
This is useful for your debugging since it's straightforward to override the CMD in a docker run or docker-compose run instruction, leaving the ENTRYPOINT in place.
docker-compose run spark \
printenv
This launches a new container based on all of the same Dockerfile setup. When it runs, it runs printenv instead of the CMD in the image. This will do the first-time setup in the ENTRYPOINT script, and then the final exec "$#" line will run printenv instead of starting the Spark application. This will show you the environment the application will have when it starts.
Mine is a bit of a peculiar situation, I created a dockerfile that "works" if not for some proiblems,
Here is a "working" version:
ARG IMGVERS=latest
FROM bensuperpc/tinycore:${IMGVERS}
LABEL maintainer "Vinnie Costante <****#gmail.com>"
ARG DOWNDIR=/tmp/download
ARG INSTDIR=/opt/vscodium
ARG REPOAPI="https://api.github.com/repos/VSCodium/vscodium/releases/latest"
ENV LANG=C.UTF-8 LC_ALL=C PATH="${PATH}:${INSTDIR}/bin/"
RUN tce-load -wic Xlibs nss gtk3 libasound libcups python3.9 tk8.6 \
&& rm -rf /tmp/tce/optional/*
RUN sudo ln -s /lib /lib64 \
&& sudo ln -s /usr/local/etc/fonts /etc/fonts \
&& sudo mkdir -p ${DOWNDIR} ${INSTDIR} \
&& sudo chown -R tc:staff ${DOWNDIR} ${INSTDIR}
#COPY VSCodium-linux-x64-1.57.1.tar.gz ${DOWNDIR}/
RUN wget http://192.168.43.6:8000/VSCodium-linux-x64-1.57.1.tar.gz -P ${DOWNDIR}
RUN tar xvf ${DOWNDIR}/VSCodium*.gz -C ${INSTDIR} \
&& rm -rf ${DOWNDIR}
CMD ["codium"]
The issues are these:
Starting the image with this command vscodium does not start, but entering the shell (adding /bin/ash to the end of the docker run) and then running codium instead vscodium starts. I tried many ways, even changing the entrypoint, the result is always the same. But if I try to add any other graphic program (like firefox) and replace the argument of the CMD instruction inside the dockerfile, everything works as it should.
docker run -it --rm \
--net=host \
--env="DISPLAY=unix${DISPLAY}" \
--workdir /home/tc \
--volume="$HOME/.Xauthority:/root/.Xauthority:rw" \
--name tc \
tinycodium
the last two versions of codium (1.58.0 and 1.58.1) don't work at all on docker but they start normally on the same distro not containerized. I tried installing other dependencies but nothing worked. Right now I don't know how to understand what's wrong with these two new versions.
I don't know how to set a volume to save codium data, I tried something like this --volume=/home/vinnie/docker:/home/tc but there are always problems with user/group permissions. I've also tried booting the container as user by adding it to the docker group but there's always a mess with permissions. If someone could explain me how to proceed, the directories I want to save are these:
/home/tc/.vscode-oss
/home/tc/.cache/mesa_shader_cache
/home/tc/.config/VSCodium
/home/tc/.config/glib-2.0/settings
/home/tc/.local/share
Try running codium --verbose and see if the container starts
I have a local dev env which requires different hosts reachable via SSH so i set up a docker-compose.yml with some services:
services:
ssh1:
build:
context: ./.project/docker/ssh1
dockerfile: Dockerfile
environment:
MYSQL_USER: app1
MYSQL_PASSWORD: app1
The Dockerfile contains following contents:
FROM ubuntu:20.04
RUN export DEBIAN_FRONTEND=noninteractive \
&& ln -fs /usr/share/zoneinfo/Europe/Berlin /etc/localtime \
&& apt update \
&& apt upgrade -y \
&& apt install -y openssh-server rsync php \
&& mkdir /run/sshd/ \
&& ssh-keygen -A \
&& for key in $(ls /etc/ssh/ssh_host_* | grep -v pub); do echo "HostKey $key" >> /etc/ssh/sshd_config; done \
&& addgroup --gid 1000 app \
&& adduser --gecos "" --disabled-password --shell /bin/bash --uid 1000 --gid 1000 app \
&& mkdir -m 700 /home/app/.ssh/ \
&& chown app:app /home/app/.ssh/ \
&& rm -rf /var/lib/apt/lists/*
COPY --chown=app:app ssh1_rsa.pub /home/app/.ssh/authorized_keys
CMD ["/usr/sbin/sshd", "-D"]
EXPOSE 22
I can verify, that the environment variables are set in the container
$ docker-compose exec ssh1 printenv | grep MYSQL
MYSQL_USER=app1
MYSQL_PASSWORD=app1
docker inspect project_ssh1_1 also shows the ENV variables.
But when i connect from another container to ssh1 via ssh, my environment variables are not set.
Why are my environment variables not set when i ssh into the container?
I also appreciate any in-depth input on how env vars are set in the container via docker and how env vars are inherited from processes or the "OS".
Solution edit:
The actual question was answered. However, i did not ask the right question. So here's my actual solution. I am able to set the ENV VARS in the SSH session with the help of a pretty hacky solution, which should not be used in PROD environments, since it could lead to information disclosure.
Add all ENV VARS as build args.
docker-compose.yml:
ssh1:
build:
context: ./.project/docker/ssh1
dockerfile: Dockerfile
args:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: app1
MYSQL_USER: app1
MYSQL_PASSWORD: app1
MYSQL_HOST: mysql1
And write them to $HOME/.ssh/environment as well as enabling PermitUserEnvironment. Don't do this in production
Dockerfile
FROM ubuntu:20.04
ARG MYSQL_ROOT_PASSWORD
ARG MYSQL_DATABASE
ARG MYSQL_USER
ARG MYSQL_PASSWORD
ARG MYSQL_HOST
RUN echo "MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD" >> /home/app/.ssh/environment \
&& echo "MYSQL_DATABASE=$MYSQL_DATABASE" >> /home/app/.ssh/environment \
&& echo "MYSQL_USER=$MYSQL_USER" >> /home/app/.ssh/environment \
&& echo "MYSQL_PASSWORD=$MYSQL_PASSWORD" >> /home/app/.ssh/environment \
&& echo "MYSQL_HOST=$MYSQL_HOST" >> /home/app/.ssh/environment \
&& sed -i 's/#PermitUserEnvironment no/PermitUserEnvironment yes/g' /etc/ssh/sshd_config
Now, when you login, SSH will read the env vars from the users .ssh/environment (app in this case) and set them in the user's SSH session.
Environment variables are present in RUN commands and in the shell you exec into when issuing docker exec command, but when you ssh into an ssh server running inside container, you actually get a brand new shell which doesn’t have those env variables set.
Your issue has actually not much to do with docker but is due to the way sshd works: for every connection sshd will setup a new environment wiping out all variables in its own environment, see the Login Process in man sshd.
(It's easy to see why this makes sense: sshd is started by the root user and so may contain sensitive data in its environment variables that should not be leaked to the users. Other variables wouldn't be reasonable to pass to the user session, e.g. HOME, PATH, SHELL)
Depending on your use case there are various ways to pass environment variables to a new ssh session, depending on whether it being an interactive or non-interactive session, running a (login) shell or not:
~/.ssh/environment: variables injected by ssh, see PermitUserEnvironment
/etc/environment: used by pam_env on login
/etc/profile, ~/.bashrc and alike: configs used by the (login) shell, see bash for example.
Also depending on your use case you have now various options how to add these files to the container:
if the variables are static: just ADD the respective file to the image or create it in the Dockerfile
if the variables are set on build-time: use ARGs (or ENV) to pass the variables to the build and create the respective file from that in the build (as you did in your solution)
if the variables should be set on container run-time:
use a custom ENTRYPOINT script to generate the respective file on startup from the passed environment variables or command line arguments
volume-mount the respective file into the container (you may also use docker secret for sensitive data here)
This could be maybe a trivial question but reading docs for ARG and ENV doesn't put things clear to me.
I am building a PHP-FPM container and I want to give the ability for enable/disable some extensions on user needs.
Would be great if this could be done in the Dockerfile by adding conditionals and passing flags on the build command perhaps but AFAIK is not supported.
In my case and my personal approach is to run a small script when container starts, something like the following:
#!/bin/sh
set -e
RESTART="false"
# This script will be placed in /config/init/ and run when container starts.
if [ "$INSTALL_XDEBUG" == "true" ]; then
printf "\nInstalling Xdebug ...\n"
yum install -y php71-php-pecl-xdebug
RESTART="true"
fi
...
if [ "$RESTART" == "true" ]; then
printf "\nRestarting php-fpm ...\n"
supervisorctl restart php-fpm
fi
exec "$#"
This is how my Dockerfile looks like:
FROM reynierpm/centos7-supervisor
ENV TERM=xterm \
PATH="/root/.composer/vendor/bin:${PATH}" \
INSTALL_COMPOSER="false" \
COMPOSER_ALLOW_SUPERUSER=1 \
COMPOSER_ALLOW_XDEBUG=1 \
COMPOSER_DISABLE_XDEBUG_WARN=1 \
COMPOSER_HOME="/root/.composer" \
COMPOSER_CACHE_DIR="/root/.composer/cache" \
SYMFONY_INSTALLER="false" \
SYMFONY_PROJECT="false" \
INSTALL_XDEBUG="false" \
INSTALL_MONGO="false" \
INSTALL_REDIS="false" \
INSTALL_HTTP_REQUEST="false" \
INSTALL_UPLOAD_PROGRESS="false" \
INSTALL_XATTR="false"
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm \
https://rpms.remirepo.net/enterprise/remi-release-7.rpm
RUN yum install -y \
yum-utils \
git \
zip \
unzip \
nano \
wget \
php71-php-fpm \
php71-php-cli \
php71-php-common \
php71-php-gd \
php71-php-intl \
php71-php-json \
php71-php-mbstring \
php71-php-mcrypt \
php71-php-mysqlnd \
php71-php-pdo \
php71-php-pear \
php71-php-xml \
php71-pecl-apcu \
php71-php-pecl-apfd \
php71-php-pecl-memcache \
php71-php-pecl-memcached \
php71-php-pecl-zip && \
yum clean all && rm -rf /tmp/yum*
RUN ln -sfF /opt/remi/php71/enable /etc/profile.d/php71-paths.sh && \
ln -sfF /opt/remi/php71/root/usr/bin/{pear,pecl,phar,php,php-cgi,phpize} /usr/local/bin/. && \
mv -f /etc/opt/remi/php71/php.ini /etc/php.ini && \
ln -s /etc/php.ini /etc/opt/remi/php71/php.ini && \
rm -rf /etc/php.d && \
mv /etc/opt/remi/php71/php.d /etc/. && \
ln -s /etc/php.d /etc/opt/remi/php71/php.d
COPY container-files /
RUN chmod +x /config/bootstrap.sh
WORKDIR /data/www
EXPOSE 9001
Currently this is working but ... If I want to add let's say 20 (a random number) of extensions or any other feature that can be enable|disable then I will end with 20 non necessary ENV (because Dockerfile doesn't support .env files) definition whose only purpose would be set this flag for let the script knows what to do then ...
Is this the right way to do it?
Should I use ENV for this purpose?
I am open to ideas if you have a different approach for achieve this please let me know about it
From Dockerfile reference:
The ARG instruction defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag.
The ENV instruction sets the environment variable <key> to the value <value>.
The environment variables set using ENV will persist when a container is run from the resulting image.
So if you need build-time customization, ARG is your best choice.
If you need run-time customization (to run the same image with different settings), ENV is well-suited.
If I want to add let's say 20 (a random number) of extensions or any other feature that can be enable|disable
Given the number of combinations involved, using ENV to set those features at runtime is best here.
But you can combine both by:
building an image with a specific ARG
using that ARG as an ENV
That is, with a Dockerfile including:
ARG var
ENV var=${var}
You can then either build an image with a specific var value at build-time (docker build --build-arg var=xxx), or run a container with a specific runtime value (docker run -e var=yyy)
So if want to set the value of an environment variable to something different for every build then we can pass these values during build time and we don't need to change our docker file every time.
While ENV, once set cannot be overwritten through command line values. So, if we want to have our environment variable to have different values for different builds then we could use ARG and set default values in our docker file. And when we want to overwrite these values then we can do so using --build-args at every build without changing our docker file.
For more details, you can refer this.
Why to use ARG or ENV ?
Let's say we have a jar file and we want to make a docker image of it. So, we can ship it to any docker engine.
We can write a Dockerfile.
Dockerfile
FROM eclipse-temurin:17-jdk-alpine
VOLUME /tmp
ARG JAR_FILE
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
Now, if we want to build the docker image using Maven, we can pass the JAR_FILE using the --build-arg as target/*.jar
docker build --build-arg JAR_FILE=target/*.jar -t myorg/myapp
However, if we are using Gradle; the above command doesn't work and we've to pass a different path: build/libs/
docker build --build-arg JAR_FILE=build/libs/*.jar -t myorg/myapp .
Once you have chosen a build system, we don’t need the ARG. We can hard code the JAR location.
For Maven, that would be as follows:
Dockerfile
FROM eclipse-temurin:17-jdk-alpine
VOLUME /tmp
COPY target/*.jar app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
here, we can build an image with the following command:
docker build -t image:tag .
When to use `ENV`?
If we want to set some values at running containers and reflect that to the image like the Port Number that your application can run/listen on. We can set that using the ENV.
Both ARG and ENV seem very similar. Both can be accessed from within our Dockerfile commands in the same manner.
Example:
ARG VAR_A 5
ENV VAR_B 6
RUN echo $VAR_A
RUN echo $VAR_B
Personal Option!
There is a tradeoff between choosing ARG over ENV. If you choose ARG you can't change it later during the run. However, if you chose ENV you can modify the value at the container.
I personally prefer ARG over ENV wherever I can, like,
In the above Example:
I have used ARG as the build system maven or Gradle impacts during build rather than runtime. It thus encapsulates a lot of details and provided a minimum set of arguments for the runtime.
For more details, you can refer to this.
I can define "static" environment variables in a Dockerfile with ENV, but is it possible to pass some value at build time to this variable? I'm attempting something like this, which doesn't work:
FROM phusion/baseimage
RUN mkdir -p /foo/2016/bin && \
FOOPATH=`ls -d /foo/20*/bin` && \
export FOOPATH
ENV PATH $PATH:$FOOPATH
Of course, in the real use case I'd be running/unpacking something that creates a directory whose name will change with different versions, dates, etc., and I'd like to avoid modifying the Dockerfile every time the directory name changes.
Edit: Since it appears it's not possible, the best workaround so far is using a symlink:
FROM phusion/baseimage
RUN mkdir -p /foo/2016/bin && \
FOOPATH=`ls -d /foo/20*/bin` && \
ln -s $FOOPATH /mypath
ENV PATH $PATH:/mypath
To pass a value in at build time, use an ARG.
FROM phusion/baseimage
RUN mkdir -p /foo/2016/bin && \
FOOPATH=`ls -d /foo/20*/bin` && \
export FOOPATH
ARG FOOPATH
ENV PATH $PATH:${FOOPATH}
Then you can run docker build --build-arg FOOPATH=/dir -t myimage .
Edit: from you comment, my answer above won't solve your issue. There's nothing in the Dockerfile you can update from the output of the run command, the output isn't parsed, only the resulting filesystem is saved. For this, I think you're best off in your run command writing the path to the image and read in from your /etc/profile or a custom entrypoint script. That depends on how you want to launch your container and the base image.