I have a Docker container running Bind9.
Inside the container named is running with bind user
bind 1 0 0 19:23 ? 00:00:00 /usr/sbin/named -u bind -g
In my named.conf.local I have
channel queries_log {
file "/var/log/bind/queries.log";
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
category queries { queries_log; };
After starting the container, the log file is created
-rw-r--r-- 1 bind bind 0 Nov 14 19:23 queries.log
but it always remains empty.
On the other side, the 'queries' logs are still visible using docker logs ...
14-Nov-2018 19:26:10.463 client #0x7f179c10ece0 ...
Using the same config without Docker works fine.
My docker-compose.yml
version: '3.6'
services:
bind9:
build: .
image: bind9:1.9.11.3
container_name: bind9
ports:
- "53:53/udp"
- "53:53/tcp"
volumes:
- ./config/named.conf.options:/etc/bind/named.conf.options
- ./config/named.conf.local:/etc/bind/named.conf.local
My Dockerfile
FROM ubuntu:18.04
ENV BIND_USER=bind \
BIND_VERSION=1:9.11.3
RUN apt-get update -qq \
&& DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends install -y \
bind9=${BIND_VERSION}* \
bind9-host=${BIND_VERSION}* \
dnsutils \
&& rm -rf /var/lib/apt/lists/*
COPY entrypoint.sh /sbin/entrypoint.sh
RUN chmod 755 /sbin/entrypoint.sh
ENTRYPOINT ["/sbin/entrypoint.sh"]
CMD ["/usr/sbin/named"]
-f
Run the server in the foreground (i.e. do not daemonize).
-g
Run the server in the foreground and force all logging to stderr.
Try to use -f instead of -g.
Related
I'm trying to build a containerized application that will basically read from a storage service do some operations and write a "processed" file into the same storage service.
I tried using NFS as a storage service but hit a few bottlenecks which led me to Minio as a storage service, which works out very well for me since I'm already familiar with AWS S3.
So I spun up a Minio container and basically wrote a simple python script which will basically check for .csv files that have been uploaded in the last 60 seconds.
I tested this locally and it worked flawlessly.
This is the sample code -
import time
from minio import Minio
from minio.error import ResponseError
# Initialize the Minio client
host = "minio_server:9000"
access_key = "ROOTNAME"
secret_key="CHANGEME123"
client = Minio (host, access_key=access_key,
secret_key=secret_key, secure=False)
# Set the name of the bucket to monitor
bucket_name = 'my-bucket'
# Set the interval for checking the bucket (in seconds)
check_interval = 60
while True:
# Get the list of objects in the bucket
objects = client.list_objects(bucket_name, prefix='', recursive=True)
# Check the time of each object
for obj in objects:
# Check if the object is a CSV file
if obj.object_name.endswith('.csv'):
# Check if the object was uploaded in the last 60 seconds
current_time = time.time()
if current_time - obj.last_modified.timestamp() < check_interval:
print(f'Found new CSV file: {obj.object_name}')
# Sleep for the specified interval before checking again
time.sleep(check_interval)
And this works exactly as intended.
So next I containerized this application and created a Dockerfile
FROM ubuntu:20.04
RUN apt-get update -y
RUN apt update
RUN apt upgrade -y
RUN apt-get install -y python3
#RUN apt-get install -y python3-pip
RUN apt update
RUN apt upgrade -y
RUN apt-get install -y python3
RUN apt-get install -y python3-pip
RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata
RUN apt-get install -y libenchant1c2a
RUN apt install git -y
RUN pip3 install \
argparse \
boto3 \
numpy==1.20.3 \
scipy \
pandas \
scikit-learn \
matplotlib \
plotly \
kaleido \
fpdf \
regex \
pyenchant \
minio \
pythonping\
openpyxl
RUN git clone https://github.com/nedap/dateinfer.git && \
cd dateinfer && \
pip3 install .
ADD core.py /core.py
CMD ["/core.py"]
ENTRYPOINT ["python3"]
Note- I need the other dependencies for the actual application that I'm going to write once this test passes.
I created a Docker-compose.yaml file to club the two containers.
version: "2.1"
services:
minio_server:
image: quay.io/minio/minio
networks:
- appnet
container_name: minio_server
restart: unless-stopped
privileged: true
command: server --address ":9000" --console-address ":9001" /data
environment:
- MINIO_ROOT_USER=ROOTNAME
- MINIO_ROOT_PASSWORD=CHANGEME123
volumes:
- /minio/data:/data
ports:
- 9090:9090
- 9000:9000
minio_core_server:
image: miniocore:latest
networks:
- appnet
depends_on:
- minio_server
networks:
appnet:
name: appnet
And obviously I Made some changes to the source code to do something super simple -
from minio import Minio
from minio.error import ResponseError
# Set the URL of the Minio server
minio_url = "minio_server:9000"
# Set the access key and secret key for the Minio server
access_key = "ROOTNAME"
secret_key = "CHANGEME123"
# Initialize the Minio client
client = Minio(minio_url,
access_key=access_key,
secret_key=secret_key,
secure=True)
# Set the name of the bucket
imput_bucket= client.make_bucket('sample.bucket')
But I'm getting the following error:
File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 592, in increment
minio_core_server_1 | raise MaxRetryError(_pool, url, error or ResponseError(cause))
minio_core_server_1 | urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='minio_server', port=9000): Max retries exceeded with url: /sample.bucket (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)')))
Trouble shooting steps that I tried:
I initially thought that the service container is unable to establish a connection between the service container and the Minio container:
So I tried to ping the Minio container from the application from the service container
from minio import Minio
#from minio.error import ResponseError
# Set the URL of the Minio server
from pythonping import ping
ping("minio_server", verbose=True)
and this is the response -
minio_core_server_1 | Reply from 192.168.192.2, 29 bytes in 0.03ms
minio_core_server_1 | Reply from 192.168.192.2, 29 bytes in 0.01ms
minio_core_server_1 | Reply from 192.168.192.2, 29 bytes in 0.01ms
minio_core_server_1 | Reply from 192.168.192.2, 29 bytes in 0.01ms
SO it's in fact able to establish a connection with the Minio container
Mine is a bit of a peculiar situation, I created a dockerfile that "works" if not for some proiblems,
Here is a "working" version:
ARG IMGVERS=latest
FROM bensuperpc/tinycore:${IMGVERS}
LABEL maintainer "Vinnie Costante <****#gmail.com>"
ARG DOWNDIR=/tmp/download
ARG INSTDIR=/opt/vscodium
ARG REPOAPI="https://api.github.com/repos/VSCodium/vscodium/releases/latest"
ENV LANG=C.UTF-8 LC_ALL=C PATH="${PATH}:${INSTDIR}/bin/"
RUN tce-load -wic Xlibs nss gtk3 libasound libcups python3.9 tk8.6 \
&& rm -rf /tmp/tce/optional/*
RUN sudo ln -s /lib /lib64 \
&& sudo ln -s /usr/local/etc/fonts /etc/fonts \
&& sudo mkdir -p ${DOWNDIR} ${INSTDIR} \
&& sudo chown -R tc:staff ${DOWNDIR} ${INSTDIR}
#COPY VSCodium-linux-x64-1.57.1.tar.gz ${DOWNDIR}/
RUN wget http://192.168.43.6:8000/VSCodium-linux-x64-1.57.1.tar.gz -P ${DOWNDIR}
RUN tar xvf ${DOWNDIR}/VSCodium*.gz -C ${INSTDIR} \
&& rm -rf ${DOWNDIR}
CMD ["codium"]
The issues are these:
Starting the image with this command vscodium does not start, but entering the shell (adding /bin/ash to the end of the docker run) and then running codium instead vscodium starts. I tried many ways, even changing the entrypoint, the result is always the same. But if I try to add any other graphic program (like firefox) and replace the argument of the CMD instruction inside the dockerfile, everything works as it should.
docker run -it --rm \
--net=host \
--env="DISPLAY=unix${DISPLAY}" \
--workdir /home/tc \
--volume="$HOME/.Xauthority:/root/.Xauthority:rw" \
--name tc \
tinycodium
the last two versions of codium (1.58.0 and 1.58.1) don't work at all on docker but they start normally on the same distro not containerized. I tried installing other dependencies but nothing worked. Right now I don't know how to understand what's wrong with these two new versions.
I don't know how to set a volume to save codium data, I tried something like this --volume=/home/vinnie/docker:/home/tc but there are always problems with user/group permissions. I've also tried booting the container as user by adding it to the docker group but there's always a mess with permissions. If someone could explain me how to proceed, the directories I want to save are these:
/home/tc/.vscode-oss
/home/tc/.cache/mesa_shader_cache
/home/tc/.config/VSCodium
/home/tc/.config/glib-2.0/settings
/home/tc/.local/share
Try running codium --verbose and see if the container starts
For my job, I would like to run Jenkins and Docker Rootless (with the sysbox runtime only for this container), all in Docker Rootless.
I would like this because I need a secure environment given I don't inspect Jenkins pipelines
But when I run docker rootless in docker rootless, I get this error:
[rootlesskit:parent] error: failed to setup UID/GID map: newuidmap 54 [0 1000 1 1 100000 65536] failed: newuidmap: write to uid_map failed: Operation not permitted
: exit status 1
I tried many actions but failed to get it to work. Someone would have a solution to do this, please?
Thanks for reading me, have a nice day!
Edit 1
Hello, I take the liberty of relaunching this question, because being essential for the safety of our environment, my bosses remind me every day. Would someone have the answer to this problem please
Things getting a little tricky when you want to use the docker build command inside a Jenkins container.
I stumbled upon this issue when wanted to build docker images without being root, under the user 'jenkins' instead.
I wrote the solution in an article in which I explain in detail what is happening under the hood.
The key point is to figure out which GID the docker.sock socket is running under (depends on the system). So here is what you gotta do:
Run the command:
$ stat /var/run/docker.sock
Output:
jenkins#wsl-ubuntu:~$ stat /var/run/docker.sock
File: /var/run/docker.sock
Size: 0 Blocks: 0 IO Block: 4096 socket
Device: 17h/23d Inode: 552 Links: 1
Access: (0660/srw-rw----) Uid: ( 0/ root) Gid: ( 1001/ docker)
Access: 2021-03-03 10:43:05.570000000 +0200
Modify: 2021-03-03 10:43:05.570000000 +0200
Change: 2021-03-03 10:43:05.570000000 +0200
Birth: -
In this case, the GID is 1001, but can also be 999 or something else in your machine.
Now, create a Dockerfile and paste the code below replacing the ENV variable with your own from the stat command output above:
FROM jenkins/jenkins:lts-alpine
USER root
ARG DOCKER_HOST_GID=1001 #Replace with your own docker.sock GID
ARG JAVA_OPTS=""
ENV DOCKER_HOST_GID $DOCKER_HOST_GID
ENV JAVA_OPTS $JAVA_OPTS
RUN set -eux \
&& apk --no-cache update \
&& apk --no-cache upgrade --available \
&& apk --no-cache add shadow \
&& apk --no-cache add docker curl --repository http://dl-cdn.alpinelinux.org/alpine/latest-stable/community \
&& deluser --remove-home jenkins \
&& addgroup -S jenkins -g $DOCKER_HOST_GID \
&& adduser -S -G jenkins -u $DOCKER_HOST_GID jenkins \
&& usermod -aG docker jenkins \
&& apk del shadow curl
USER jenkins
WORKDIR $JENKINS_HOME
For the sake of a working example, here is a docker-compose file:
version: '3.3'
services:
jenkins:
image: jenkins_master
container_name: jenkins_master
hostname: jenkins_master
restart: unless-stopped
env_file:
- jenkins.env
build:
context: .
cpus: 2
mem_limit: 1024m
mem_reservation: 800M
ports:
- 8090:8080
- 50010:50000
- 2375:2376
volumes:
- ./jenkins_data:/var/jenkins_home
- /var/run/docker.sock:/var/run/docker.sock
networks:
- default
volumes:
jenkins_data: {}
networks:
default:
driver: bridge
Now lets create the ENV variables:
cat > jenkins.env <<EOF
DOCKER_HOST_GID=1001 #Replace with your own docker.sock GID
JAVA_OPTS=-Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
EOF
and lastly, run the command docker-compose up -d.
It will build the image, and run it.
Then visit HTTP://host_machine_ip:8090 , and that's all.
If you run docker inspect --format '{{ index (index .Config.Env) }}' jenkins_master you will see that the 1st and 2nd variables are the ones we set.
More details can be found here: How to run rootless docker in dockerized Jenkins installation
I have a local dev env which requires different hosts reachable via SSH so i set up a docker-compose.yml with some services:
services:
ssh1:
build:
context: ./.project/docker/ssh1
dockerfile: Dockerfile
environment:
MYSQL_USER: app1
MYSQL_PASSWORD: app1
The Dockerfile contains following contents:
FROM ubuntu:20.04
RUN export DEBIAN_FRONTEND=noninteractive \
&& ln -fs /usr/share/zoneinfo/Europe/Berlin /etc/localtime \
&& apt update \
&& apt upgrade -y \
&& apt install -y openssh-server rsync php \
&& mkdir /run/sshd/ \
&& ssh-keygen -A \
&& for key in $(ls /etc/ssh/ssh_host_* | grep -v pub); do echo "HostKey $key" >> /etc/ssh/sshd_config; done \
&& addgroup --gid 1000 app \
&& adduser --gecos "" --disabled-password --shell /bin/bash --uid 1000 --gid 1000 app \
&& mkdir -m 700 /home/app/.ssh/ \
&& chown app:app /home/app/.ssh/ \
&& rm -rf /var/lib/apt/lists/*
COPY --chown=app:app ssh1_rsa.pub /home/app/.ssh/authorized_keys
CMD ["/usr/sbin/sshd", "-D"]
EXPOSE 22
I can verify, that the environment variables are set in the container
$ docker-compose exec ssh1 printenv | grep MYSQL
MYSQL_USER=app1
MYSQL_PASSWORD=app1
docker inspect project_ssh1_1 also shows the ENV variables.
But when i connect from another container to ssh1 via ssh, my environment variables are not set.
Why are my environment variables not set when i ssh into the container?
I also appreciate any in-depth input on how env vars are set in the container via docker and how env vars are inherited from processes or the "OS".
Solution edit:
The actual question was answered. However, i did not ask the right question. So here's my actual solution. I am able to set the ENV VARS in the SSH session with the help of a pretty hacky solution, which should not be used in PROD environments, since it could lead to information disclosure.
Add all ENV VARS as build args.
docker-compose.yml:
ssh1:
build:
context: ./.project/docker/ssh1
dockerfile: Dockerfile
args:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: app1
MYSQL_USER: app1
MYSQL_PASSWORD: app1
MYSQL_HOST: mysql1
And write them to $HOME/.ssh/environment as well as enabling PermitUserEnvironment. Don't do this in production
Dockerfile
FROM ubuntu:20.04
ARG MYSQL_ROOT_PASSWORD
ARG MYSQL_DATABASE
ARG MYSQL_USER
ARG MYSQL_PASSWORD
ARG MYSQL_HOST
RUN echo "MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD" >> /home/app/.ssh/environment \
&& echo "MYSQL_DATABASE=$MYSQL_DATABASE" >> /home/app/.ssh/environment \
&& echo "MYSQL_USER=$MYSQL_USER" >> /home/app/.ssh/environment \
&& echo "MYSQL_PASSWORD=$MYSQL_PASSWORD" >> /home/app/.ssh/environment \
&& echo "MYSQL_HOST=$MYSQL_HOST" >> /home/app/.ssh/environment \
&& sed -i 's/#PermitUserEnvironment no/PermitUserEnvironment yes/g' /etc/ssh/sshd_config
Now, when you login, SSH will read the env vars from the users .ssh/environment (app in this case) and set them in the user's SSH session.
Environment variables are present in RUN commands and in the shell you exec into when issuing docker exec command, but when you ssh into an ssh server running inside container, you actually get a brand new shell which doesn’t have those env variables set.
Your issue has actually not much to do with docker but is due to the way sshd works: for every connection sshd will setup a new environment wiping out all variables in its own environment, see the Login Process in man sshd.
(It's easy to see why this makes sense: sshd is started by the root user and so may contain sensitive data in its environment variables that should not be leaked to the users. Other variables wouldn't be reasonable to pass to the user session, e.g. HOME, PATH, SHELL)
Depending on your use case there are various ways to pass environment variables to a new ssh session, depending on whether it being an interactive or non-interactive session, running a (login) shell or not:
~/.ssh/environment: variables injected by ssh, see PermitUserEnvironment
/etc/environment: used by pam_env on login
/etc/profile, ~/.bashrc and alike: configs used by the (login) shell, see bash for example.
Also depending on your use case you have now various options how to add these files to the container:
if the variables are static: just ADD the respective file to the image or create it in the Dockerfile
if the variables are set on build-time: use ARGs (or ENV) to pass the variables to the build and create the respective file from that in the build (as you did in your solution)
if the variables should be set on container run-time:
use a custom ENTRYPOINT script to generate the respective file on startup from the passed environment variables or command line arguments
volume-mount the respective file into the container (you may also use docker secret for sensitive data here)
I'm running Celery under Docker Compose. I'd like to make Celery's Flower persistent. So I do:
version: '2'
volumes:
[...]
flower_data: {}
[...]
flower:
image: [base code image]
ports:
- "5555:5555"
volumes:
- flower_data:/flower
command:
celery -A proj flower --port=5555 --persistent=True --db=/flower/flower
However, then I get:
IOError: [Errno 13] Permission denied: 'flower.dat'
I ran the following to elucidate why:
bash -c "ls -al /flower; whoami; celery -A proj flower --persistent=True --db=/flower/flower"
This made it clear why:
flower_1 | drwxr-xr-x 3 root root 4096 Mar 10 23:05 .
flower_1 | drwxr-xr-x 7 root root 4096 Mar 10 23:05 ..
Namely, the directory is mounted as root, yet in [base code image] I ensure the user running is not root, as per Celery's docks to never run as root:
FROM python:2.7
...
RUN groupadd user && useradd --create-home --home-dir /usrc/src/app -g user user
USER user
What would be the best way for Celery Flower to continue to run not as root, yet be able to use this named volume?
The following works: In the Dockerfile, install sudo and add user to the sudo group, requiring a password:
RUN apt-get update
RUN apt-get -y install sudo
RUN echo "user:SECRET" | chpasswd && adduser user sudo
Then, in the Docker Compose config, the command will be:
bash -c "echo SECRET | sudo -S chown user:user /flower; celery -A proj flower --power=5555 --persistent --db=/flower/flower"
I'm not sure if this is the best way, though, or what the security implications of this are.