How to pass a YML config file in Docker Image during run - docker

I have built a docker image. It has a YML config file which is a large configuration something like this
contrydata:
continent: europe
country: luxembourg
county: luxembourg
town: luxembourg
...
I am using this configuration file during the run of docker. I want to use this as a default configuration. If any user wants to change the configuration how does this config file passes to the docker image during the run.
I have seen a few docker run command like this which uses an external config file
docker run --rm -it -v $(pwd):/data -p 80:80 dockerhub/mydockerimage:v3 --config config.yml
Where users can configure this file and can pass it from an external source.
In a nutshell, if users pass a config file then the default config file will be replaced with the provided config.yml file otherwise the default config.yml file will be used.

The way this is usually done, as far as I can tell, is to map the config file to a specific location in the container. Nginx does this and has a volume mapping on the docker run command like this -v /host/path/nginx.conf:/etc/nginx/nginx.conf:ro. The Nginx container then always looks for the config file in the same location - /etc/nginx/nginx.conf.
I've never seen it passed as a parameter as you show. Those parameters will be passed to the program defined as the entrypoint in the container. That program doesn't have access to the host file system, so unless the config file has been mapped to the container using a volume, the program won't have access to the file.
Edit:
Just for the fun of it, I tried making this Dockerfile that builds a small C program that prints out any arguments passed to it. I also created a shell script that passes any arguments on to pgm. This is pretty much how Tileserver does it.
FROM ubuntu
RUN apt-get update && apt-get install -y gcc
WORKDIR /app
RUN echo '#include <stdio.h> \n\
int main(int argc, char **argv) { \n\
for (int i=0; i<argc; i++) { \n\
printf("%s\\n", argv[i]); \n\
} \n\
}' > pgm.c
RUN gcc -o pgm pgm.c
RUN echo '#!/bin/bash\n\
./pgm "$#"' > docker-entrypoint.sh
RUN chmod 775 docker-entrypoint.sh
ENTRYPOINT ["./docker-entrypoint.sh"]
If you build and run it using the commands
docker build -t test .
docker run --rm test --config config.file
it prints out
./pgm
--config
config.file
If you wanted to do it like Tileserver, the C program would then need to open config.file and use the contents. But of course it would only be able to get at it, if the file is mapped using a volume.

Related

How can I use a several line command in a Dockerfile in order to create a file within the resulting Image

I'm following installation instructions for RedhawkSDR, which rely on having a Centos7 OS. Since my machine uses Ubuntu 22.04, I'm creating a Docker container to run Centos7 then installing RedhawkSDR in that.
One of the RedhawkSDR installation instructions is to create a file with the following command:
cat<<EOF|sed 's#LDIR#'`pwd`'#g'|sudo tee /etc/yum.repos.d/redhawk.repo
[redhawk]
name=REDHAWK Repository
baseurl=file://LDIR/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhawk
EOF
How do I get a Dockerfile to execute this command when creating an image?
(Also, although I can see that this command creates the file /etc/yum.repos.d/redhawk.repo, which consists of the lines from [redhawk] to gpgkey=...., I have no idea how to parse this command and understand exactly why it does that...)
Using the text editor of your choice, create the file on your local system. Remove the word sudo from it; give it an additional first line #!/bin/sh. Make it executable using chmod +x create-redhawk-repo.
Now it is an ordinary shell script, and in your Dockerfile you can just RUN it.
COPY create-redhawk-repo ./
RUN ./create-redhawk-repo
But! If you look at what the script actually does, it just writes a file into /etc/yum.repos.d with a LDIR placeholder replaced with some other directory. The filesystem layout inside a Docker image is fixed, and there's no particular reason to use environment variables or build arguments to hold filesystem paths most of the time. You could use a fixed path in the file
[redhawk]
name=REDHAWK Repository
baseurl=file:///redhawk-yum/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhawk
and in your Dockerfile, just COPY that file in as-is, and make sure the downloaded package archive is in that directory. Adapting the installation instructions:
ARG redhawk_version=3.0.1
RUN wget https://github.com/RedhawkSDR/redhawk/releases/download/$redhawk_version/\
redhawk-yum-$redhawk_version-el7-x86_64.tar.gz \
&& tar xzf redhawk-yum-$redhawk_version-el7-x86_64.tar.gz \
&& rm redhawk-yum-$redhawk_version-el7-x86_64.tar.gz \
&& mv redhawk-yum-$redhawk_version-el7-x86_64 redhawk-yum \
&& rpm -i redhawk-yum/redhawk-release*.rpm
COPY redhawk.repo /etc/yum.repos.d/
Remember that, in a Dockerfile, you are root unless you've switched to another USER (and in that case you can use USER root to switch back); you do not need generally sudo in Docker at all, and can just delete sudo where it appears in these instructions.
How do I get a Dockerfile to execute this command when creating an image?
Just use printf and run this command as single line:
FROM image_name:image_tag
ARG LDIR="/default/folder/if/argument/not/set"
# if container has sudo command and default user is not root
# you should choose this variant
RUN printf '[redhawk]\nname=REDHAWK Repository\nbaseurl=file://%s/\nenabled=1\ngpgcheck=1\ngpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhawk\n' "$LDIR" | sudo tee /etc/yum.repos.d/redhawk.repo
# if default container user is root this command without piping may be used
RUN printf '[redhawk]\nname=REDHAWK Repository\nbaseurl=file://%s/\nenabled=1\ngpgcheck=1\ngpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhawk\n' "$LDIR" > /etc/yum.repos.d/redhawk.repo
Where LDIR is an argument and docker build process should be run like:
docker build ./ --build-arg LDIR=`pwd`

Best practice to include a bash script in a Docker image

I'm creating a Dockerfile that needs to execute a command, let's call it foo
In order to execute foo, I need to create a .cfc in current directory with token information to call this foo service.
So basically I should do something like
ENV FOO_TOKEN token
ENV FOO_HOST host
ENV FOO_SHARED_DIRECTORY directory
ENV LIBS_TARGET target
and then put the first three variables in a .cfg file and then launch a command using the last variable as target.
Given that if run more than one CMD in a Dockerfile, only the last one will be considered, how should I do that?
My ideal execution is docker run -e "FOO_TOKEN=aaaaaaa" -e "FOO_HOST=myhost" -e "FOO_SHARED_DIRECTORY=Shared" -e "LIBS_TARGET=target/scala-2.11/*.jar" -it --rm --name my-ci-deploy foo/foo:latest
If you wanted to keep everything in the Dockerfile (something I think is rather desirable), you can do something nasty like:
ENV SCRIPT=IyEvdXNyL2Jpbi9lbnYgYmFzaApwZG9fc3Fsc3J2PTAKc3Vkbz0KdmVuZG9yPSQoIGxzYl9yZWxlYXNlIC1p
RUN echo -n "$SCRIPT" | base64 -d | /usr/bin/env bash
Where the contents of SCRIPT= are derived by piping your shell script thusly:
cat my_script.sh | base64 --wrap=0
You may have to adjust the /usr/bin/env bash if you have a really minimal (Alpine) setup.

Pipe output from Dockerfile RUN command to host

I have the following image:
FROM some:image
ADD app /app
ADD https://get.aquasec.com/microscanner /
RUN chmod +x /microscanner
RUN /microscanner my_xxx_token >> /microscan.log
the RUN /microscanner command outputs a bunch of stuff. If I don't >> pipe it to a file, it's gonna be printed to my console.
What I want is to pipe it directly to the host, because the command, if having a false result, interrupts the build of the image, so I cannot execute any other Docker commands after it. Even a && cat /microscan.log on the same RUN command after the microscanner will not execute.
I tried doing >> /app/microscanner.log since the folder is used as a shared volume, but the file doesn't appear on the host.
So, I assume that I have to write from the container directly into a file on the host. Is that a possibility at all?

Docker Logs Issue : Logs are not created or displayed in Tomcat's logs folder in docker container

We are using Docker container and created a Dockerfile. Inside this container we deployed war file using tomcat image
and we can see tomcat logs at console but console logs is not updating
after sending a request to tomcat via URL.
Also we can not see any log file inside tomcat logs folder
Can anyone help me out that how we can see tomcat logs like localhost.logs/catalina.logs/manager.logs etc
MY Dockerfile is :-
FROM openjdk:6-jre
ENV CATALINA_HOME /usr/local/tomcat
ENV PATH $CATALINA_HOME/bin:$PATH
COPY tomcat $CATALINA_HOME
ADD newui.war $CATALINA_HOME/webapps
CMD $CATALINA_HOME/bin/startup.sh && tail -F $CATALINA_HOME/logs/catalina.out
EXPOSE 8080
Used below script to build
$ docker build -t tomcat .
and below used to run tomcat
$ docker run -p 8080:8080 tomcat
Here are a few things wrong with your dockerfile:
You mention that you need java 6, and yet the line FROM java as of this writing is set to use java:8.
You need to replace the FROM line with FROM java:6-jre or as suggested by the official page: FROM openjdk:6-jre if in 2018 you still need java 6, which is dangerous. I would also strongly suggest to use at least FROM tomcat:7 which should be able to run java 6 applets but will include some bug fixes including support for longer Diffie-Hellman primes for HTTPS (if you are serious about your app's security).
Copt tomcat $CATALINA_HOME you either miss-typed the line to SO, or your image should not build at all. It should be COPY tomcat $CATALINA_HOME
Given that you are using the COPY command there is no need to use RUN mkdir -p prior to this, since the COPY command will automatically create all the required folders.
CMD $CATALINA_HOME/bin/startup.sh && tail -f $CATALINA_HOME/logs/catalina.out
First the tail -f part: since you are looking to tail a log file which might be created and recreated during the server's operation instead of following the FD you should be following the path by doing tail -F (capital F)
startup.sh && tail - tail will never start until startup.sh exits. A better approach is to do tail -F $CATALINA_HOME/logs/catalina.out & inside your startup.sh right before you start your tomcat server. That way tail will be running in the background.
Regardless this is a somewhat dangerous approach and you risk zombie processes because bash does not manage its children processes and neither does docker. I would recommend to use supervisord or something similar.
(From https://docs.docker.com/engine/admin/multi-service_container/)
FROM ubuntu:latest
RUN apt-get update && apt-get install -y supervisor
RUN mkdir -p /var/log/supervisor
COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
COPY my_first_process my_first_process
COPY my_second_process my_second_process
CMD ["/usr/bin/supervisord"]
Note: this dockerfile sample omits a few of the best practices, e.g. removing the apt cache in the same run command as doing the apt-get update.
Personal favorite is the phusion/baseimage, but it is harder to setup since you'll need to install everything including the java into the image.
If with all of these modifications you still have no luck in seeing the console update, then you'll need to also post the contents of your startup.sh file or other tomcat related configurations.
P.S.: it might be a good idea to do RUN mkdir -p $CATALINA_HOME/logs just to make sure that the logs folder exists for tomcat to write to.
P.P.S.: the java base image is actually using openjdk instead of the oracle one. Just thought I'd point it out
You should check tomcat logging settings. The default logging.properties in the JRE specifies a ConsoleHandler that routes logging to System.err. The default conf/logging.properties in Apache Tomcat also adds several FileHandlers that write to files.
Example logging.properties file to be placed in $CATALINA_BASE/conf:
handlers = 1catalina.org.apache.juli.FileHandler, \
2localhost.org.apache.juli.FileHandler, \
3manager.org.apache.juli.FileHandler, \
java.util.logging.ConsoleHandler
.handlers = 1catalina.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler
############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################
1catalina.org.apache.juli.FileHandler.level = FINE
1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.FileHandler.prefix = catalina.
2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.FileHandler.prefix = localhost.
3manager.org.apache.juli.FileHandler.level = FINE
3manager.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
3manager.org.apache.juli.FileHandler.prefix = manager.
3manager.org.apache.juli.FileHandler.bufferSize = 16384
java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
############################################################
# Facility specific properties.
# Provides extra control for each logger.
############################################################
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = \
2localhost.org.apache.juli.FileHandler
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].handlers = \
3manager.org.apache.juli.FileHandler
# For example, set the org.apache.catalina.util.LifecycleBase logger to log
# each component that extends LifecycleBase changing state:
#org.apache.catalina.util.LifecycleBase.level = FINE
Example logging.properties for the servlet-examples web application to be placed in WEB-INF/classes inside the web application:
handlers = org.apache.juli.FileHandler, java.util.logging.ConsoleHandler
############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################
org.apache.juli.FileHandler.level = FINE
org.apache.juli.FileHandler.directory = ${catalina.base}/logs
org.apache.juli.FileHandler.prefix = servlet-examples.
java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
More info at https://tomcat.apache.org/tomcat-6.0-doc/logging.html
we can not see the logs in Docker container until unless we mount it.
To build the Dockerfile:-
docker build -t tomcat
To run the Dockerfile Image:-
docker run -p 8080:8080 tomcat
To copy the logs of tomcat present in docker container to mounted container :-
Run this cmd to mount the container:
1stpath : 2ndpath
docker run \\-d \\-p 8085:8085 \\-v /usr/local/tomcat/logs:/usr/local/tomcat/logs \tomcat
or simply
docker run \\-d \\-v /usr/local/tomcat/logs:/usr/local/tomcat/logs \tomcat
1st:-/usr/local/tomcat/logs: path of root dir: where we want to copy
the logs or destination
2nd:- /usr/local/tomcat/logs: path of tomcat/logs folder present in
docker container
tomcat:-name of image
need to change the port if it is busy
now the container is get mount
to get the list of container run : docker ps -a
now get the container id of latest created container:
docker exec -it < mycontainer > bash
then we can see the logs by
cd /usr/local/tomcat/logs
usr/local/tomcat/logs# less Log Name Here
this to Copy any folder in docker container on root:-
docker cp <containerId>:/file/path/within/container /host/path/target

Understanding the difference in sequence of ENTRYPOINT/CMD between Dockerfile and docker run

Docker noob here...
I am trying to build and run an IBM DataPower container from a Dockerfile, but it doesn't seem to work the same as when just running docker run and passing the same parameters in the terminal.
This works (docker run)
docker run -it \
-v $PWD/config:/drouter/config \
-e DATAPOWER_ACCEPT_LICENSE=true \
-e DATAPOWER_INTERACTIVE=true \
-e DATAPOWER_WORKER_THREADS=4 \
-p 9090:9090 \
--name mydatapower \
ibmcom/datapower
... the key part being that it mounts the ./config folder and the custom configuration is picked up by datapower running in the container.
This doesn't (Dockerfile)
Dockerfile:
FROM ibmcom/datapower
ENV DATAPOWER_ACCEPT_LICENSE=true
ENV DATAPOWER_INTERACTIVE=true
ENV DATAPOWER_WORKER_THREADS=4
EXPOSE 9090
COPY config/auto-startup.cfg /drouter/config/auto-startup.cfg
Build:
docker build -t local/datapower .
Run:
docker run -it \
-p 9090:9090 \
--name mydatapower local/datapower
The problem is that DataPower doesn't pick up the auto-startup.cfg file, so the additional config options doesn't get used. I know the source file path is correct because if I misspell the file name docker throws an error.
I have a theory that it might be running the inherited ENTRYPOINT or CMD before the config file is available. I don't know how to test or prove this. I don't know what the ENTRYPOINT or CMD is because the inherited image is not open source and I can't figure out how to find it.
Does that seem likely?
UPDATE:
The content of the auto-startup.cfg is:
top; co
ssh
web-mgmt
admin enabled
port 9090
exit
It simply enables the DataPower WebGUI.
The output when running it in the commandline with:
docker run -it -v $PWD/config:/drouter/config -v $PWD/local:/drouter/local -e DATAPOWER_ACCEPT_LICENSE=true -e DATAPOWER_INTERACTIVE=true -e DATAPOWER_WORKER_THREADS=4 -p 9091:9090 --name myconfigureddatapower ibmcom/datapower`
...contains this:
20170908T121729.015Z [0x8100006e][system][notice] : Executing startup configuration.
20170908T121729.970Z [0x00350014][mgmt][notice] web-mgmt(WebGUI-Settings): tid(303): Operational state up
...but with Dockerfile it doesn't. That's why I think the config files may be copied into place too late.
I've tried adding CMD ["/bin/drouter"] to the end of my Dockerfile to no avail.
I have tested your Dockerfile and it seems to be working. My auto-startup.cfg file is copied in the proper location and when I launch the container it's reading the file.
I get this output:
[root#ip-172-30-2-164 tmp]# docker run -ti -p 9090:9090 test
20170908T123728.818Z [0x8040006b][system][notice] logging target(default-log): Logging started.
20170908T123729.067Z [0x804000fe][system][notice] : Container instance UUID: 36bcca0e-6139-4694-91b0-2b7b66c3a498, Cores: 4, vCPUs: 4, CPU model: Intel(R) Xeon(R) CPU E5-2676 v3 # 2.40GHz, Memory: 16049.1MB, Platform: docker, OS: dpos, Edition: developers-limited, Up time: 0 minutes
20170908T123729.071Z [0x8040001c][system][notice] : DataPower IDG is on-line.
20170908T123729.071Z [0x8100006f][system][notice] : Executing default startup configuration.
20170908T123729.416Z [0x8100006d][system][notice] : Executing system configuration.
20170908T123729.417Z [0x8100006b][mgmt][notice] domain(default): tid(8143): Domain operational state is up.
708f98be1390
Unauthorized access prohibited.
20170908T123731.239Z [0x806000dd][system][notice] cert-monitor(Certificate Monitor): tid(399): Enabling Certificate Monitor to scan once every 1 days for soon to expire certificates
20170908T123731.552Z [0x8100006e][system][notice] : Executing startup configuration.
20170908T123732.436Z [0x8100003b][mgmt][notice] domain(default): Domain configured successfully.
20170908T123732.449Z [0x00350014][mgmt][notice] web-mgmt(WebGUI-Settings): tid(303): Operational state up
login:
To check that your file has been copied to the container you can run docker run -ti local/datapower sh to enter the container and then check the content of /drouter/config/.
Your base image command is: CMD ["/bin/drouter"] you can check it running docker history ibmcom/datapower.
UPDATE:
The drouter user in the container must be able to read the auto-startup.cfg file. You have 2 options:
set your local auto-startup.cfg with the proper permissions (chmod 644 config/autostart.cfg).
or add these line in the Dockerfile so drouter can read the file:
USER root
RUN chown drouter /drouter/config/auto-startup.cfg
USER drouter

Resources