Docker: Change word in file at container startup - docker

I'm creating a docker image for our fluentd.
The image contains a file called http_forward.conf
It contains:
<store>
type http
endpoint_url ENDPOINTPLACEHOLDER
http_method post # default: post
serializer json # default: form
rate_limit_msec 100 # default: 0 = no rate limiting
raise_on_error true # default: true
authentication none # default: none
username xxx # default: ''
password xxx # default: '', secret: true
</store>
So this is in our image. But we want to use the image for all our environments. Specified with environment variables.
So we create an environment variable for our environment:
ISSUE_SERVICE_URL = http://xxx.dev.xxx.xx/api/fluentdIssue
This env variable contains dev on our dev environment, uat on uat etc.
Than we want to replace our ENDPOINTPLACEHOLDER with the value of our env variable. In bash we can use:
sed -i -- 's/ENDPOINTPLACEHOLDER/'"$ISSUE_SERVICE_URL"'/g' .../http_forward.conf
But how/when do we have to execute this command if we want to use this in our docker container? (we don't want to mount this file)

We did that via ansible coding.
Put the file http_forward.conf as template, and deploy the change depend on the environment, then mount the folder (include the conf file) to docker container.
ISSUE_SERVICE_URL = http://xxx.{{ environment }}.xxx.xx/api/fluentdIssue
playbook will be something like this, I don't test it.
- template: src=http_forward.conf.j2 dest=/config/http_forward.conf mode=0644
- docker:
name: "fluentd"
image: "xxx/fluentd"
restart_policy: always
volumes:
- /config:/etc/fluent

In your DockerFile you should have a line starting with CMD somewhere. You should add it there.
Or you can do it cleaner: set the CMD line to call a script instead. For example CMD ./startup.sh. The file startup.sh will then contain your sed command followed by the command to start your fluentd (I assume that is currently the CMD).

Related

Apache Nifi (on docker): only one of the HTTP and HTTPS connectors can be configured at one time error

Have a problem adding authentication due to a new needs while using Apache NiFi (NiFi) without SSL processing it in a container.
The image version is apache/nifi:1.13.0
It's said that SSL is unconditionally required to add authentication. It's recommended to use tls-toolkit in the NiFi image to add SSL. Worked on the following process:
Except for environment variable nifi.web.http.port for HTTP communication, and executed up the standalone mode container with nifi.web.https.port=9443
docker-compose up
Joined to the container and run the tls-toolkit script in the nifi-toolkit.
cd /opt/nifi/nifi-toolkit-1.13.0/bin &&\
sh tls-toolkit.sh standalone \
-n 'localhost' \
-C 'CN=yangeok,OU=nifi' \
-O -o $NIFI_HOME/conf
Attempt 1
Organized files in directory $NIFI_HOME/conf. Three files keystore.jks, truststore.jsk, and nifi.properties were created in folder localhost that entered the value of the option -n of the tls-toolkit script.
cd $NIFI_HOME/conf &&
cp localhost/*.jks .
The file $NIFI_HOME/conf/localhost/nifi.properties was not overwritten as it is, but only the following properties were imported as a file $NIFI_HOME/conf/nifi.properties:
nifi.web.http.host=
nifi.web.http.port=
nifiweb.https.host=localhost
nifiweb.https.port=9443
Restarted container
docker-compose restart
The container died with below error log:
Only one of the HTTP and HTTPS connectors can be configured at one time
Attempt 2
After executing the tls-toolkit script, all files a were overwritten, including file nifi.properties
cd $NIFI_HOME/conf &&
cp localhost/* .
Restarted container
docker-compose restart
The container died with the same error log
Hint
The dead container volume was also accessible, so copied and checked file nifi.properties, and when did docker-compose up or restart, it changed as follows:
The part I overwritten or modified:
nifi.web.http.host=
nifi.web.http.port=
nifi.web.http.network.interface.default=
#############################################
nifi.web.https.host=localhost
nifi.web.https.port=9443
The changed part after re-executing the container:
nifi.web.http.host=a8e283ab9421
nifi.web.http.port=9443
nifi.web.http.network.interface.default=
#############################################
nifi.web.https.host=a8e283ab9421
nifi.web.https.port=9443
I'd like to know how to execute the container with http.host, http.port empty. docker-compose.yml file is as follows:
version: '3'
services:
nifi:
build:
context: .
args:
NIFI_VERSION: ${NIFI_VERSION}
container_name: nifi
user: root
restart: unless-stopped
network_mode: bridge
ports:
- ${NIFI_HTTP_PORT}:8080/tcp
- ${NIFI_HTTPS_PORT}:9443/tcp
volumes:
- ./drivers:/opt/nifi/nifi-current/drivers
- ./templates:/opt/nifi/nifi-current/templates
- ./data:/opt/nifi/nifi-current/data
environment:
TZ: 'Asia/Seoul'
########## JVM ##########
NIFI_JVM_HEAP_INIT: ${NIFI_HEAP_INIT} # The initial JVM heap size.
NIFI_JVM_HEAP_MAX: ${NIFI_HEAP_MAX} # The maximum JVM heap size.
########## Web ##########
# NIFI_WEB_HTTP_HOST: ${NIFI_HTTP_HOST} # nifi.web.http.host
# NIFI_WEB_HTTP_PORT: ${NIFI_HTTP_PORT} # nifi.web.http.port
NIFI_WEB_HTTPS_HOST: ${NIFI_HTTPS_HOST} # nifi.web.https.host
NIFI_WEB_HTTP_PORT: ${NIFI_HTTPS_PORT} # nifi.web.https.port
Thank you

Logstash missing config

I have following issue . Every time when I'm trying to set config for logstash it doesn't see my file. I am sure that the path is properly set.
There is info:
[2018-09-14T09:28:44,073][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/home/jakub/IdeaProjects/test/logstash.conf"}
My docker-compose.yml looks following:
logstash:
image: docker.elastic.co/logstash/logstash:6.4.0
networks: ['stack']
ports:
- "9290:9290"
- "4560:4560"
command: logstash -f /home/jakub/IdeaProjects/test/logstash.conf
depends_on: ['elasticsearch']
and logstash.conf:
input {
redis {
host => "redis"
key => "log4j2"
data_type => "list"
password => "RedisTest"
}
}
output {
elasticsearch {
host => "elasticsearch"
}
}
What I'm doing wrong ? Can you give me some advice or solve my issue ?
Thanks for everything.
Cheers
I guess your logstash.conf is on your host under /home/jakub/IdeaProjects/test/logstash.conf.
Thus, it is not inside your container (unless there is some hidden mount). The command will be executed from within the container, thus it points to a non-existing file.
So, you may use docker cp /home/jakub/IdeaProjects/test/logstash.conf :/home/jakub/IdeaProjects/test/logstash.conf (provided the directory exists in your container)
... or (better!) mount the path from your host to your container. Such as :
volumes:
- /home/jakub/IdeaProjects/test/logstash.conf:/home/jakub/IdeaProjects/test/logstash.conf:ro
.. or to use config (the best option to moo if you are in swarm mode!). The mount is close to the "volume" option above, but you also have to pre-create (from command line or from docker-compose file)
There are other options, but the main point is that you have to make your file available from within your container!

How to create proper .lando.yml custom file?

is there any way how to create a proper, really custom .lando.yml file so it will not use any recipe? How do I specify "just give me Apache, MariaDB, PHP" in Lando?
I tried this
# The name of the app
name: mariadb
# Give me http://mariadb.lndo.site and https://mariadb.lndo.site
proxy:
html:
- mariadb.lndo.site
# Set up my services
services:
# Set up a basic webserver running the latest nginx with ssl turned on
html:
type: nginx
ssl: true
webroot: www
# Spin up a mariadb container called "database"
# NOTE: "database" is arbitrary, you could just as well call this "db" or "kanye"
database:
# Use mariadb version 10.1
type: mariadb:10.1
# Optionally allow access to the database at localhost:3307
# You will need to make sure port 3307 is open on your machine
#
# You can also set `portforward: true` to have Lando dynamically assign
# a port. Unlike specifying an actual port setting this to true will give you
# a different port every time you restart your app
portforward: 3307
# Optionally set the default db credentials
#
# Note: You will need to `lando destroy && lando start` to change these if you've
# already started your app
# See: https://docs.devwithlando.io/tutorials/lando-info.html
creds:
user: mariadb
password: mariadb
database: mariadb
# Optionally load in all the mariadb config files in the config directory
# This is relative to the app root
# NOTE: these files need to end in .cnf
config:
confd: config
but after lando start I am getting a ERROR: No such service: appserver error and the documentation for this is extremely confusing.
Thanks.
You'll want to look at the Building a Custom Stack section of the lando custom project page.
I won't do your entire project, but the basics are as follows:
# LAMP stack example
name: lamp
proxy:
appserver:
- lamp.lndo.site # Allows you to access the site at http[s]://lamp.lndo.site
# This may actually get done automatically
services: # Define your services
appserver: # Create a web server container
type: php:5.3 # Specify what version of php to use
via: apache # This could be nginx, should you choose so
webroot: www # Specify webroot
config: # If you want to add/edit
server: config/apache/lamp.conf # Use an alternate apache config file
conf: path/from/app/root/php.ini # Alter php configuration with a custom file
database: # Create a database server container
type: mysql
portforward: 3308
creds: # Specify what creds/db to use
user: lamp
password: lamp
database: lamp
tooling: # These toolings allow you to connect land <command> to the appropriate containers
composer: # Call with "lando composer..."
service: appserver
description: Run composer commands
cmd: composer --ansi
php: # Call with "lando php..."
service: appserver
mysql: # Call with "lando mysql..."
user: root
service: database
description: Drop into a MySQL shell

configure docker variables with ansible

I have a docker image for an FTP server in a repository, this image will be used for several machines, I need to deploy container and change PORT variable depending on the destination machine.
This is my image (I've deleted lines for proftpd installation cause it is not relevant to this case):
FROM alpine:3.5
ARG vcs_ref="Unknown"
ARG build_date="Unknown"
ARG build_number="1"
LABEL org.label-schema.vcs-ref=$vcs_ref \
org.label-schema.build-date=$build_date \
org.label-schema.version="alpine-r${build_number}"
ENV PORT=10000
COPY assets/port.conf /usr/local/etc/ports.conf
COPY replace.sh /replace.sh
#It is for a proFTPD Server
CMD ["/replace.sh"]
My port.conf file (Also deleted not relevant information for this case)
# This is a basic ProFTPD configuration file (rename it to
# 'proftpd.conf' for actual use. It establishes a single server
# and a single anonymous login. It assumes that you have a user/group
# "nobody" and "ftp" for normal operation and anon.
ServerName "ProFTPD Default Installation"
ServerType standalone
DefaultServer on
# Port 21 is the standard FTP port.
Port {{PORT}}
.
.
.
And replace.sh script is:
#!/bin/bash
set -e
[ -z "${PORT}" ] && echo >&2 "PORT is not set" && exit 1
sed -i "s#{{PORT}}#$PORT#g" /usr/local/etc/ports.conf
/usr/local/sbin/proftpd -n -c /usr/local/etc/proftpd.conf
... Is there any way to avoid using replace.sh and use ansible as the one who replace PORT variable in /usr/local/etc/proftpd.conf the file inside the container?
My actual ansible script for container is:
- name: (ftpd) Run container
docker_container:
name: "myimagename"
image: "myimage"
state: present
pull: true
restart_policy: always
env:
"PORT": "{{ myportUsingAnsible}}"
networks:
- name: "{{ network }}"
Resuming all that I need is to use Ansible to replace configuration variable instead of using a shell script that replaces variables before running services, is it possible?
Many thanks
You are using the docker_container module which will need a pre-built image. The file port.conf is baked inside the image. What you need to do is set a static port inside this file. Inside the container, you always use
the static port 21 and depending on the machine, you map this port onto a different port using ansible.
Inside port.conf always use port 21
# Port 21 is the standard FTP port.
Port 21
The ansible task would look like:
- name: (ftpd) Run container
docker_container:
name: "myimagename"
image: "myimage"
state: present
pull: true
restart_policy: always
networks:
- name: "{{ network }}"
ports:
- "{{myportUsingAnsible}}:21"
Now when you connect to the container, you need to use the <hostnamne>:{{myportUsingAnsible}}. This is the standard docker way of doing things. The port inside the image is static and you change the port mappings based on the
available ports that you have.

How to pass variable as attribute to xml configuration file in Wildfly with Docker

I'm trying to pass values from docker-compose.yml file to Wildfly configuration dynamically.
I want to have flexibility of mail configuration - just for quick change of addres, or username, or port..
In this case, I tried to do that by forwarding environment variables from docker-compose.yml, by dockerfile as arguments "-Dargumentname=$environmentvariable.
Currently wildfly interupts on start with error:
[org.jboss.as.controller.management-operation] (ServerService Thread
Pool -- 45) WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "mail"),
("mail-session" => "default") ]) - failure description: "WFLYCTL0097: Wrong type for ssl. Expected [BOOLEAN] but was STRING"
Same situation, if I try to pass PORT as value in outbound-socket-binding block.
I have no idea how to pass integers/booleans from docker-compose file to Wildfly configuration.
docker-compose.yml (part)
...
services:
some_service:
image: image_name:tag
environment:
- USERNAME=some_username#...
- PASSWORD=some_password
- SSL=true // I also tried with value 1
- HOST=smtp.gmail.com
- PORT=465 // also doesn't work
...
Dockerfile:
FROM some_wildfly_base_image
# install cgroup-bin package
USER root
RUN apt-get update
RUN apt-get install -y cgroup-bin
RUN apt-get install -y bc
USER jboss
ADD standalone-myapp.xml /opt/jboss/wildfly/standalone/configuration/
ADD standalone.conf /opt/jboss/wildfly/bin/
ADD modules/ /opt/jboss/wildfly/modules/
RUN wildfly/bin/add-user.sh usr usr --silent
# Set the default command to run on boot
# This will boot WildFly in the standalone mode and bind to all interface
CMD [ "/opt/jboss/wildfly/bin/standalone.sh", "-c", "standalone-myapp.xml", "-Dmail.username=$USERNAME", "-Dmail.password=$PASSWORD", "-Dmail.ssl=$SSL", "-Drm.host=$HOST", "-Drm.port=$PORT" ]
standalone-myapp.xml:
...
<subsystem xmlns="urn:jboss:domain:mail:2.0">
<mail-session name="default" jndi-name="java:jboss/mail/Default">
<smtp-server password="${mail.password}" username="${mail.username}" ssl="${mail.ssl}" outbound-socket-binding-ref="mail-smtp"/>
</mail-session>
</subsystem>
...
<outbound-socket-binding name="mail-smtp">
<remote-destination host="${rm.host}" port="465"/>
</outbound-socket-binding>
...
Almost there. In your docker file, you have defined environmental variables therefore you need to reference them as environmental variables in your wildfly config. The easiest way is to prefix your env var with env. prefix. So in your example, you have env variables HOST, SSL, USERNAME... which you can reference in standalone.xml like this:
<smtp-server password="${env.PASSWORD}" username="${env.USERNAME}" ssl="${env.SSL}" outbound-socket-binding-ref="mail-smtp"/> </mail-session>
Without env. prefix, jboss/wildfly will try to resolve the expression as jvm property, which you'd have to specify as jvm -D flag.
You can also use default value fallback in your expressions such as:
ssl="${env.SSL:true}"
This way, the ssl will be set the the value of environmental variable named SSL, and if such var does not exist, server will fallback to true.
Happy hacking

Resources