I'm attempting to create a makefile that will launch my db container, wait for it to complete before launching the rest of my app.
I have 2 compose files.
docker-compose.db.yml
docker-compose.yml
My make file is as follows:
default:
#echo "Preparing database"
docker-compose -f docker-compose.db.yml build
docker-compose -f docker-compose.db.yml pull
docker-compose -f docker-compose.db.yml up -d
#echo ""
#echo "Waiting for database \"ready for connections\""
#while [ -z "$(shell docker logs $(PROJECT_NAME)_mariadb 2>&1 | grep -o "ready for connections")" ]; \
do \
sleep 5; \
done
#echo "Database Ready for connections!"
#echo ""
#echo "Launching App Containers"
docker-compose build
docker-compose pull
docker-compose up -d
What happens is that it immediately goes to "Database Ready for connections!" even before the database is ready. If I run the same command in terminal it response with empty for about the first 20 seconds and then finally returns "ready for connections".
Thank you in advance
The GNU make $(shell ...) function gets run once when the Makefile is processed. So when your rule has
#while [ -z "$(shell docker logs $(PROJECT_NAME)_mariadb 2>&1 | grep -o "ready for connections")" ]
Make first runs the docker logs command on its own, then substitutes the result in the shell command it runs
while [ -z "ready for connections" ]
which is trivially false, and the loop exits immediately.
Instead you probably want to escape the $ in the shell substitution command
#while [ -z "$$(docker-compose logs mariadb ...) "]
It's fairly typical to configure containers to be able to wait for the database startup themselves, and to run the application and database from the same docker-compose.yml file. Docker Compose wait for container X before starting Y describes this setup.
Related
As per documentation Docker volumes are advertised this way:
Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. While bind mounts are dependent on the directory structure and OS of the host machine, volumes are completely managed by Docker.
But if they are so good, why there are no operations to manage them like: copy, rename?
the command:
docker volume --help
gives only these options:
Usage: docker volume COMMAND
Manage volumes
Commands:
create Create a volume
inspect Display detailed information on one or more volumes
ls List volumes
prune Remove all unused local volumes
rm Remove one or more volumes
Documentation also states no other commands, nor any workarounds for having the copy or rename functionality.
I would like to rename currently existing volume and create another (blank) in place of the originally named volume and populate it with the new data for test.
After doing my test I may want (or not) to remove the newly created volume and rename the other one to its previous (original) name to restore the volume setup as it was before.
I would like to not create a backup of the original volume that I want to rename. Renaming is good enough for me and much faster than creating the backup and restoring form it.
Editing the docker-compose file and changing the name of the volume there is something I would like to avoid as well.
Is there any workaround that can work for renaming of a volume?
Can low level manual management from the shell targeting the Docker Root Dir: /var/lib/docker and volumes sub-dir be a solution or that approach may lead to some docker demon data inconsistency?
Not really the answer but I'll post this copy example because I couldn't find any before and searching for it took me to this question.
Docker suggest --volumes-from for backup purposes here.
For offline migration (stopped container) I don't see the point in using --volumes-from. So I just used a middle container with both volumes mounted and a copy command.
To finish off the migration a new container can use the new volume
Here's a quick test
Prepare a volume prova
docker run --name myname -d -v prova:/usr/share/nginx/html nginx:latest
docker exec myname touch /usr/share/nginx/html/added_file
docker stop myname
Verify the volume has nginx data + our file added_file
sudo ls /var/lib/docker/volumes/prova/_data
Output:
50x.html added_file index.html
Migrate the data to volume prova2
docker run --rm \
-v prova:/original \
-v prova2:/migration \
ubuntu:latest \
bash -c "cp -R /original/* /migration/"
Verify the new volume has the same data
sudo ls /var/lib/docker/volumes/prova2/_data
Output:
50x.html added_file index.html
Run a new container with the migrated volume:
docker run --name copyname -d -v prova2:/user/share/nginx/html nginx:latest
Verify the new container sees the migrated data at the original volume moint point:
docker exec copyname ls -al /user/share/nginx/html
For next searchers, I made a script that can do a copy of volume by #Lennonry example. Here it is https://github.com/KOYU-Tech/docker-volume-copy
Script itself for history:
#!/bin/bash
if (( $# < 2 )); then
echo ""
echo "No arguments provided"
echo "Use command example:"
echo "./dcv.sh OLD_VOLUME_NAME NEW_VOLUME_NAME"
echo ""
exit 1
fi
OLD_VOLUME_NAME="$1"
NEW_VOLUME_NAME="$2"
echo "== From '$OLD_VOLUME_NAME' to '$NEW_VOLUME_NAME' =="
function isVolumeExists {
local isOldExists=$(docker volume inspect "$1" 2>/dev/null | grep '"Name":')
local isOldExists=${isOldExists#*'"Name": "'}
local isOldExists=${isOldExists%'",'}
local isOldExists=${isOldExists##*( )}
if [[ "$isOldExists" == "$1" ]]; then
return 1
else
return 0
fi
}
# check if old volume exists
isVolumeExists ${OLD_VOLUME_NAME}
if [[ "$?" -eq 0 ]]; then
echo "Volume $OLD_VOLUME_NAME doesn't exist"
exit 2
fi
# check if new volume exists
isVolumeExists ${NEW_VOLUME_NAME}
if [[ "$?" -eq 0 ]]; then
echo "creating '$NEW_VOLUME_NAME' ..."
docker volume create ${NEW_VOLUME_NAME} 2>/dev/null 1>/dev/null
isVolumeExists ${NEW_VOLUME_NAME}
if [[ "$?" -eq 0 ]]; then
echo "Cannot create new volume"
exit 3
else
echo "OK"
fi
fi
# most important part, data migration
docker run --rm --volume ${OLD_VOLUME_NAME}:/source --volume ${NEW_VOLUME_NAME}:/destination ubuntu:latest bash -c "echo 'copying volume ...'; cp -R /source/* /destination/"
if [[ "$?" -eq 0 ]]; then
echo "Done successfuly 🎉"
else
echo "Some error occured ðŸ˜"
fi
I have a legacy Docker application I'm working with that uses multiple Celery workers. There is a long running process I need to track. I'm able to write data to a file that is visible from the CLI interface of the worker thread:
I'm writing to the file like this:
def log(msg):
now = datetime.now()
dt_string = now.strftime("%Y-%m-%d %H:%M:%S")
fu.mkdirs(defs.LRP_LOG_DIR)
fu.append_string_to_file(dt_string + ": " + msg + "\n", defs.LRP_LOG_FILE)
def append_string_to_file(string, file_path):
with open(file_path, "a") as text_file:
text_file.write(string)
LRP_LOG_DIR = "/opt/project/backend"
LRP_LOG_FILE = LRP_LOG_DIR + "/lrp-log.txt"
The question is: If I add multiple Celery workers will they each write to their own file (not the desired behaviory) or will they all write to a common /opt/project/backend/lrp-log.txt file (the desired behavior)?
If they don't write to a common file, what do I need to do to get multiple Celery workers to write to the same file?
Also, it would be nice if this file was available on the host file system (I'm running on a Windows machine).
I ended up writing a couple of .sh scripts for Cygwin (I'm on windows). I would like to get the tail to work in the same script but this is good enough for now.
Script to start Docker and write to log file
echo
echo
echo
# STOP CONTAINERS
echo "Stopping all Containers..."
docker kill $(docker ps -q)
# DELETE CONTAINERS
echo "Deleting Containers..."
docker rm $(docker ps -aq)
echo
# PRUNE VOLUMES
echo "Pruning orphaned volumes"
docker volume prune -f
echo
# CREATE LOG DIR
mkdir ./logs
# DELETE OLD FULL LOG FILE
echo "Deleting old full log file..."
touch ./logs/full-log.txt
rm ./logs/full-log.txt
touch ./logs/full-log.txt
# SET UP LRP LOG FILE
echo "Deleting old lrp log file..."
touch ./logs/lrp-log.txt
rm ./logs/lrp-log.txt
# TAIL THE LOG FILE (display the running process in a cygwin window)
cygstart tail -f ./logs/full-log.txt
cygstart tail -f ./logs/lrp-log.txt
# START AES
echo "Starting anonlink entity service (aes)..."
echo "Process is running and writing log to ./full-log.txt"
echo "Long Running Process Log (LRP) is being written to lrp-log.txt"
echo "! ! ! DO NOT CLOSE THIS WINDOW ! ! !"
echo "(<ctrl-c> to quit the process)"
docker-compose -p anonlink -f ../tools/docker-compose.yml up --remove-orphans > ./logs/full-log.txt
echo
echo
echo "Done."
echo
echo
Script to create truncated log file to track long running processes
tail -f ./logs/full-log.txt | grep --line-buffered "LOG_FILE:" > ./logs/lrp-log.txt
I have sh code (DashBoardImport.sh) like down below. It checks apı response to import a kibana dashboard in a infinite loop, If it gets a reponse with success, it breaks the loop :
#!/bin/sh
# use while loop to check if kibana is running
while true
do
response=$(curl -X POST elk:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=#/etc/elasticsearch/CityCountDashBoard.ndjson | grep -oE "^\{\"success")
#curl -X GET elk:9200/git-demo-topic | grep -oE "^\{\"git" > /dev/null
#match=$?
echo $response
if [ '{"success' = $response ]
then
echo "Running import dashboard.."
#curl -X POST elk:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=#/etc/elasticsearch/CityCountDashBoard.ndjson
break
else
echo "Kibana is not running yet"
sleep 5
fi
done
I run DashBoardImport.sh via docker file:
ADD ./CityCountDashBoard.ndjson /etc/elasticsearch/CityCountDashBoard.ndjson
ADD ./DashBoardImport.sh /etc/elasticsearch/DashBoardImport.sh
#ENTRYPOINT /etc/elasticsearch/DashBoardImport.sh &
USER root
RUN chmod +x /etc/elasticsearch/DashBoardImport.sh
#RUN /etc/elasticsearch/DashBoardImport.sh &
RUN nohup bash -c "/etc/elasticsearch/DashBoardImport.sh" >/dev/null 2>&1 &
I tried many options as you can see commented out. The sh works perfectly when I run it manually on the Docker Container. I kill the kibana service. then run the code. after I started the kibana, code succesfully workes as expected and imports the dashboard. But It does not work when it start on container automatically.
Do you have any idea?
Thanks alot in advance :)
A RUN step executes in a temporary container until the command returns and then docker captures the changes to the filesystem as a new layer in your image. Nothing else remains, no environment variables, running processes, etc, only the filesystem changes.
So when you RUN nohup ... & that process immediately returns since it's in the background (nohup ... & explicitly does that), and so the container exits, killing any processes that were running in the container, and captures the filesystem changes made, if any, to your image.
If you want something to run when you start the container, add it to your ENTRYPOINT or CMD.
I am trying to import a pipeline into streamsets, during container start up, by using the Docker CMD command in Dockerfile. The image builds, but while creating the container there is no error but it exits with code 0. So it never comes up. Here is what I did:
Dockerfile:
FROM streamsets/datacollector:3.18.1
COPY myPipeline.json /pipelinejsonlocation/
EXPOSE 18630
ENTRYPOINT ["/bin/sh"]
CMD ["/opt/streamsets-datacollector-3.18.1/bin/streamsets","cli","-U", "http://localhost:18630", \
"-u", \
"admin", \
"-p", \
"admin", \
"store", \
"import", \
"-n", \
"myPipeline", \
"--stack", \
"-f", \
"/pipelinejsonlocation/myPipeline.json"]
Build image:
docker build -t cmp/sdc .
Run image:
docker run -p 18630:18630 -d --name sdc cmp/sdc
This outputs the container id. But the container is in the Exited status as shown below.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
537adb1b05ab cmp/sdc "/bin/sh /opt/stream…" 5 seconds ago Exited (0) 3 seconds ago sdc
When I do not specify the CMD command in the Dockerfile, the streamsets container spins up and then when I run the streamsets import command in the running container in shell, it works. But how do I get it done during provisioning itself? Is there something I am missing in the Dockerfile?
In your Dockerfile you overwrite the default CMD and ENTRYPOINT from the StreamSets Data Collector Dockerfile. So the container only executes your command during startup and exits without errors afterwards. This is the reason why your container is in Exited (0) status.
In general this is good and expected behavior. If you want to keep your container alive you need to execute another command in the foreground, which never ends. But unfortunately, you cannot run multiple CMDs in your docker file.
I dug a little deeper. The default entry point of the image is ENTRYPOINT ["/docker-entrypoint.sh"]. This script sets up a few things and starts the Data Collector.
It is required that the Data Collector is running before the pipeline is imported. So a solution could be to copy the default docker-entrypoint.sh and modify it to start the Data Collector and import the pipeline afterwards. You could to it like this:
Dockerfile:
FROM streamsets/datacollector:3.18.1
COPY myPipeline.json /pipelinejsonlocation/
# Replace docker-entrypoint.sh
COPY docker-entrypoint.sh /docker-entrypoint.sh
EXPOSE 18630
docker-entrypoint.sh (https://github.com/streamsets/datacollector-docker/blob/master/docker-entrypoint.sh):
#!/bin/bash
#
# Copyright 2017 StreamSets Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
set -e
# We translate environment variables to sdc.properties and rewrite them.
set_conf() {
if [ $# -ne 2 ]; then
echo "set_conf requires two arguments: <key> <value>"
exit 1
fi
if [ -z "$SDC_CONF" ]; then
echo "SDC_CONF is not set."
exit 1
fi
grep -q "^$1" ${SDC_CONF}/sdc.properties && sed 's|^#\?\('"$1"'=\).*|\1'"$2"'|' -i ${SDC_CONF}/sdc.properties || echo -e "\n$1=$2" >> ${SDC_CONF}/sdc.properties
}
# support arbitrary user IDs
# ref: https://docs.openshift.com/container-platform/3.3/creating_images/guidelines.html#openshift-container-platform-specific-guidelines
if ! whoami &> /dev/null; then
if [ -w /etc/passwd ]; then
echo "${SDC_USER:-sdc}:x:$(id -u):0:${SDC_USER:-sdc} user:${HOME}:/sbin/nologin" >> /etc/passwd
fi
fi
# In some environments such as Marathon $HOST and $PORT0 can be used to
# determine the correct external URL to reach SDC.
if [ ! -z "$HOST" ] && [ ! -z "$PORT0" ] && [ -z "$SDC_CONF_SDC_BASE_HTTP_URL" ]; then
export SDC_CONF_SDC_BASE_HTTP_URL="http://${HOST}:${PORT0}"
fi
for e in $(env); do
key=${e%=*}
value=${e#*=}
if [[ $key == SDC_CONF_* ]]; then
lowercase=$(echo $key | tr '[:upper:]' '[:lower:]')
key=$(echo ${lowercase#*sdc_conf_} | sed 's|_|.|g')
set_conf $key $value
fi
done
# MODIFICATIONS:
#exec "${SDC_DIST}/bin/streamsets" "$#"
check_data_collector_status () {
watch -n 1 ${SDC_DIST}/bin/streamsets cli -U http://localhost:18630 ping | grep -q 'version' && echo "Data Collector has started!" && import_pipeline
}
function import_pipeline () {
sleep 1
echo "Start to import pipeline"
${SDC_DIST}/bin/streamsets cli -U http://localhost:18630 -u admin -p admin store import -n myPipeline --stack -f /pipelinejsonlocation/myPipeline.json
echo "Finished importing pipeline"
}
# Start checking if Data Collector is up (in background) and start Data Collector
check_data_collector_status & ${SDC_DIST}/bin/streamsets $#
I commented out the last line exec "${SDC_DIST}/bin/streamsets" "$#" of the default docker-entrypoint.sh and added two functions. check_data_collector_status () pings the Data Collector service until it is available. import_pipeline () imports your pipeline.
check_data_collector_status () runs in background and ${SDC_DIST}/bin/streamsets $# is started in foreground as before. So the pipeline is imported after the Data Collector service is started.
Run this image with sleep command:
docker run -p 18630:18630 -d --name sdc cmp/sdc sleep 300
300 is the time to sleep in seconds.
Then exec your script manually within the docker container and find out what's wrong.
We currently use Elasticsearch for storage of Spring Boot App logs that are sent by Filebeat and use Kibana to visualise this.
Our entire architecture is dockerized inside a docker-compose file. Currently, our when we start the stack, we have to wait for Elasticsearch to start, then PUT our Ingest Pipeline, then restart Filebeat, and only then do our logs show up properly ingested in Kibana.
I'm quite new to this, but I was wondering if there is no way to have Elasticsearch save ingest pipelines so that you do not have to load them every single time? I read about mounting volumes or running custom scripts to wait for ES and PUT when ready, but all of this seems very cumbersome for a use case that to me seems like the default?
We used a similar approach to ozlevka, by running a script during the build process of our custom Elasticsearch image.
This is our script:
#!/bin/bash
# This script sets up the Elasticsearch docker instance with the correct pipelines and templates
baseUrl='localhost:9200'
contentType='Content-Type:application/json'
# filebeat
ingestUrl=$baseUrl'/_ingest/pipeline/our-pipeline?pretty'
payload='/usr/share/elasticsearch/config/our-pipeline.json'
/usr/share/elasticsearch/bin/elasticsearch -p /tmp/pid > /dev/null &
# wait until Elasticsearch is up
# you can get logs if you change /dev/null to /dev/stderr
while [[ "$(curl -s -o /dev/null -w ''%{http_code}'' -XPUT $ingestUrl -H$contentType -d#$payload)" != "200" ]]; do
echo "Waiting for Elasticsearch to start and posting pipeline..."
sleep 5
done
kill -SIGTERM $(cat /tmp/pid)
rm /tmp/pid
echo -e "\n\n\nCompleted Elasticsearch Setup, refer to logs for details"
I suggest using the startscript into filebeat container.
The script will ping elasticsearch be ready, after this create pipeline and start filebeat.
#!/usr/bin/env bash -e
START_FILE=/tmp/.es_start_file
http () {
local path="${1}"
curl -XGET -s -k --fail http://${ELASTICSEARCH_HOST}:{$ELASTICSEARCH_PORT}${path}
}
pipeline() {
curl -XPUT -s -k --fail http://${ELASTICSEARCH_HOST}:{$ELASTICSEARCH_PORT}/_ingest/pipeline/$PIPELINE_NAME -d #pipeline.json
}
while true; do
if [ -f "${START_FILE}" ]; then
pipeline
/usr/bin/filebeat -c filebeat.yaml &
exit 0
else
echo 'Waiting for elasticsearch cluster to become green'
if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
touch ${START_FILE}
fi
fi
done
This method will be good for docker-compose and docker swarm. For k8s preferable create readiness probe.