Are the files in the cli for Docker celery worker the same, if not what's a good way to create a common file for the threads to write to? - docker

Question

I have a legacy Docker application I'm working with that uses multiple Celery workers. There is a long running process I need to track. I'm able to write data to a file that is visible from the CLI interface of the worker thread:
I'm writing to the file like this:
def log(msg):
now = datetime.now()
dt_string = now.strftime("%Y-%m-%d %H:%M:%S")
fu.mkdirs(defs.LRP_LOG_DIR)
fu.append_string_to_file(dt_string + ": " + msg + "\n", defs.LRP_LOG_FILE)
def append_string_to_file(string, file_path):
with open(file_path, "a") as text_file:
text_file.write(string)
LRP_LOG_DIR = "/opt/project/backend"
LRP_LOG_FILE = LRP_LOG_DIR + "/lrp-log.txt"
The question is: If I add multiple Celery workers will they each write to their own file (not the desired behaviory) or will they all write to a common /opt/project/backend/lrp-log.txt file (the desired behavior)?
If they don't write to a common file, what do I need to do to get multiple Celery workers to write to the same file?
Also, it would be nice if this file was available on the host file system (I'm running on a Windows machine).

Answer 1

I ended up writing a couple of .sh scripts for Cygwin (I'm on windows). I would like to get the tail to work in the same script but this is good enough for now.
Script to start Docker and write to log file
echo
echo
echo
# STOP CONTAINERS
echo "Stopping all Containers..."
docker kill $(docker ps -q)
# DELETE CONTAINERS
echo "Deleting Containers..."
docker rm $(docker ps -aq)
echo
# PRUNE VOLUMES
echo "Pruning orphaned volumes"
docker volume prune -f
echo
# CREATE LOG DIR
mkdir ./logs
# DELETE OLD FULL LOG FILE
echo "Deleting old full log file..."
touch ./logs/full-log.txt
rm ./logs/full-log.txt
touch ./logs/full-log.txt
# SET UP LRP LOG FILE
echo "Deleting old lrp log file..."
touch ./logs/lrp-log.txt
rm ./logs/lrp-log.txt
# TAIL THE LOG FILE (display the running process in a cygwin window)
cygstart tail -f ./logs/full-log.txt
cygstart tail -f ./logs/lrp-log.txt
# START AES
echo "Starting anonlink entity service (aes)..."
echo "Process is running and writing log to ./full-log.txt"
echo "Long Running Process Log (LRP) is being written to lrp-log.txt"
echo "! ! ! DO NOT CLOSE THIS WINDOW ! ! !"
echo "(<ctrl-c> to quit the process)"
docker-compose -p anonlink -f ../tools/docker-compose.yml up --remove-orphans > ./logs/full-log.txt
echo
echo
echo "Done."
echo
echo
Script to create truncated log file to track long running processes
tail -f ./logs/full-log.txt | grep --line-buffered "LOG_FILE:" > ./logs/lrp-log.txt