I have monolithic application that i am trying to containerize. The foler structure is like this:
--app
|
|-file.py <-has a variable foo that is passed in
--configs
|
|-variables.py <- contains foo variable
Right now, I have the app in a container and the configs in a container. When I try to start up the app container, it fails because a dependency on the config container variable.
What am i doing wrong? And how should I approach this issue. Should the app and config be in one big container for now?
I was thinking docker-compose could solve this issue. Thoughts?
The variables.py file could be (in) a volume accessed by the app container that you import from the config container with --volumes-from config option to docker run. With Docker Compose you would use the volumes_from directive.
Less recommended way -
Run the Config Container first, it will have its own docker.sock.
You can mount the above Docker Socket in first app Container via -v "/var/run/docker.sock:/var/run/docker.sock", which will let the App container access the Config container, but I think this will need privileged access.
This is similar to Docker in Docker concept.
You can also consider design changes to your application by serving that foo variable over HTTP, which will result in much simpler solution. You can use simple web server and urllib3 module in Python to have a simple solution which will serve the variable via Internal Docker Networking.
Related
TLDR: When using docker compose, I can simply recreate a container by changing its configuration and/or image in the docker-compose.yml file along with running docker-compose up. Is there any generic equivalent for recreating a container (to apply changes) which was created by a bare docker create/run command?
Elaborating a bit:
The associated docker compose documentation states:
If there are existing containers for a service, and the service’s configuration or image was changed after the container’s creation, docker-compose up picks up the changes by stopping and recreating the containers (preserving mounted volumes).
I'm having troubles to understand which underlaying steps are actually performed during this recreation, as e.g. the docker (without compose) documentation doesn't really seem to use the recreate term at all.
Is it safe to simply run docker container rm xy and then docker container create/run (along with passing the full and modified configuration)? Or is docker compose actually doing more under the hood?
I already found answers about applying specific configuration changes like e.g. this one about port mappings, but I'm still wondering whether there is a more general answer to this.
I'm having troubles to understand which underlaying steps are actually performed during this recreation, as e.g. the docker (without compose) documentation doesn't really seem to use the recreate term at all.
docker-compose is a high level tool; it performs in a single operation what would require multiple commands using the docker cli. When docker-compose says, "docker-compose up picks up the changes by stopping and recreating the containers", it means it is doing the equivalent of:
docker stop <somecontainer>
docker rm <somecontainer>
docker run ...
(Where ... represents whatever configuration is implied by the service definition in your docker-compose.yaml).
Let's say it recognizes a change in container1 it does (not really, working via API):
docker compose rm -fs container1
docker compose create (--build) container1
docker compose start container1
What is partially close to (depending on your compose-config):
docker rm -f projectname_container1
(docker build --flags)
docker create --allDozensOfAttributes projectname_container1
docker start projectname_container1
docker network connect (--flags) projectname_networkname projectname_container1
and maybe more..
so i would advise to use the docker compose commands for single services instead of docker cli if suitable..
The issue is that the variables and settings are not exposed through any docker apis. It may be possible by way of connecting directly to the docker socket, parsing the variables, and then stopping/removing the container and recreating it.
This would be prone to all kinds of errors and would require lots of debugging to get these values.
What I do is to simply store my docker commands in a shell script. You can just save the command you need to run into a text file, name it .sh, set the -x on the file, then run it. Then when you stop/delete the container, you can just rerun the shell script.
Another thing you can do would be to replace the docker command with a function (in something like your ~/.bashrc) that stores the arguments to a text file and rechecks that text file with a passed argument (like "recreate" followed by a name). However, I'm more a fan of doing docker containers in their own shell scripts as its more portable.
I need to work with all the files from a docker container, my approach is to copy all the list of files from the container to my host.
I'm using the next docker commands, for example with the postgres image:
docker create -ti --name dummy_1 postgres bash
docker cp dummy_1:/. Documents/docker/dockerOne
With this I have all the container folders and files in my host.
And then the idea is to transverse all the files with the java API, and work with them and finally delete the files and folders from local, but I would like to know if is it a better approach, maybe with Java and access directly to the container files, instead of create a local copy of the container files in my host.
Any ideas?
You can build a small server app inside your docker container which feeds you the information you need at an exposed port. Thats how i would have done it.
Maybe I don't understand the question, but you can mount a volume when you run, not create the container
docker run -v /host/path:/container/path your_container
Any code in the container (e.g. Java) that modifies files at /container/path will be reflected on the host, and not need to be copied back in/out. Similarly, any modifications on the host filesystem will be seen in the container.
I don't think I can implement an API in the docker container
Yes you can. You bind a TCP port using -p flag
I have a Microservices based application and the services work fine if I deploy them on a host machine. But now, I'd like to learn Docker, so I started to use containers on a linux based machine. Here is a sample Docker file, it is really simple:
FROM openjdk:11-jdk-slim
MAINTAINER BeszterceKK
COPY ./tao-elszamolas-config.jar /usr/src/taoelszamolas/tao-elszamolas-config.jar
WORKDIR /usr/src/taoelszamolas
ENV SPRING_PROFILES_ACTIVE prod
EXPOSE 9001
ENTRYPOINT ["java", "-jar", "tao-elszamolas-config.jar", "-Dlog4j.configurationFile=file:/tao-elszamolas/services/tao-config/log4j2- prod.xml", "-DlogFileLocation=/tao-elszamolas/logs"]
My problem is that, I try to write my Spring boot application log to the host machine. This is why I use data volumes. At the end this is the command how I run the container:
docker run -d --name=tao-elszamolas-config-server --publish=9001:9001 -v /tao-elszamolas/logs:/tao-elszamolas/logs -v /tao-elszamolas/services/tao-config/log4j2-prod.xml:/tao-elszamolas/services/tao-config/log4j2-prod.xml tao-elszamolas-config:latest
But on a longer term all of the services will go under "docker-compose". This is just for the test, something like a proof of concept.
First question is, why it is not writing the log to the right place. (In one of the volumes defined.) That is what I set in the Log4j2 config xml. If I use the config XML on local without Docker everything works fine. When I log into the container, then I can see the mounted volumes and I can "cd" into it. And I also can do this:
touch something.txt
So the file will be created and can be seen both from container and host machine. What am I doing wrong? I think, the application can pick up the log config, because when I just set an internal folder as the location of the log file, it logs the stuff inside the container.
And I also set the permissions of the whole volume (and its children) to 777 temporarily to test out if the permissions were the problem. But not. Any help would be very much appreciated!
My second question, is there any good web based tool on linux where I can manage my containers. Start them, stop then, etc... I googled it out and found some but not sure which one is the best and free for basic needs, and which one is enough secure.
UPDATE:
Managed to resolve this problem after spending couple of nights with this.
I had multiple problems. First of all, the order of the system properties in the Dockerfile ENTRYPOINT section wasn't quite right. The
-Dsomething=something
must be before the "-jar". Otherwise it is not working in Docker. I haven't found any official documentation stating that, but this is how it is working for me. So the right ENDPOINT definition looks like this:
ENTRYPOINT ["java", "-DlogFileLocation=/tao-elszamolas/logs", "-jar", "tao-elszamolas-config.jar"]
Secondly, when I mounted some folders up to the container with docker run command like this:
-v /tao-elszamolas/logs:/tao-elszamolas/logs
then the log file wasn't written, if the folder in the Docker container doesn't exist by default. But if I create that folder at some point before the ENTRYPOINT in the Dockerfile, then the logging is fine, the system writes its logs to the host machine. I also didn't find any documentation stating these facts, but this is my experience.
Just to provide some steps for verification:
Both Log4j and spring boot in general, should not be aware of any docker-related things, like volumes, mapped folders and so forth.
Instead, configure the logging of the application as if it works without docker at all, so if you want a local file - make sure that the application indeed produces the logging file in a folder of your choice.
The next step would be mapping the folder with volumes in docker / docker-compose.
But first please validate the first step:
docker ps // to see the container id
docker exec -it <CONTAINER_ID> bash
// now check the logging file from within the docker container itself even without volumes
If the logging file does not exist its a java issue and you should configure logging properly. If not - it's a docker issue.
You have a space in your entrypoint after log4j2- and before prod.xml:
ENTRYPOINT ["java", "-jar", "tao-elszamolas-config.jar", "-Dlog4j.configurationFile=file:/tao-elszamolas/services/tao-config/log4j2- prod.xml", "-DlogFileLocation=/tao-elszamolas/logs"]
It might be a problem.
I am using Docker to deploy my ASP.NET Core Web API microservices, and am looking at the options for injecting configuration into each container. The standard way of using an appsettings.json file in the application root directory is not ideal, because as far as I can see, that means building the file into my docker images, which would then limit which environment the image could run in.
I want to build an image once which can they be provided configuration at runtime and rolled through the dev, test UAT and into Production without creating an image for each environment.
Options seem to be:
Providing config via environment variables. Seems a bit tedious.
Somehow mapping a path in the container to a standard location on the host server where appsettings.json sits, and getting the service to pick this up (how?)
May be possible to provide values on the docker run command line?
Does anyone have experience with this? Could you provide code samples/directions, particularly on option 2) which seems the best at the moment?
It's possible to create data volumes in the docker image/container. And also mount a host directory into a container. The host directory will then by accessible inside the container.
Adding a data volume
You can add a data volume to a container using the -v flag with the docker create and docker run command.
$ docker run -d -P --name web -v /webapp training/webapp python app.py
This will create a new volume inside a container at /webapp.
Mount a host directory as a data volume
In addition to creating a volume using the -v flag you can also mount a directory from your Docker engine’s host into a container.
$ docker run -d -P --name web -v /src/webapp:/webapp training/webapp python app.py
This command mounts the host directory, /src/webapp, into the container at /webapp.
Refer to the Docker Data Volumes
We are using other packaging system for now (not docker itself), but still have same issue - package can be deployed in any environment.
So, the way we are doing it now:
Use External configuration management system to hold and manage configuration per environment
Inject to our package the basic environment variables to hold the configuration management system connection details
This way we are not only allowing the package to run in almost any "known" environment, but also run-time configuration management.
When you are running docker, you can use environment variable options of the run command:
$ docker run -e "deep=purple" ...
I am setting up a brand new development environment, nginx+php-fpm and decided to create application containers (using docker) for each service.
Normally I would install nginx and php and modify the configuration (with and editor like vim), reload the services until the services were correctly configured.
To establish a similar procedure starting the initial container and copying the /etc/nginx to the host. Modify the config files in the host and use a docker file (containing another COPY) to test the changes.
Given that the containers are somewhat ephemeral and aren't really meant to contain utilities like vim I was wondering how people set up the initial configuration ?
Once I have a working config I know the options with regards to configuration management for managing the files. It's really the establishment of new containers that I'm curious about.
You can pass configuration through environment variables or mount a host file as a data volume.
Example for nginx configuration:
docker run --name some-nginx -v /yoursystem/nginx.conf:/etc/nginx/nginx.conf:ro -d nginx
You can use many images from Docker Hub as starting point: nginx-php-fmp.