How to package several services in one docker image? - docker

I have a docker compose application, which works fine in local. I would like to create an image from it and upload it to the docker hub in order to pull it from my azure virtual machine without passing all files. Is this possible? How can I do it?
I tried to upload the image I see from docker desktop and then pull it from the VM but the container does not start up.
Here I attach my .yml file. There is only one service at the moment but in the future there will be multiple microservices, this is why I want to use compose.
version: "3.8"
services:
dbmanagement:
build: ./dbmanagement
container_name: dbmanagement
command: python manage.py runserver 0.0.0.0:8000
volumes:
- ./dbmanagement:/dbmandj
ports:
- "8000:8000"
environment:
- POSTGRES_HOST=*******
- POSTGRES_NAME=*******
- POSTGRES_USER=*******
- POSTGRES_PASSWORD=*******
Thank you for your help

The answer is: yes, you can but you should not
According to the Docker official docs:
It is generally recommended that you separate areas of concern by using one service per container
Also check this:
https://stackoverflow.com/a/68593731/3957754
docker-compose is enough
docker-compose exist just for that: Run several services with one click (minimal configurations) and commonly in the same server.
foreground process
In order to works a docker container needs a foreground process. To understand what is this, check the following links. As a extremely summary we can said you that a foreground process is something that when you launch it using the shell, the shell is taken and you can and you cannot enter more commands. You need to press ctrl + c to kill the process and get back your shell.
https://unix.stackexchange.com/questions/175741/what-is-background-and-foreground-processes-in-jobs
https://linuxconfig.org/understanding-foreground-and-background-linux-processes
The "fat" container
Anyway, if you want to join several services or process in one container (previously an image) you can do it with supervisor.
Supervisor could works a our foreground process. Basically you need to register one or many linux processes and then, supervisor will start them.
how to install supervisor
sudo apt-get install supervisor
source: https://gist.github.com/hezhao/bb0bee800531b89d7be1#file-supervisor_cmd-sh
add single config: /etc/supervisor/conf.d/myapp.conf
[program:myapp]
autostart = true
autorestart = true
command = python /home/pi/myapp.py
environment=SECRET_ID="secret_id",SECRET_KEY="secret_key_avoiding_%_chars"
stdout_logfile = /home/pi/stdout.log
stderr_logfile = /home/pi/stderr.log
startretries = 3
user = pi
source: https://gist.github.com/hezhao/bb0bee800531b89d7be1
start it
sudo supervisorctl start myapp
sudo supervisorctl tail myapp
sudo supervisorctl status
In the previous sample, we are used supervisor to start a python process.
multiple process with supervisor
You just need to add more [program] sections to the config file:
[program:php7.2]
command=/usr/sbin/php-fpm7.2-zts
process_name=%(program_name)s
autostart=true
autorestart=true
[program:dropbox]
process_name=%(program_name)s
command=/app/.dropbox-dist/dropboxd
autostart=true
autorestart=true
Here some examples, just like your requirement: several process in one container:
canvas lms : Basically starts 3 process: postgress, redis and a ruby app
https://github.com/harvard-dce/canvas-docker/blob/master/assets/supervisord.conf
ngnix + php + ssh
https://gist.github.com/pollend/b1f275eb7f00744800742ae7ce403048#file-supervisord-conf
nginx + php
https://gist.github.com/lovdianchel/e306b84437bfc12d7d33246d8b4cbfa6#file-supervisor-conf
mysql + redis + mongo + nginx + php
https://gist.github.com/nguyenthanhtung88/c599bfdad0b9088725ceb653304a91e3
Also you could configure a web dashboard:
https://medium.com/coinmonks/when-you-throw-a-web-crawler-to-a-devops-supervisord-562765606f7b
Another samples with docker + supervisor:
https://gist.github.com/chadrien/7db44f6093682bf8320c
https://gist.github.com/damianospark/6a429099a66bfb2139238b1ce3a05d79

Related

How to avoid service dependencies from being stopped in Docker Compose?

Given the following Docker Compose file....
version: '3.8'
services:
producer:
image: producer
container_name: producer
depends_on: [db]
build:
context: ./producer
dockerfile: ./Dockerfile
db:
image: some-db-image
container_name: db
When I do docker-compose up producer obviously the db service gets started too. When I CTRL+C both services are stopped. This is expected and fine.
But sometimes, the db service is started before, on a different shell and so doing docker-compose up producer understands that db is running and only starts producer. But when I hit CTRL+C, both producer and db are stopped even though db was not started as part of this docker compose up command.
Is there a way to avoid getting the dependencies services stopped when stopping its "parent" ?
When running just docker-compose up, the CTRL+C command always stops all running services in the current compose scope. It doesn't care about depends_on.
You would need to spin it up with detach option -d, like
docker-compose up -d producer
Then you can do
docker stop producer
And db service should still be running.
As I understand your question: You want to stop a container A which depends on another container B. But when stopping A, you don't want docker-compose to stop B.
Docker-compose stops the dependent containers ('B' in this case) when 'A' is stopped.
How I would approach this:
Split up the docker-compose files into A and B
In docker-compose for A create a health check testing (and waiting) for container B to be alive.
Since this is a database, you could do this with a dummy query.
Then you still have dependency, but not the docker-compose connection of stopping dependant containers.
You can't simply do that with CTRL+C.
Your docker-compose file and the services defined in it are treated as a project. You may notice that all containers, networks and volumes are prefixed with the name of the directory where the docker-compose file is located by default. This is the project name. It can be changed via an environment variable or the -p flag of the docker-compose command.
What docker-compose does is it keeps track of all the resources for a given project.
In your case there are two services: db and producer. Whenever you run docker-compose up, both of them start up. They both end up being part of the same project. The same applies when you only start one of the services (e.g. with docker-compose up db). You can later start the other service and it will still be part of the same project.
One more thing to note here: Whenever you run docker-compose without the -d (detached) flag, you get attached to the whole project, meaning whenever you hit CTRL+C, you'll stop all services. It does not matter if the last compose command started only one of the services or if they depend on each other. Attaching to the project and hitting CTRL+C will stop them.
A possible solution to your problem would be the following:
Start up your services via docker-compose up -d (both db and producer will get created). They are now in detached mode. If you still want to check the logs in real time (kinda like attaching), use docker-compose logs -f. Now, however, if you want to stop only one of the services you can simply do docker-compose stop $SVC_NAME (where $SVC_NAME is either db or producer) and this will keep the other one running. This way, whatever happens to your terminal session, your services won't stop, unless you explicitly tell them to.
Is there a way to avoid getting the dependencies services stopped when stopping its "parent" ?
Yes.
Using the new version docker compose instead of docker-compose might solve your problem Reference.
Simple example
Assuming now you are using the new version, your process could be something like this.
docker-compose.yml
version: "3.8"
services:
db:
build: .
producer:
build: .
depends_on: [db]
extra:
build: .
Dockerfile
FROM node:alpine
WORKDIR /app
COPY . .
ENTRYPOINT [ "/bin/sh", "script.sh" ]
script.sh
while :; do sleep 1; done
Suppose db has started before with
$ docker compose up -d db.
Then later,
$ docker compose up -d producer.
Now you can stop only producer with
$ docker compose stop producer.
You can check if db is still running with
$ docker compose ps.
Notice the use of -d flag for detached mode, as pointed out in another answer, so you don't need to kill the process with CTRL+C. Also, using detached flag allows you to check the services that are running with docker compose ps.
A similar issue as yours was reported and fixed a while ago, as you can see here.
I was not able to reproduce the behavior you observe with a complete minimal example. Namely, when running docker compose stop producer, the underlying db is not stopped AFAICT.
Anyway, you may be interested in an alternative command that is a bit more flexible than docker compose up, regarding how to run "one-off commands": docker compose run.
The typical use cases are as follows:
docker compose run db bash → run the db service, replacing the default CMD with bash
docker compose run -d db → run the db service in the background (detach mode)
docker compose run --service-ports producer → run the service producer and its dependencies (unless they were run with docker compose up), enabling the ports mapping.
So for your specific use case, you could run:
docker compose up -d db
docker compose run --service-ports producer

Docker Compose attach one service to stdin and stdout

Something I'm trying to do is create a docker-compose application that has a single service act as a REPL that can interact with the rest of the services. I tried a variety of ways to get only this service attached to stdin and stdout but I haven't found anything elegant that worked. This stackoverflow post's answer said stdin_open: true and tty: true would work and here's what I made with it:
version: '3'
services:
redis:
image: redis
python:
image: python
entrypoint: /bin/sh
stdin_open: true
tty: true
Running docker-compose up still sends a log of both services and docker-compose up -d detaches both of the services. For this example is there an elegant way to get an interactive shell to the python service while only running docker-compose up ... (i.e. not running docker exec, etc)?
You can docker-compose run an alternate command using the image: and other settings in a Docker Compose YAML file. If that service depends_on: other services, it will start them. The one thing to be aware of is that it will not by default publish the declared ports:.
docker-compose run python /bin/sh
(The Docker setup tends to be a little more optimized around long-running network server processes, like the Redis installation here, and less for "commands" that rely on their stdin for input. Consider packaging your application into an image, but generally using host tools for learning a language and day-to-day development. For Python in particular, a virtual environment gives a self-contained playground where you can install packages, as your user account, without interfering with the system Python.)

Putting file into HDFS using docker-compose

Is there a way to put some file, let's say data.json, into HDFS automatically right from Docker-compose/Dockerfile?
When I start namenode and datanode I can enter into containers with
docker exec -it namenode [datanode] bash, and use
hdfs dfs -put data.json hdfs:/ (when safe mode is finished)
and that works, but I need a way to run this automatically. When I try to build containers from Dockerfile and put comands:
FROM bde2020/hadoop-namenode:1.1.0-hadoop2.8-java8
WORKDIR /data
ADD hdfs_writer/data.json /data
# ADD python_script.py /data
CMD ["hdfs dfsadmin -safemode wait && hdfs dfs -put ./data.json hdfs:/"]
# CMD ["python python_script.py"]
Container namenode immediately terminates. I also tried with the python script, that I add to container and run it with CMD.
python_script
import time
import os
os.system("hdfs dfsadmin -safemode wait")
os.system("hdfs dfs -put -f data.json hdfs:/")
while True:
time.sleep(5)
in that case, container is running, but if I check logs and try to list hdfs with hdfs dfs -ls hdfs:/, there is following error
safemode: Call From 662aae005e8b/172.20.0.5 to namenode:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
19/04/18 14:36:36 WARN ipc.Client: Failed to connect to server: namenode/172.20.0.5:8020: try once and fail.
I read recommended link from error log, and to be honest, I am not sure that I understand what should I do.
Any your suggestions or ideas about possible solution is highly valuable for me, as I am new to this field and I don't have much experience.
If you need some more info, I will be happy to provide it.
docker-compose.yml (just part of it)
namenode:
#docker-compose.yml and Dockerfile are in the dame directory
build: .
volumes:
- ./data/namenode:/hadoop/dfs/name
environment:
- CLUSTER_NAME=cluster
env_file:
- ./hadoop.env
ports:
- 50070:50070
datanode:
image: bde2020/hadoop-datanode:1.1.0-hadoop2.8-java8
depends_on:
- namenode
volumes:
- ./data/datanode:/hadoop/dfs/data
env_file:
- ./hadoop.env
hadoop.env
CORE_CONF_fs_defaultFS=hdfs://namenode:8020
CORE_CONF_hadoop_http_staticuser_user=root
CORE_CONF_hadoop_proxyuser_hue_hosts=*
CORE_CONF_hadoop_proxyuser_hue_groups=*
HDFS_CONF_dfs_webhdfs_enabled=true
HDFS_CONF_dfs_permissions_enabled=false
HDFS_CONF_dfs_blocksize=1m
YARN_CONF_yarn_log___aggregation___enable=true
YARN_CONF_yarn_resourcemanager_recovery_enabled=true
YARN_CONF_yarn_resourcemanager_store_class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
YARN_CONF_yarn_resourcemanager_fs_state___store_uri=/rmstate
YARN_CONF_yarn_nodemanager_remote___app___log___dir=/app-logs
YARN_CONF_yarn_log_server_url=http://historyserver:8188/applicationhistory/logs/
YARN_CONF_yarn_timeline___service_enabled=true
YARN_CONF_yarn_timeline___service_generic___application___history_enabled=true
YARN_CONF_yarn_resourcemanager_system___metrics___publisher_enabled=true
YARN_CONF_yarn_resourcemanager_hostname=resourcemanager
YARN_CONF_yarn_timeline___service_hostname=historyserver
YARN_CONF_yarn_resourcemanager_address=resourcemanager:8032
YARN_CONF_yarn_resourcemanager_scheduler_address=resourcemanager:8030
YARN_CONF_yarn_resourcemanager_resource__tracker_address=resourcemanager:8031
You can't write to networked services in a Dockerfile. Imagine running docker build, running your combined application, tearing it down, and running it again. You'll reuse the same built image without re-running the Dockerfile steps; only the content in the image itself is kept. In most cases you need some minor amount of setup to communicate between services (Docker Compose can do this for you) but that is not set up during a build sequence. This is the same answer as "you can't run database migrations from a Dockerfile", but it applies equally to Hadoop.
A container only does one thing. Your sample Dockerfile sets a different CMD that waits for the namenode to be running and sets it up. This happens instead of starting the namenode process. A Docker container runs one main command and one main command only; there is not a way to run a main command and also a side support script of some form. The container you show would probably work, but you'd need to run it as a separate container alongside the namenode container.
You don't need to be "in Docker" to access Docker-hosted services. You can use a Docker Compose ports: directive to make services visible to the host, at which point you can use ordinary clients to interact with them. The docker exec path is the equivalent of "I ssh to my server as root, and then...", which isn't how you normally deal with any service at all.
Your server containers should only run servers. In your example you're both trying to launch an HDFS namenode and also populate the server from the same container; you'd be better off having the namenode container only be the namenode and running the setup job from another container or from the host. (See the standard postgres image's entrypoint script for some idea of the gyrations required otherwise.)
Docker Compose isn't great for one-off jobs. Every time you run docker-compose up it will discover that your setup container isn't running and try to start it again. Other more powerful orchestrators could be a better fit; for example, a Kubernetes Job is a reasonable fit for what you're describing.

Docker compose - run shell and application inside shell

I'm using docker compose for running my application in dev. environment.
version: '3.4'
services:
web:
build:
context: .
target: base
ports:
- "5000:5000"
stdin_open: true
tty: true
volumes:
- ./src:/src
command: node src/main/server/index.js
Composer is starting container and I can see logs output from node application. When I press CTR-C - container is stopped and my application is stopped as well.
I would like to have my application to be stopped when I press CTRL-C instead of whole container.
The same behavior when running an app within Windows CMD or Linux shell. For example, to restart an app - press CTRL-C, repeat startup command (node src/main/server/index.js by pressing top arrow key), and press enter.
I was thinking I could use something like this, but it does not work.
command: bash -c "node src/main/server/index.js
I know I can use command below to achieve expected behavior:
docker-compose up -d (to start in detached mode)
docker-compose exec web bash (run interactive shell)
node src/main/server/index.js (start node manually)
But maybe there is a way to start bash interactive bash and run an application in bash using singe command docker-compose up ?
Docker runs a main process in its containers, as such, stopping the main process will also stop the container.
I will attempt to answer your question, but I don't think that you should work like that in a Dev environment.
Answering your question, you can "trap" the container in a main process, then just bash into the container and perform the app start.
In order to trap the container, just change the docker-compose command to:
command: while true; do sleep 1; done;
To get into an interactive bash in the container:
docker exec -it <CONTAINER-ID> bash
And then you can start or stop the node app.
It seems that the problem you are facing is a container taking a lot to start, you should probably reorder your Dockerfile to prevent it from redownloading all dependencies (or other long process) every time a file changes.
You should place your COPY command after all commands that should persist from across builds, and take advantage of docker's image layering.
If you need a "hot reload" feature, you can research Webpack hot reloading.
You would need to bind your host volume to the container's work directory in order to let webpack properly watch the files and reload the app.

Development workflow for server and client using Docker Compose?

I'm developing a server and its client simultaneously and I'm designing them in Docker containers. I'm using Docker Compose to link them up and it works just fine for production but I can't figure out how to make it work with a development workflow in which I've got a shell running for each one.
My docker-compose-devel.yml:
server:
image: node:0.10
client:
image: node:0.10
links:
- server
I can do docker-compose up client or even docker-compose run client but what I want is a shell running for both server and client so I can make rapid changes to both as I develop iteratively.
I want to be able to do docker-compose run server bash in one window and docker-compose run --no-deps client bash in another window. The problem with this is that no address for the server is added to /etc/hosts on the client because I'm using docker-compose run instead of up.
The only solution I can figure out is to use docker run and give up on Docker Compose for development. Is there a better way?
Here's a solution I came up with that's hackish; please let me know if you can do better.
docker-compose-devel.yml:
server:
image: node:0.10
command: sleep infinity
client:
image: node:0.10
links:
- server
In window 1:
docker-compose --file docker-compose-dev.yml up -d server
docker exec --interactive --tty $(docker-compose --file docker-compose-dev.yml ps -q server) bash
In window 2:
docker-compose --file docker-compose-dev.yml run client bash
I guess your main problem is about restarting the application when there are changes in the code.
Personnaly, I launch my applications in development containers using forever.
forever -w -o log/out.log -e log/err.log app.js
The w option restarts the server when there is a change in the code.
I use a .foreverignore file to exclude the changes on some files:
**/.tmp/**
**/views/**
**/assets/**
**/log/**
If needed, I can also launch a shell in a running container:
docker exec -it my-container-name bash
This way, your two applications could restart independently without the need to launch the commands yourself. And you have the possibility to open a shell to do whatever you want.
Edit: New proposition considering that you need two interactive shells and not simply the possibility to relaunch the apps on code changes.
Having two distinct applications, you could have a docker-compose configuration for each one.
The docker-compose.yml from the "server" app could contain this kind of information (I added different kind of configurations for the example):
server:
image: node:0.10
links:
- db
ports:
- "8080:80"
volumes:
- ./src:/src
db:
image: postgres
environment:
POSTGRES_USER: dev
POSTGRES_PASSWORD: dev
The docker-compose.yml from the "client" app could use external_links to be able to connect to the server.
client:
image: node:0.10
external_links:
- project_server_1:server # Use "docker ps" to know the name of the server's container
ports:
- "80:80"
volumes:
- ./src:/src
Then, use docker-compose run --service-ports service-name bash to launch each configuration with an interactive shell.
Alternatively, the extra-hosts key may also do the trick by calling the server app threw a port exposed on the host machine.
With this solution, each docker-compose.yml file could be commited in the repository of the related app.
First thing to mention, for development environment you want to utilize volumes from docker-compose to mount your app to the container when it's started (at the runtime). Sorry if you're already doing it and I mention this, but it's not clear from your definition of docker-compose.yml
To answer your specific question - start your containers normally, then when doing docker-compose ps, you'll see a name of your container. For example 'web_server' and 'web_client' (where web is the directory of your docker-compose.yml file or name of the project).
When you got name of the container you want to connect to, you can run this command to run bash exactly in the container that's running your server:
docker exec -it web_server bash.
If you want to learn more about setting up development environment for reasonably complex app, checkout this article on development with docker-compose

Resources