I'm trying to figure out best practices for Docker Compose, which I'm pretty new to.
Scenario:
I have one container with a Flask app, served through gunicorn (which binds on port 8080).
I have an nginx container that acts as a reverse proxy for TLS termination, request routing to other containers, etc. It needs to be able to forward requests to gunicorn/Flask in the first container.
I'm using Docker Compose to do a quick deployment of these containers. Since the gunicorn/Flask container should only accept traffic from nginx, and not any direct requests, I'd like to configure it such that the gunicorn/Flask port (8080 in the container) is only exposed to the nginx container, and not to anything else on the host.
Is this possible?
Just don't publish ports: for the thing you don't want accessible from outside Docker space.
version: '3.8'
services:
backend:
build: .
# no ports:
proxy:
image: nginx
volumes: ['./nginx.conf:/etc/nginx/nginx.conf']
ports:
- '80:80'
Your Nginx configuration can still proxy_pass http://backend:8080, but the Flask application won't be directly reachable from outside of Docker.
ports: specifically allows a connection from outside Docker into the container. It's not a required option. Connections between containers don't require ports:, and if ports: remap one of the application ports, that remapping is ignored.
Technically this setup only allows connections from within the same Docker network, so if you had other services in this setup they could also reach the Flask container. On a native-Linux host there are tricks to directly contact the container, but they don't work on other host OSes or from another system. IME usually these limitations are acceptable.
Yes, this is one of the main use cases for using docker-compose. You basically only need to expose the port of nginx in your docker-compose.yaml and let the flask app not expose any ports to the outside world of the container.
From the docker-compose docs: By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.
For example, imagine your flask app service inside your docker-compose.yml is called my-flask-app where the app inside the container is running on port 8080. Then you can access the endpoint from the within nginx container by the servicename such as http://my-flask-app:8080. You can try this by using curl from inside the flask container.
Related
I am new to docker and have trouble setting up the network between the containers to not allow unnecessary connections from outside.
I have a Docker running on a VPS with three containers on a remote IP 123.xxx.xxx.xxx
container name published ports IP adress
sqldb 3306:3306 172.xxx.xxx.4
applet1 80:3306 172.xxx.xxx.5
applet2 4444:4444 172.xxx.xxx.3
One is database and two are java apps. The trouble I am having right now is that when I create the containers the ports on the containers become exposed to the global internet so my database sqldb is exposed by 123.xxx.xxx.xxx:3306
Right now ny java apps are connect through JDBC like so jdbc:mysql://172.xxx.xxx.4:3306/db.
I am trying to accomplish the following:
port 80 on host so 123.xxx.xxx.xxx connects to java app applet1.
The goal is to give applet1 the ability to connect to sqldb and also applet2 but I don't wan't unecessary ports to be exposed to the whole internet. Preferably that internal URIs would be left but connections from outside (apart from SSH on port 22 and TCP on port 80) would be forbidden for ports 4444, 3306. Also, I don't yet know how to use docker-compose so if possible how can I solve it without it?
*I have heard you can connect to containers by writing container names like that: have not had success with it yet jdbc:mysql://sqldb/db.
If all your containers are running on the same docker bridge network, you don't need to expose any ports for them to communicate with each other.
Docker Compose is a particularly good tool for organising several containers like this as it automatically configures a network for you
# docker-compose.yaml
version: '3.9'
services:
sqldb:
image: sqldb
applet1:
image: applet1
ports:
- '80:3306' # you sure about this container port?
depends_on:
- sqldb
applet2:
image: applet2
depends_on:
- sqldb
Now only your applet1 container will have a host port mapping. Both applets will be able to connect to any other service within the network on their container ports.
Do I have to expose the port if I am using the ports config?
In the docker-compose.yml below, do I have to keep expose 2022 or can I remove it? Is there a difference between them?
myproject-app:
build: ../myproject-app
container_name: myproject-app
image: myproject-app
expose:
- 2022
ports:
- 2022:2022
volumes:
- ../myproject-app/:/home/myproject/myproject-app/
- /home/myproject/myproject-app/node_modules
Exposed ports needed for another service, if you want to connect two services inside docker, so service with exposed port will be available inside docker net.
Ports property (ports:) make service port available on your host machine, so you can connect it with your OS net.
So you can delete expose property.
"Expose" means basically nothing in modern Docker. There is pretty much no reason to put an expose: line in your docker-compose.yml file. It's considered polite to include an EXPOSE line in your Dockerfile to document what port(s) your service will listen on, but it's not strictly necessary.
In modern Docker, with named networks, any container can connect to any port of any other container on the same network, provided a process is listening there. Before there were named networks, one container had to explicitly "link" to another to be able to call it, and then it could only reach the exposed ports of the target container. This setup is considered obsolete now (you never need links: either).
Plain Docker has an option (docker run -P, with a capital P) to publish all exposed ports on random host ports. Compose doesn't have an equivalent option. Ports that are exposed but not published also show up in the docker ps output. But those are really the only things "expose" does at all.
What is the difference between ports and expose options in docker-compose.yml?
According to the docker-compose reference,
Ports is defined as:
Expose ports. Either specify both ports (HOST:CONTAINER), or just the container port (a random host port will be chosen).
Ports mentioned in docker-compose.yml will be shared among different services started by the docker-compose.
Ports will be exposed to the host machine to a random port or a given port.
My docker-compose.yml looks like:
mysql:
image: mysql:5.7
ports:
- "3306"
If I do docker-compose ps, it will look like:
Name Command State Ports
-------------------------------------------------------------------------------------
mysql_1 docker-entrypoint.sh mysqld Up 0.0.0.0:32769->3306/tcp
Expose is defined as:
Expose ports without publishing them to the host machine - they’ll only be accessible to linked services. Only the internal port can be specified.
Ports are not exposed to host machines, only exposed to other services.
mysql:
image: mysql:5.7
expose:
- "3306"
If I do docker-compose ps, it will look like:
Name Command State Ports
---------------------------------------------------------------
mysql_1 docker-entrypoint.sh mysqld Up 3306/tcp
Edit
In recent versions of Dockerfile, EXPOSE doesn't have any operational impact anymore, it is just informative. (see also)
ports:
Activates the container to listen for specified port(s) from the world outside of the docker(can be same host machine or a different machine) AND also accessible world inside docker.
More than one port can be specified (that's is why ports not port)
expose:
Activates container to listen for a specific port only from the world inside of docker AND not accessible world outside of the docker.
More than one port can be specified
Ports
This section is used to define the mapping between the host server and Docker container.
ports:
- 10005:80
It means the application running inside the container is exposed at port 80. But external system/entity cannot access it, so it need to be mapped to host server port.
Note: you have to open the host port 10005 and modify firewall rules to allow external entities to access the application.
They can use
http://{host IP}:10005
something like this
EXPOSE
This is exclusively used to define the port on which application is running inside the docker container.
You can define it in dockerfile as well. Generally, it is good and widely used practice to define EXPOSE inside dockerfile because very rarely anyone run them on other port than default 80 port
Ports
The ports section will publish ports on the host. Docker will set up a forward for a specific port from the host network into the container. By default, this is implemented with a userspace proxy process (docker-proxy) that listens on the first port, and forwards into the container, which needs to listen on the second point. If the container is not listening on the destination port, you will still see something listening on the host, but get a connection refused if you try to connect to that host port, from the failed forward into your container.
Note, the container must be listening on all network interfaces since this proxy is not running within the container's network namespace and cannot reach 127.0.0.1 inside the container. The IPv4 method for that is to configure your application to listen on 0.0.0.0.
Also note that published ports do not work in the opposite direction. You cannot connect to a service on the host from the container by publishing a port. Instead you'll find docker errors trying to listen to the already-in-use host port.
Expose
Expose is documentation. It sets metadata on the image, and when running, on the container too. Typically, you configure this in the Dockerfile with the EXPOSE instruction, and it serves as documentation for the users running your image, for them to know on which ports by default your application will be listening. When configured with a compose file, this metadata is only set on the container. You can see the exposed ports when you run a docker inspect on the image or container.
There are a few tools that rely on exposed ports. In docker, the -P flag will publish all exposed ports onto ephemeral ports on the host. There are also various reverse proxies that will default to using an exposed port when sending traffic to your application if you do not explicitly set the container port.
Other than those external tools, expose has no impact at all on the networking between containers. You only need a common docker network, and connecting to the container port, to access one container from another. If that network is user created (e.g. not the default bridge network named bridge), you can use DNS to connect to the other containers.
I totally agree with the answers before.
I just like to mention that the difference between expose and ports is part of the security concept in docker. It goes hand in hand with the networking of docker.
For example:
Imagine an application with a web front-end and a database back-end.
The outside world needs access to the web front-end (perhaps on port
80), but only the back-end itself needs access to the database host
and port. Using a user-defined bridge, only the web port needs to be
opened, and the database application doesn’t need any ports open,
since the web front-end can reach it over the user-defined bridge.
This is a common use case when setting up a network architecture in docker.
So for example in a default bridge network, not ports are accessible from the outer world.
Therefor you can open an ingresspoint with "ports". With using "expose" you define communication within the network. If you want to expose the default ports you don't need to define "expose" in your docker-compose file.
I have a typical web stack that consists of nginx + django + database components.
I have set them up in different docker containers with docker-compose and it's running fine.
services:
billing_app_dev:
image: jameslin/billing_app:latest
expose:
- 8000
billing_postgres:
image: postgres:10.5
restart: always
volumes:
- pg_data:/var/lib/postgresql/data
environment:
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
billing_nginx:
image: jameslin/billing_nginx:${TAG}
volumes:
- app_files:/static
links:
- 'billing_app'
ports:
- 80:80
Now I am wondering how I can set up DEV and QA environments on a single machine. I can change the django and database containers to listen to different ports, but looks like I cannot run nginx containers individually since port 80 can only be listened by one container.
I will have to share the nginx container for those 2 environments which doesn't seem very clean.
Are there any better ideas if running 2 VMs is not possible?
I have 3 apache containers and 1 nginx running in the same server, so pretty sure is not a issue.
For each stack of webserver + database i have a different docker-compose file, in this way docker will create a different network for each stack, avoiding possible problems with simultaneous port, and you only will have to bind your nginx in different ports of your server, because, you only can bind one service to one port. still, each container is a separated "machine", so even over the same network they can use the same port.
if you really need run all your services in the port 80 and 443 of your server, may be you will need to put a nginx running in those ports like a reverse proxy calling in the internal docker network those services, is a option but i never try it before over docker internal network.
I think what you needed is virtual ip or maybe called ip aliasing. Even you just have one network card, you can still set 2 ips on it.
Then, you can set 2 different nginx container on host, and use different ip but same 80 port.
Something like follows:
cd /etc/sysconfig/network-script/
cp ifcfg-eth0 ifcfg-eth0:1
vi ifcfg-eth0:1
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth0:1 ----> sub network card
HWADDR=00:0C:29:45:62:3B
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.109.108 ----> configure a new different ip
NETMASK=255.255.255.0
Detail refers to Create Multiple IP Addresses to One Single Network Interface
For nginx, from nginx guide, you had to change your nginx docker to modify listen 80 to listen your_ip:80, then it will not listen on all ip address.
What is the difference between ports and expose options in docker-compose.yml?
According to the docker-compose reference,
Ports is defined as:
Expose ports. Either specify both ports (HOST:CONTAINER), or just the container port (a random host port will be chosen).
Ports mentioned in docker-compose.yml will be shared among different services started by the docker-compose.
Ports will be exposed to the host machine to a random port or a given port.
My docker-compose.yml looks like:
mysql:
image: mysql:5.7
ports:
- "3306"
If I do docker-compose ps, it will look like:
Name Command State Ports
-------------------------------------------------------------------------------------
mysql_1 docker-entrypoint.sh mysqld Up 0.0.0.0:32769->3306/tcp
Expose is defined as:
Expose ports without publishing them to the host machine - they’ll only be accessible to linked services. Only the internal port can be specified.
Ports are not exposed to host machines, only exposed to other services.
mysql:
image: mysql:5.7
expose:
- "3306"
If I do docker-compose ps, it will look like:
Name Command State Ports
---------------------------------------------------------------
mysql_1 docker-entrypoint.sh mysqld Up 3306/tcp
Edit
In recent versions of Dockerfile, EXPOSE doesn't have any operational impact anymore, it is just informative. (see also)
ports:
Activates the container to listen for specified port(s) from the world outside of the docker(can be same host machine or a different machine) AND also accessible world inside docker.
More than one port can be specified (that's is why ports not port)
expose:
Activates container to listen for a specific port only from the world inside of docker AND not accessible world outside of the docker.
More than one port can be specified
Ports
This section is used to define the mapping between the host server and Docker container.
ports:
- 10005:80
It means the application running inside the container is exposed at port 80. But external system/entity cannot access it, so it need to be mapped to host server port.
Note: you have to open the host port 10005 and modify firewall rules to allow external entities to access the application.
They can use
http://{host IP}:10005
something like this
EXPOSE
This is exclusively used to define the port on which application is running inside the docker container.
You can define it in dockerfile as well. Generally, it is good and widely used practice to define EXPOSE inside dockerfile because very rarely anyone run them on other port than default 80 port
Ports
The ports section will publish ports on the host. Docker will set up a forward for a specific port from the host network into the container. By default, this is implemented with a userspace proxy process (docker-proxy) that listens on the first port, and forwards into the container, which needs to listen on the second point. If the container is not listening on the destination port, you will still see something listening on the host, but get a connection refused if you try to connect to that host port, from the failed forward into your container.
Note, the container must be listening on all network interfaces since this proxy is not running within the container's network namespace and cannot reach 127.0.0.1 inside the container. The IPv4 method for that is to configure your application to listen on 0.0.0.0.
Also note that published ports do not work in the opposite direction. You cannot connect to a service on the host from the container by publishing a port. Instead you'll find docker errors trying to listen to the already-in-use host port.
Expose
Expose is documentation. It sets metadata on the image, and when running, on the container too. Typically, you configure this in the Dockerfile with the EXPOSE instruction, and it serves as documentation for the users running your image, for them to know on which ports by default your application will be listening. When configured with a compose file, this metadata is only set on the container. You can see the exposed ports when you run a docker inspect on the image or container.
There are a few tools that rely on exposed ports. In docker, the -P flag will publish all exposed ports onto ephemeral ports on the host. There are also various reverse proxies that will default to using an exposed port when sending traffic to your application if you do not explicitly set the container port.
Other than those external tools, expose has no impact at all on the networking between containers. You only need a common docker network, and connecting to the container port, to access one container from another. If that network is user created (e.g. not the default bridge network named bridge), you can use DNS to connect to the other containers.
I totally agree with the answers before.
I just like to mention that the difference between expose and ports is part of the security concept in docker. It goes hand in hand with the networking of docker.
For example:
Imagine an application with a web front-end and a database back-end.
The outside world needs access to the web front-end (perhaps on port
80), but only the back-end itself needs access to the database host
and port. Using a user-defined bridge, only the web port needs to be
opened, and the database application doesn’t need any ports open,
since the web front-end can reach it over the user-defined bridge.
This is a common use case when setting up a network architecture in docker.
So for example in a default bridge network, not ports are accessible from the outer world.
Therefor you can open an ingresspoint with "ports". With using "expose" you define communication within the network. If you want to expose the default ports you don't need to define "expose" in your docker-compose file.