I have been trying to reproduce (in other machine in the same network) the video played by this container on the docker-compose via swarm.
services:
vlc:
image: boydachina/vlc-server
ports:
- 8080:8080
- 8554:8554
networks:
- vlc_net
command:
- cvlc -vvv /opt/vlc-media/python.mp4 --sout '#transcode{vcodec=h264,acodec=mpga,ab=128,channels=2,samplerate=44100}:rtp{sdp=rtsp://:8554/}'
volumes:
- ./media:/opt/vlc-media/
networks:
vlc_net:
But it is as if there was no network from the container of the other machine to my machine. I thought that putting it in bridge mode would solve it, but I saw that you can't put the Docker Swarm in bridge mode. I need to play the video on several machines on the network, does anyone have any solutions?
Before you deploy the stack to the swarm, create a Docker Network with the overlay driver (note that network names must be unique):
docker network create --driver overlay vlc_net
This will create an overlay network that spans the entire swarm.
Then try setting the network options like this:
networks:
vlc_net:
driver: overlay
external: true
It might also help you to look at how Traefik manages its network in a docker swarm and try to replicate it, since all containers in a swarm can connect to Traefik, and that seems like the use case you are trying to solve.
Related
The goal: To deploy on docker swarm a set of services, one of which is only available for me when I am connected to the OpenVPN server which has also been spun up on docker swarm.
How can I, step by step, only connect to a whoami example container, with a domain in the browser, when I am connected to a VPN?
Background
The general idea would be have, say, kibana and elasticsearch running internally which can only be accessed when on the VPN (rather like a corporate network), with other services running perfectly fine publicly as normal. These will all be on separate nodes, so I am using an overlay network.
I do indeed have OpenVPN running on docker swarm along with a whoami container, and I can connect to the VPN, however it doesn't look like the IP is changing and I have no idea how to make it so that the whoami container is only available when on the VPN, especially considering I'm using an overlay network which is multi-host. I'm also using traefik, a reverse proxy which provides me with a mostly automatic letsencrypt setup (via DNS challenge) for wildcard domains. With this I can get:
https://traefik.mydomain.com
But I also want to connect to vpn.mydomain.com (which I can do right now), and then be able to visit:
https://whoami.mydomain.com
...which I cannot. Yet. I've posted my traefik configuration in a different place in case you want to take a look, as this thread will grow too big if I post it here.
Let's start with where I am right now.
OpenVPN
Firstly, the interesting thing about OpenVPN and docker swarm is that OpenVPN needs to run in privileged mode because it has to make network interfaces changes amongst other things, and swarm doesn't have CAP_ADD capabilities yet. So the idea is to launch the container via a sort of 'proxy container' that will run the container manually with these privileges added for you. It's a workaround for now, but it means you can deploy the service with swarm.
Here's my docker-compose for OpenVPN:
vpn-udp:
image: ixdotai/swarm-launcher:latest
hostname: mainnode
environment:
LAUNCH_IMAGE: ixdotai/openvpn:latest
LAUNCH_PULL: 'true'
LAUNCH_EXT_NETWORKS: 'app-net'
LAUNCH_PROJECT_NAME: 'vpn'
LAUNCH_SERVICE_NAME: 'vpn-udp'
LAUNCH_CAP_ADD: 'NET_ADMIN'
LAUNCH_PRIVILEGED: 'true'
LAUNCH_ENVIRONMENTS: 'OVPN_NATDEVICE=eth1'
LAUNCH_VOLUMES: '/etc/openvpn:/etc/openvpn:rw'
volumes:
- '/var/run/docker.sock:/var/run/docker.sock:rw'
networks:
- my-net
deploy:
placement:
constraints:
- node.hostname==mainnode
I can deploy the above with: docker stack deploy --with-registry-auth --compose-file docker/docker-compose.prod.yml my-app-name and this is what I'm using for the rest. Importantly I cannot just deploy this as it won't load yet. OpenVPN configuration needs to exist in /etc/openvpn on the node, which is then mounted in the container, and I do this during provisioning:
// Note that you have to create the overlay network with --attachable for standalone containers
docker network create -d overlay app-net --attachable
// Create the config
docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn ovpn_genconfig -u udp://vpn.mydomain.com:1194 -b
// Generate all the vpn files, setup etc
docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn bash -c 'yes yes | EASYRSA_REQ_CN=vpn.mydomain.com ovpn_initpki nopass'
// Setup a client config and grab the .ovpn file used for connecting
docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn easyrsa build-client-full client nopass
docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn ovpn_getclient client > client.ovpn
So now, I have an attachable overlay network, and when I deploy this, OpenVPN is up and running on the first node. I can grab a copy of client.ovpn and connect to the VPN. Even if I check "send all traffic through the VPN" though, it looks like the IP isn't being changed, and I'm still nowhere near hiding a container behind it.
Whoami
This simple container can be deployed with the following in docker-compose:
whoami:
image: "containous/whoami"
hostname: mainnode
networks:
- ${DOCKER_NETWORK_NAME}
ports:
- 1337:80
deploy:
placement:
constraints:
- node.hostname==mainnode
I put port 1337 there for testing, as I can visit my IP:1337 and see it, but this doesn't achieve my goal of having whoami.mydomain.com only resolving when connected to OpenVPN.
I can ping a 192.168 address when connected to the vpn
I ran the following on the host node:
ip -4 address add 192.168.146.16/24 dev eth0
Then when connected to the VPN, I can resolve this address! So it looks like something is working at least.
How can I achieve the goal stated at the top? What is required? What OpenVPN configuration needs to exist, what network configuration, and what container configuration? Do I need a custom DNS solution as I suggest below? What better alternatives are there?
Some considerations:
I can have the domains, including the private one whoami.mydomain.com public. This means I would have https and get wildcard certificates for them easily, I suppose? But my confusion here is - how can I get those domains only on the VPN but also have tls certs for them without using a self-signed certificate?
I can also run my own DNS server for resolving. I have tried this but I just couldn't get it working, probably because the VPN part isn't working properly yet. I found dnsmasq for this and I had to add the aforementioned local ip to resolve.conf to get anything working locally for this. But domains would still not resolve when connected to the VPN, so it doesn't look like DNS traffic was going over the VPN either (even though I set it as such - my client is viscosity.
Some mention using a bridge network, but a bridge network does not work for multi-host
Resources thus far (I will update with more)
- Using swarm-launcher to deploy OpenVPN
- A completely non-explanatory answer on stackexchange which I have seen referenced as basically unhelpful by multiple people across other Github threads, and one of the links is dead
So I was banging my head head against a brick wall about this problem and just sort of "solved" it by pivoting your idea:
Basically I opened the port of the vpn container to its host. And then enable a proxy. This means that I can reach that proxy by visiting the ip of the pc in which the vpn resides (AKA the Docker Host of the VPN container/stack).
Hang with me:
I used gluetun vpn but I think this applies also if you use openvpn one. I just find gluetun easier.
Also IMPORTANT NOTE: I tried this in a localhost environment, but theoretically this should work also in a multi-host situation since I'm working with separated stacks. Probably, in a multi-host situation you need to use the public ip of the main docker host.
1. Create the network
So, first of all you create an attachable network for this docker swarm stacks:
docker network create --driver overlay --attachable --scope swarm vpn-proxy
By the way, I'm starting to think that this passage is superfluous but need to test it more.
2. Set the vpn stack
Then you create your vpn stack file, lets call it stack-vpn.yml:
(here I used gluetun through swarm-launcher "trick". This gluetun service connects through a VPN via Wireguard. And it also enables an http proxy at the port 8888 - this port is also mapped to its host by setting LAUNCH_PORTS: '8888:8888/tcp')
version: '3.7'
services:
vpn_launcher:
image: registry.gitlab.com/ix.ai/swarm-launcher
volumes:
- '/var/run/docker.sock:/var/run/docker.sock:rw'
networks:
- vpn-proxy
environment:
LAUNCH_IMAGE: qmcgaw/gluetun
LAUNCH_PULL: 'true'
LAUNCH_EXT_NETWORKS: 'vpn-proxy'
LAUNCH_PROJECT_NAME: 'vpn'
LAUNCH_SERVICE_NAME: 'vpn-gluetun'
LAUNCH_CAP_ADD: 'NET_ADMIN'
LAUNCH_ENVIRONMENTS: 'VPNSP=<your-vpn-service> VPN_TYPE=wireguard WIREGUARD_PRIVATE_KEY=<your-private-key> WIREGUARD_PRESHARED_KEY=<your-preshared-key> WIREGUARD_ADDRESS=<addrs> HTTPPROXY=on HTTPPROXY_LOG=on'
LAUNCH_PORTS: '8888:8888/tcp'
deploy:
placement:
constraints: [ node.role == manager ]
restart_policy:
condition: on-failure
networks:
vpn-proxy:
external: true
Notice that either the swarm-launcher and the gluetun containers are using the network previously created vpn-proxy.
3. Set the workers stack
For the time being we will set an example with 3 replicas of alpine image here (filename stack-workers.yml):
version: '3.7'
services:
alpine:
image: alpine
networks:
- vpn-proxy
command: 'ping 8.8.8.8'
deploy:
replicas: 3
networks:
vpn-proxy:
external: true
They also use the vpn-proxy overlay network.
4. Launch our stacks
docker stack deploy -c stack-vpn.yml vpn
docker stack deploy -c stack-workers workers
Once they are up you can access any worker task and try to use the proxy by using the host ip where the proxy resides.
As I said before, theoretically this should work on a multi-host situation, but probably you need to use the public ip of the main docker host (although if they share the same overlay network it could also work with the internal ip address (192...) ).
I have multiple stacks running in docker swarm with traefik, where services in each stack are connected to an overlay network (traefik-net) so traefik can talk to them.
If I have a service in each stack that's called the same service name (service1), and then have another service (service2) in either stack try to access it by the service name (ping http://service1), it'll sometimes hit service1 in the other stack, and sometimes hit service1 in the same stack.
docker network create --driver overlay traefik-net
stack1:
services:
service1:
networks:
- default
- traefik-net
service2:
networks:
- default
- traefik-net
networks:
traefik-net:
external: true
stack2:
services:
service1:
networks:
- default
- traefik-net
networks:
traefik-net:
external: true
I want service2 to only hit service1 that is in the same stack.
I assumed that a service could only hit a service in another stack by prefixing the stack name to the service name (ping http//stack2_service1). But I learned that because of the traefik-net overlay network, they apparently can call each other without the stack name prefix.
Is there a way to turn off service communication across stacks without stack name prefixes?
Or maybe there's a traefik specific solution to the problem?
If anyone has run into this problem I would a very much appreciate a solution.
Yes there is a solution to what you want to achieve you just need to make proper use of overlay networks.
By default all the services that are connected in the same overlay network can talk/resolve each other.
So let's visualize your current implementation. Now you have one network the traefik-net and you have connected there all your services so your design looks like that:
What you need to do in order to isolate services on different stacks but keep them accessible by traefik is to create a different overlay network in each stack file and connect traefik service to these networks by defining them as external in traefik stack file. You are going to end up like this:
In this implementation all the traffic between different stacks is only possible via traefik service and not directly.
I found a similar question: Docker DNS with Multiple Projects Using the Same Network
There doesn't seem to be any solution other than to always hit stack_service instead of service.
For testing our escrow build, I'm attempting to set up a docker network that's isolated from the host and from the outside world.
I've got the following docker-compose.yml (inspired by this forum post):
version: '3'
services:
redis:
image: "redis:2.8.23"
networks:
- isolated
# ... more services (TODO)
networks:
isolated:
driver: overlay
internal: true
When I run docker-compose up -d; it creates the network, but then fails to create the containers, reporting the following:
ERROR: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
But I'm not using docker swarm, nor do I want to.
If I remove the services: stanza from the file, it brings up the network without the error. It warns that the network is unused (obviously).
If I remove the services/redis/networks stanza, it brings up the container correctly.
What am I doing wrong?
I found this answer, which uses driver: bridge for the network, but that still allows access to the host.
Docker version 18.09.3, build 774a1f4
docker-compose version 1.21.2, build a133471
You have specified the network driver to be overlay.
The overlay network driver very much depends on swarm mode and can effectively be considered to be a swarm mode component.
Instead, you should choose a driver that is a local scope driver rather than a swarm scope driver.
The driver you should use is the bridge driver. The bridge network driver is not part of swarm mode and does not depend on swarm mode being active to utilize it.
Since you are using docker-compose, you can just leave the specific driver out entirely, and it will choose the appropriate driver for you. I would recommend removing the driver: overlay line out completely and leaving the rest of the file as-is:
version: '3'
services:
redis:
image: "redis:2.8.23"
networks:
- isolated
# ... more services (TODO)
networks:
isolated:
internal: true
I learned from docker documentation that I can not use docker DNS to find containers using their hostnames without utilizing user-defined bridge network. I created one using the command:
docker network create --driver=overlay --subnet=172.22.0.0/16 --gateway=172.22.0.1 user_defined_overlay
and tried to deploy a container that uses it. compose file looks like:
version: "3.0"
services:
web1:
image: "test"
ports:
- "12023:22"
hostname: "mytest-web1"
networks:
- test
web2:
image: "test"
ports:
- "12024:22"
hostname: "mytest-web2"
networks:
- test
networks:
test:
external:
name: user_defined_overlay
my docker version is: Docker version 17.06.2-ce, build cec0b72
and I got the following error when I tried deploying the stack:
network "user_defined_bridge" is declared as external, but it is not in the right scope: "local" instead of "swarm"
I was able to create an overlay network and define it in compose file. that worked fine but it didn't for bridge.
result of docker network ls:
NETWORK ID NAME DRIVER SCOPE
cd6c1e05fca1 bridge bridge local
f0df22fb157a docker_gwbridge bridge local
786416ba8d7f host host local
cuhjxyi98x15 ingress overlay swarm
531b858419ba none null local
15f7e38081eb user_defined_overlay overlay swarm
UPDATE
I tried creating two containers running on two different swarm nodes(1st container runs on manager while second runs on worker node) and I specified the user-defined overlay network as shown in stack above. I tried pinging mytest-web2 container from within mytest-web1 container using hostname but I got unknown host mytest-web2
As of 17.06, you can create node local networks with a swarm scope. Do so with the --scope=swarm option, e.g.:
docker network create --scope=swarm --driver=bridge \
--subnet=172.22.0.0/16 --gateway=172.22.0.1 user_defined_bridge
Then you can use this network with services and stacks defined in swarm mode. For more details, you can see PR #32981.
Edit: you appear to have significantly overcomplicated your problem. As long as everything is being done in a single compose file, there's no need to define the network as external. There is a requirement to use an overlay network if you want to communicate container-to-container. DNS discovery is included on bridge and overlay networks with the exception of the default "bridge" network that docker creates. With a compose file, you would never use this network without explicitly configuring it as an external network with that name. So to get container to container networking to work, you can let docker-compose or docker stack deploy create the network for your project/stack automatically with:
version: "3.0"
services:
web1:
image: "test"
ports:
- "12023:22"
web2:
image: "test"
ports:
- "12024:22"
Note that I have also removed the "hostname" setting. It's not needed for DNS resolution. You can communicate directly with a service VIP with the name "web1" or "web2" from either of these containers.
With docker-compose it will create a default bridge network. Swarm mode will create an overlay network. These defaults are ideal to allow DNS discovery and container-to-container communication in each of the scenarios.
The overlay network is the network to be used in swarm. Swarm is meant to be used to manage containers on multiple hosts and overlay networks are docker's multi-host networks https://docs.docker.com/engine/userguide/networking/get-started-overlay/
I tried to setup an nginx-proxy container to access my other containers via subdomains on port 80 instead of special ports. As you can guess, I could not get it to work.
I'm kind of new to docker itself and found that it's more comfortable for me to write docker-compose.yml files so I don't have to constantly write long docker run ... commands. I thought there's no difference in how you start the containers, either with docker or docker-compose. However, one difference I noticed is that starting the container with docker does not create any new networks, but with docker-compose there will be a xxx_default network afterwards.
I read that containers on different networks cannot access each other and maybe that might be the reason why the nginx-proxy is not forwarding the requests to the other containers. However, I was unable to find a way to configure my docker-compose.yml file to not create any new networks, but instead join the default bridge network like docker run does.
I tried the following, but it resulted in an error saying that I cannot join system networks like this:
networks:
default:
external:
name: bridge
I also tried network_mode: bridge, but that didn't seem to make any difference.
How do I have to write the docker-compose.yml file to not create a new network, or is that not possible at all?
Bonus question: Are there any other differences between docker and docker-compose that I should know of?
Adding network_mode: bridge to each service in your docker-compose.yml will stop compose from creating a network.
If any service is not configured with this bridge (or host), a network will be created.
Tested and confirmed with:
version: "2.1"
services:
app:
image: ubuntu:latest
network_mode: bridge