Using custom local domain with Docker - docker

I am running Docker using Docker Desktop on Windows.
I would like to set-up a simple server.
I run it using:
$ docker run -di -p 1234:80 yahya/example-server
This works as expected and runs fine on localhost:1234.
However, I want to give it's own local domain name (e.g. api.example.test), which should only be accessible locally.
Normally for a VM setup I would edit the Windows hosts file, get the IP address of the VM (let's say it's 192.168.90.90) and add something like the following:
192.168.90.90 api.example.test
How would I do something similar in Docker.
I know you can enter an ip address for port forwarding, but if I enter any local IP I get the following error:
$ docker run -di -p 192.168.90.90:1234:80 yahya/example-server
docker: Error response from daemon: Ports are not available: exposing port TCP 192.168.90.90:80 -> 0.0.0.0:0: listen tcp 192.168.90.90:80: can't bind on the specified endpoint.
However, it does work for 10.0.0.7 for some reason (I found this IP automatically added in the hosts file after installing Docker Desktop).
$ docker run -di -p 10.0.0.7:1234:80 yahya/example-server
This essentially solves the issue, but would become an issue again if I have more than 1 project.
Is there a way I can use another local IP address (preferably without a nginx proxy)?

I think there is no simple way to do this without some kind of reverse-proxy.
In my dev environment I use Traefik and dnscrypt-proxy to achieve automatic *.test domain names for multiple projects at same time
First, start Traefik proxy on ports 80 and 433, example docker-compose.yml:
---
networks:
traefik:
name: traefik
services:
traefik:
image: traefik:2.8.3
container_name: traefik
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- traefik
ports:
- 80:80
- 443:443
environment:
TRAEFIK_API: 'true'
TRAEFIK_ENTRYPOINTS_http: 'true'
TRAEFIK_ENTRYPOINTS_http_ADDRESS: :80
TRAEFIK_ENTRYPOINTS_https: 'true'
TRAEFIK_ENTRYPOINTS_https_ADDRESS: :443
TRAEFIK_ENTRYPOINTS_https_HTTP_TLS: 'true'
TRAEFIK_GLOBAL_CHECKNEWVERSION: 'false'
TRAEFIK_GLOBAL_SENDANONYMOUSUSAGE: 'false'
TRAEFIK_PROVIDERS_DOCKER: 'true'
TRAEFIK_PROVIDERS_DOCKER_EXPOSEDBYDEFAULT: 'false'
Then, attach your service to traefik network, and set labels for routing (see Traefik & Docker). Example docker-compose.yml:
---
networks:
traefik:
external: true
services:
example:
image: yahya/example-server
restart: always
labels:
traefik.enable: true
traefik.docker.network: traefik
traefik.http.routers.example.rule: Host(`example.test`)
traefik.http.services.example.loadbalancer.server.port: 80
networks:
- traefik
Finally, add to hosts:
127.0.0.1 example.test
Instead of manually adding all future domains to hosts, you can setup local DNS resolver. I prefer to use cloaking feature of dnscrypt-proxy for this.
You can install it using Installation instructions, then uncomment following line in dnscrypt-proxy.toml:
cloaking_rules = 'cloaking-rules.txt'
and add to cloaking-rules.txt:
*.test 127.0.0.1
finally, setup your network connection to use 127.0.0.1 as DNS resolver

Related

Sending API-requests between two docker containers

I have running a DDEV-Environment for Magento2, locally on my Mac OSX (Ventura)
https://ddev.readthedocs.io/en/stable/users/quickstart/#magento-2
For testing purpose I included Nifi per docker-compose.yaml inside my ddev project .ddev/docker-compose.nifi.yaml
Below you can see the docker-compose, which is really minimal at this point. Nifi works like expected, because I can login etc, although it is not persistent yet, but thats a different problem
version: '3'
services:
nifi:
image: apache/nifi:latest
container_name: ddev-${DDEV_SITENAME}-nifi
ports:
# HTTP
- "8080:8080"
# HTTPS
- "8443:8443"
volumes:
# - ./nifi/database_repository:/opt/nifi/nifi-current/database_repository
# - ./nifi/flowfile_repository:/opt/nifi/nifi-current/flowfile_repository
# - ./nifi/content_repository:/opt/nifi/nifi-current/content_repository
# - ./nifi/provenance_repository:/opt/nifi/nifi-current/provenance_repository
# - ./nifi/state:/opt/nifi/nifi-current/state
# - ./nifi/logs:/opt/nifi/nifi-current/logs
# - ./nifi/conf/login-identity-providers.xml:/opt/nifi/nifi-current/conf/login-identity-providers.xml
- ".:/mnt/ddev_config"
All I want to do is sending a POST-requst from Nifi to my Magento2 module.
I tried several IPs now, which I got from docker inspect ddev-ddev-magento2-web but I always receive "Connection refused"
My output from docker network ls:
NETWORK ID NAME DRIVER SCOPE
95bea4031396 bridge bridge local
692b58ca294e ddev-ddev-magento2_default bridge local
46be47991abe ddev_default bridge local
7e19ae1626f1 host host local
f8f4f1aeef04 nifi_docker_default bridge local
dbdba30546d7 nifi_docker_mynetwork bridge local
ca12e667b773 none null local
My Magento2-Module is working properly, because sending requests from Postmanto it works fine
You don't want most of what you have. Please remove the ports statement, which you shouldn't need at all; if you need anything, you'll need an expose. But I doubt you need that in this case?
You'll want to look at the docs:
Additional services and add-ons
Additional services with docker-compose
Then create a .ddev/docker-compose.nifi.yaml with something like
services:
nifi:
image: apache/nifi:latest
container_name: ddev-${DDEV_SITENAME}-nifi
container_name: "ddev-${DDEV_SITENAME}-someservice"
labels:
com.ddev.site-name: ${DDEV_SITENAME}
com.ddev.approot: ${DDEV_APPROOT}
expose:
- "8080"
environment:
- VIRTUAL_HOST=$DDEV_HOSTNAME
- HTTP_EXPOSE=8080:8080
- HTTPS_EXPOSE=9999:8080
volumes:
- ".:/mnt/ddev_config"
The name of the "web" container from inside your nifi container will be "web", curl http://web:8080, assuming that you have nifi on port 8080.
I don't know what you're trying to accomplish, but this may get you started. Feel free to come over to the DDEV Discord channel for more interactive help.

Setting Traefik to intercept traffic between Docker containers

I have Traefik container (traefik v. 2.8) running as reverse proxy for local development.
I also use docker-compose.yml file to define my services.
Also, I have exposed the services present in docker-compose.yml via host.docker.internal by setting these lines in my /etc/hosts file:
127.0.0.1 host.docker.internal
localhost host.docker.internal
My setup is as such:
host.docker.internal:8443 used for service_a
host.docker.internal:8453 used for service_b
I have setup Traefik route from a.localhost that goes to host.docker.internal:8443.
I can access a.localhost from the host outside the containers just fine, and Traefik does really route the traffic to host.docker.internal:8443 as I want.
Problem is that I have a reason to have service B (host.docker.internal:8453) call service A via the a.localhost hostname.
This does not work, as in service B, I get unknown host when trying to access a.localhost
Here is extract from my docker-compose.yml file:
version: '3'
services:
reverse_proxy:
image: traefik:v2.8
# Enables the web UI and tells Traefik to listen to docker
command: --api.insecure=true --providers.docker
ports:
- "80:80"
- "443:443"
# The Web UI (enabled by --api.insecure=true)
- "9000:8080"
volumes:
# So that Traefik can listen to the Docker events
- /var/run/docker.sock:/var/run/docker.sock
- ./dev-traefik/traefik.yml:/etc/traefik/traefik.yml
- ./dev-traefik:/configurations
service_a:
ports:
- "8443:8443"
service_b:
ports:
- "8453:8453"
Also I'm using a yml-based configuration for Traefik, present in dynamic-config.yml:
http:
routers:
service-a-router:
service: service-a
rule: "Host(`a.localhost`)"
tls: "true" # using tls
services:
service-a:
loadBalancer:
servers:
- url: "https://host.docker.internal:8443" # service A
tls:
certificates:
- certFile: "/etc/https/tls.crt"
keyFile: "/etc/https/tls.key"
stores:
default:
defaultCertificate:
certFile: "/etc/https/tls.crt"
keyFile: "/etc/https/tls.key"
It seems like Traefik is able to listen to requests made from host network, as accessing https://a.localhost from browser outside the container network works just fine.
On the other hand, requests made by service_a container don't seem to be caught by Traefik.
What I have also tried is to add a.localhost to /etc/hosts in the host machine running the containers like this:
127.0.0.1 a.localhost
localhost a.localhost
And then using curl inside service B container to access service A.
This resulted in getting connection refused as opposed to Could not resolve host: a.localhost. This leads me to suggest that traefik couldn't intercept traffic from service b container
What am I doing wrong?
Is there a way to make such setup work? I do have a legit reason for it, which relates to having as close setup as possible in local development as on other environments which are deployed to cloud.
I wasn't able to have Traefik intercept container -> container traffic the way I originally specified I'd want, but was able to actually nevertheless get similar setup working.
Here's the scenario:
I have two services that are accessed via HTTP & TLS:
service_a
service_b
I want to be able to use host.docker.internal special DNS name to my advantage and actually have Traefik to proxy traffic from https://host.docker.internal/service_a to service_a port 8443 both outside Docker container network (from the host machine running Docker) AND from service_b via the fact that both the host machine running Docker can access host.docker.internal and also hosts inside the Docker network.
Using this fact to my advantage, I just defined a path for service_a as such in Traefik's YML configuration file:
http:
routers:
service-a-router:
service: service-a
rule: "Host(`host.docker.internal`) && PathPrefix(`/service_a`)"
tls: "true" # using tls
services:
service-a:
loadBalancer:
servers:
- url: "https://host.docker.internal:8443" # service A
tls:
certificates:
- certFile: "/etc/https/tls.crt"
keyFile: "/etc/https/tls.key"
stores:
default:
defaultCertificate:
certFile: "/etc/https/tls.crt"
keyFile: "/etc/https/tls.key"
And docker-compose.yml was made to look like this:
version: '3'
services:
reverse_proxy:
image: traefik:v2.8
# Enables the web UI and tells Traefik to listen to docker
command: --api.insecure=true --providers.docker
ports:
- "80:80"
- "443:443"
# The Web UI (enabled by --api.insecure=true)
- "9000:8080"
volumes:
# So that Traefik can listen to the Docker events
- /var/run/docker.sock:/var/run/docker.sock
- ./dev-traefik/traefik.yml:/etc/traefik/traefik.yml
- ./dev-traefik:/configurations
service_a:
ports:
- "8443:8443"
service_b:
ports:
- "8453:8453"
Now as I use host.docker.internal in Host rule in my Traefik YML config, Traefik is in fact able to intercept both traffic from the Docker host machine and from the Docker container service_b.
service_b just need to configure URL of https://host.docker.internal/service_a to access service_a through Traefik.

How to set bind9 docker container as dns of other container

I'm trying to set up ssl for my home network and I've set up a bind9 container with a custom domain that points to my unraid server. So far so good. I've also set up a private step-ca certificate authority which needs it's dns set to the bind9 container so that it knows about my private domain. The setup works if I set the dns of the step container to the internal docker ip address of the bind container but since these ip addresses are ephemeral I can't rely on that hence why I'm binding the bind9 ip address to something within 192.168.0.1/24 and accessing it there. This works if I set the dns server of my pc to the bind9 container but I am unable to do so for other docker containers for some reason
in short, step-ca and my proxy traefik need their dns set to bind9 which I want to have set up with a static ip address on the 192.168.0.1/24 subnet. traefik also needs to be able to talk to containers on the bridge network br0 otherwise it won't be able to proxy requests to the containers
The addresses of your containers don't need to be ephemeral. We can set up a custom network using the networks top-level element that defines a static range for the network using the ipam option, and then we can assign our containers static address on this network.
We can use the dns option to configure containers to use the bind9 container for name resolution.
Here's an example docker-compose.yaml that sets up a bind9 container and a couple of additional containers that will use it for DNS:
version: "3"
services:
bind9:
image: docker.io/internetsystemsconsortium/bind9:9.19
volumes:
- "./bind:/etc/bind"
- bind9_cache:/var/cache/bind
- bind9_log:/var/log
- bind9_lib:/var/lib/bind
networks:
bind9:
ipv4_address: 192.168.133.10
web1:
image: docker.io/alpinelinux/darkhttpd:latest
networks:
bind9:
ipv4_address: 192.168.133.20
dns: 192.168.133.10
web2:
image: docker.io/alpinelinux/darkhttpd:latest
networks:
bind9:
ipv4_address: 192.168.133.21
dns: 192.168.133.10
networks:
bind9:
ipam:
driver: default
config:
- subnet: 192.168.133.0/24
gateway: 192.168.133.1
volumes:
bind9_cache:
bind9_lib:
bind9_log:
In in the bind directory, I have bind configured to serve the following zonefile:
$TTL 604800
# IN SOA docker.example. root.docker.example. (
2 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
# IN NS ns.docker.example.
ns IN A 192.168.133.10
web1 IN A 192.168.133.20
web2 IN A 192.168.133.21
web IN A 192.168.133.20
web IN A 192.168.133.21
From either the web1 or web2 containers, we can confirm that they are using our bind instance for name resolution:
/ $ wget -O- web1.docker.example:8080
Connecting to web1.docker.example:8080 (192.168.133.20:8080)
writing to stdout
<html>
.
.
.
</html>
Recall that docker-compose is just a fancy wrapper for docker run, so you can accomplish the same thing without using docker-compose (although it will of course make life much easier).
If you need to access the bind9 container, you would of course just publish the appropriate ports on your host by adding the necessary ports section to the compose configuration (or by using the --publish/-p option on the docker run command line):
bind9:
image: docker.io/internetsystemsconsortium/bind9:9.19
ports:
- "53:53/udp"
- "53:53/tcp"
volumes:
- "./bind:/etc/bind"
- bind9_cache:/var/cache/bind
- bind9_log:/var/log
- bind9_lib:/var/lib/bind
networks:
bind9:
ipv4_address: 192.168.133.10

Traefik basic configuration for running in a Docker Swarm

From what I can see it goes like this:
docker-traefik.yml:
version: '3'
services:
traefik:
image: traefik
command: --docker # enable Docker Provider
# use Docker Swarm Mode as data provider
--docker.swarmmode
ports:
- "80:80"
volumes:
# for it to be able to listen to Docker events
- /var/run/docker.sock:/var/run/docker.sock
docker-whoami.yml:
version: '3'
networks:
traefik_default:
external: true
services:
whoami:
image: containous/whoami
networks:
# add to traefik network
- traefik_default
deploy:
labels:
# whoami is on port 80
- "traefik.port=80"
# whoami is on traefik_default network
- "traefik.docker.network=traefik_default"
# when to forward requests to whoami
- "traefik.frontend.rule=Host:example.com"
Let me quote the documentation here:
Required labels:
traefik.frontend.rule
traefik.port - Without this the debug logs will show this service is deliberately filtered out.
traefik.docker.network - Without this a 504 may occur.
...
traefik.docker.network Overrides the default docker network to use for connections to the container. [1]
traefik.port=80 Registers this port. Useful when the container exposes multiples ports.
But why can't it just take the exposed ports for a default value of traefik.port? And from what I can see it works without traefik.docker.network (that is, if traefik_default is the first service's network). When do I get 504's?
But why can't it just take the exposed ports for a default value of traefik.port?
If ur container has 3 or 4 exposed ports, which should traefik use? So who saying to traefik, which of these ports the right one? So you do - with traefik.port. Where is the problem to use the default port of your configured service?
U should expose 80, 443 and 8080 - so 80 and 443 for http/https webpages and 8080 for traefik dashboard. If u dont wanna use the dashboard, u dont need to expose 8080.
And i dont see any network configured # traefik in your composer file - should this have no network? Ur service and traefik need to be in the same network. Otherwise traefik cant reach ur service and forward.
Also where are the endpoints?

Docker container internal vs external dns resolution issue using Traefik

Docker 18.06.1-ce, traefik 1.7.3, dnsmasq, Mac 10.14
I have docker-compose setup with Traefik and need to access several services from inside the docker network/containers and externally.
On a linux box (with Let'sEncrypt and http redirected to https), everything works using the same FQDN for both docker container internal and external access, and I don't have to use the service names.
When I run curl http://belapi.dev.biodati.test from inside the pipeline container using docker-compose exec belapi /bin/bash I get the following error (and I don't see it showing up in the Traefik access logs):
api#407cf7105aee:/app$ curl http://belapi.dev.biodati.test/status
curl: (7) Failed to connect to belapi.dev.biodati.test port 80: Connection refused
This works fine (using the servicename):
curl http://belapi:8000/status
I can also run the following fine from a bash shell on my Mac outside the docker containers (and I see it hitting the Traefik access logs):
curl http://belapi.dev.biodati.test/status
I have dnsmasq setup to forward all *.test domains to 127.0.0.1.
From inside the pipeline container:
dig belapi.dev.biodati.test
;; QUESTION SECTION:
;belapi.dev.biodati.test. IN A
;; ANSWER SECTION:
belapi.dev.biodati.test. 7 IN A 127.0.0.1
My docker-compose file:
networks:
biodati:
external: true
services:
pipeline:
container_name: pipeline
image: biodati/bel_pipeline:dev
networks:
biodati:
traefik:
image: traefik:1.7
container_name: traefik
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./conf/traefik.toml:/traefik.toml
- ./logs:/logs
networks:
biodati:
labels:
- traefik.enable=true
- traefik.backend=traefik
- traefik.frontend.rule=Host:traefik.${BS_HOST_NAME:?err}
- traefik.port=8080
- traefik.docker.network=biodati
# BEL API - core requirement
belapi:
container_name: belapi
image: belbio/bel_api:localdev
networks:
biodati:
labels:
- traefik.enable=true
- traefik.backend=belapi
- traefik.frontend.rule=Host:belapi.${BS_HOST_NAME:?err};
- traefik.port=8000
- traefik.docker.network=biodati
For full details on how to solve this: https://medium.com/#williamhayes/local-dev-on-docker-fun-with-dns-85ca7d701f0a
Basically - DNSMasq was working great, Mac Docker Desktop DNS mapping was working great. I could query for my service domain name (e.g. service1.test) dig service1.test1 and get back 127.0.0.1 which is exactly what I set up in DNSMasq. So my domain name was returning the correct IP address for my host. Except - I was getting this inside my container - so 127.0.0.1 was referring to my container environment.
Running the following command on the Mac host level in a terminal:
sudo ifconfig lo0 alias 10.254.254.254
added an alias for 127.0.0.1 that I could use in DNSMasq instead of 127.0.0.1 that would still map to my localhost but it would also work for routing from my docker containers.
Now I can use local domains on my Mac for local development in Docker and get to my containers from my host AND via inter-container requests.

Resources