How should I setup Traefik on ECS? - docker

 In Short
I've managed to run Traefik locally and on AWS ECS but now I'm wondering how should I setup some sort of load balancing to make my two services with random IPs available to the public.
My current setup on ECS
[Internet]
|
[Load balancer on port 443 + ALB Security group on 443]
|
[Target group on port 443 + Security group from *any* port]
|
[cluster]
|
[service1 container ports "0:5000"]
While this works, I'd now like to add another container, eg. service2 also with random ports eg 0:8000. And that's why I need something like Traefik.
What I did
Here's Toml file:
[api]
address = ":8080"
[ecs]
clusters = ["my-cluster"]
watch = true
domain = "mydomain.com"
region = "eu-central-1"
accessKeyID = "AKIA..."
secretAccessKey = "..."
Also I've added the host entry in /etc/hosts:
127.0.0.1 service1.mydomain.com
127.0.0.1 service2.mydomain.com
And the relative labels on the containers and I can curl service1.mydomain.com/status and get a 200.
Now my last bit is just the following question:
How should publish all this to the internet? AWS ALB? AWS Network LB? Network Bridge/host/other?

AWS ALB vs AWS Network LB depends on who do you want to handle SSL.
If you have a wildcard certificate and all your services are subdomains of the same domain ALB may be a good choice
If you want to use Let's encrypt with traefik Network LB may be a better choice
In both case your setup will look something like this :
[Internet]
|
[LB]
|
[Target group]
|
[Traefik]
| |
[service1] [service2]
In both case, easiest way to get this is to make traefik ecs services to auto register to the target group.
This can be done at service creation (network configuration section) and can not be done later. Link to documentation
Screen of configuration console

Related

HAProxy config for TCP load balancing in docker container

I'm trying to put a HAProxy loadbalancer in front of my RabbitMQ cluster(which is setup with nodes in separate docker containers). I cannot find many examples for haproxy config for the above setup,
global
debug
defaults
log global
mode tcp
timeout connect 5000
timeout client 50000
timeout server 50000
frontend main
bind *:8089
default_backend app
backend app
balance roundrobin
mode http
server rabbit-1 172.18.0.2:8084
server rabbit-2 172.18.0.3:8085
server rabbit-3 172.18.0.4:8086
In this example, what should I give in the place of the ip addresses for docker containers?

Traefik - Forward to external IP

I've setup my traefik in docker, and it works as intended for container discovery etc.
But I'm getting tired of having to forward port 80 to my Synology NAS in order to renew LetsEncrypt certificates.
Therefore, I want all traffic on port 80 to be forwarded to my NAS (192.168.1.4) on port 80. Based on this answer How to get traefik to redirect to specific non-docker port from inside docker, I have added the following to my docker-compose:
labels:
- "--providers.file=true"
- "--providers.file.filename=/rules.toml"
volumes:
- "/opt/docker_volumes/traefik/rules.toml:/rules.toml"
My rules.toml looks like this:
[http.routers]
# Define a connection between requests and services
[http.routers.nasweb]
rule = "Host(`nas.example.com`)"
entrypoints = ["web"]
service = "nas"
[http.services]
# Define how to reach an existing service on our infrastructure
[http.services.nas.loadBalancer]
[[http.services.nas.loadBalancer.servers]]
url = "http://192.168.1.4:80"
However, I don't see any services in the traefik dashboard, nor does the certificate renew sucessfully. Can anyone spot any errors in the above?
I'm also completely open for a different solution.

kubeadm join times out on non-default NIC/IP

I am trying to configure a K8s cluster on-prem and the servers are running Fedora CoreOS using multiple NICs.
I am configuring the cluster to use a non-default NIC - a bond which is defined with 2 interfaces. All servers can reach each-other over that interface and have HTTP + HTTPS connectivity to the internet.
kubeadm join hangs at:
I0513 13:24:55.516837 16428 token.go:215] [discovery] Failed to request cluster-info, will try again: Get https://${BOND_IP}:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
The relevant kubeadm init config looks like this:
[...]
localAPIEndpoint:
advertiseAddress: ${BOND_IP}
bindPort: 6443
nodeRegistration:
kubeletExtraArgs:
volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
runtime-cgroups: "/systemd/system.slice"
kubelet-cgroups: "/systemd/system.slice"
node-ip: ${BOND_IP}
criSocket: /var/run/dockershim.sock
name: master
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
[...]
The join config that am using looks like this:
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
bootstrapToken:
token: ${TOKEN}
caCertHashes:
- "${SHA}"
apiServerEndpoint: "${BOND_IP}:6443"
nodeRegistration:
kubeletExtraArgs:
volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
runtime-cgroups: "/systemd/system.slice"
kubelet-cgroups: "/systemd/system.slice"
If I am trying to configure it using default eth0, it works without issues.
This is not a connectivity issue. The port test works fine:
# nc -s ${BOND_IP_OF_NODE} -zv ${BOND_IP_OF_MASTER} 6443
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Connected to ${BOND_IP_OF_MASTER}:6443.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
I suspect this happens due to kubelet listening on eth0, if so, can I change it to use a different NIC/IP?
LE: The eth0 connection has been cut off completely (cable out, interface down, connection down).
Now, when we init, if we choose port 0.0.0.0 for the kube-api it defaults to the bond, which we wanted initially:
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 0.0.0.0
result:
[certs] apiserver serving cert is signed for DNS names [emp-prod-nl-hilv-quortex19 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.0.0.1 ${BOND_IP}]
I have even added the 6443 port in iptables for accept and it still times out.. All my CALICO pods are up and running (all pods for that matter in kube-system namespace)
LLE:
I have tested calico and weavenet and both show the same issue. The api-server is up and can be reached from the master using curl but it times out from the nodes.
LLLE:
On the premise that the kube-api is nothing but an HTTPS server, I have tried two options from the node that cannot reach it when doing the kubeadm join:
Ran a python3 simple http server over 6443 and WAS ABLE TO CONNECT from node
Ran an nginx pod and exposed it over another port as NodePort and WAS ABLE TO CONNECT from node
the node just cant reach the api-server on 6443 or any other port for that matter ....
what am i doing wrong...
The cause:
The interface used was in BOND of type ACTIVE-ACTIVE. This made it so kubeadm tried another interface from the 2 bonded, which was not in the same subnet as the IP of the advertised server apparently...
Using ACTIVE-PASSIVE did the trick and was able to join the nodes.
LE: If anyone knows why kubeadm join does not support LACP with ACTIVE-ACTIVE bond setups on FEDORA COREOS please advise here. Otherwise, if additional configurations are required, I would very much like to know what I have missed.

NGINX whitelist internal docker IP

I have a server that runs 2 docker containers, a Node.js API container, and an NGINX-RTMP container. The server itself also uses NGINX as a reverse proxy to sort traffic between these two containers based on port.
The NGINX-RTMP server accesses the API server via it's network alias like so:
on_publish http://api-server:3000/authorize
Which works great to communicate container-to-container. I can also go the other way by using urls like
http://nginx-server:8080/some-endpoint
Now I have a route on the NGINX server that I would like to restrict to just local traffic (i.e. only the API server should be able to hit this location). Now normally I can do this with a simple
# nginx conf file
location /restricted {
allow 127.0.0.1;
deny all;
}
What I would like to do is something like this:
# nginx conf file
location /restricted {
allow api-server;
deny all;
}
But I need to use the actual IP of the container. Now I can get the IP of the container by inspecting it, and I see the IP is 172.17.0.1. However when I look at other instances of this server I see some servers are 172.18.0.1 and 17.14.0.2 so it's not 100% consistent across servers. Now I could just write out all 256 variations of 172.*.0.0/24 but I imagine there must be a 'proper' way to wildcard this in nginx, or even a better way of specifying the container IP in my NGINX conf file. The only information I have found so far is to modify the type of network I'm using for my containers, but I don't want to do that.
How do I properly handle this?
# nginx conf file
location /restricted {
allow 172.*.0.0/24;
deny all;
}
I might have solved this one on my own actually.
Originally I thought I could 172.0.0.1/8 the block to allow all the IPs I thought possible for the local network, but this is wrong.
After reading this article: https://www.arin.net/reference/research/statistics/address_filters/ (archive mirror)
According to standards set forth in Internet Engineering Task Force (IETF) document RFC-1918 , the following IPv4 address ranges are reserved by the IANA for private internets
10.0.0.0/8 IP addresses: 10.0.0.0 – 10.255.255.255
172.16.0.0/12 IP addresses: 172.16.0.0 – 172.31.255.255
192.168.0.0/16 IP addresses: 192.168.0.0 – 192.168.255.255
Notice that the 172 net is a /12 and not /8.
Which is explained as
In August 2012, ARIN began allocating “172” address space to internet service, wireless, and content providers.
So I believe the correct method is:
# nginx conf file
location /restricted {
allow 172.16.0.0/12;
deny all;
}

Traefik - Don't understand whitelistSourceRange

I don't understand how to work whitelistSourceRange...
When I set the docker label traefik.frontend.whitelistSourceRange with my public IP, I cannot access to the container.
And in the logs, we can see that the IP is a private IP of... I don't know what ? (not the ip contianer and not my private ip)
time="2018-01-28T15:35:55Z" level=debug msg="source-IP 10.255.0.2 matched none of the whitelists - rejecting"
time="2018-01-28T15:35:55Z" level=debug msg="source-IP 10.255.0.2 matched none of the whitelists - rejecting"
Somebody can help me ?
thanks,
It looks like you have something between your machine and your Docker running Traefik. Perhaps an ELB in AWS or an Ingress controller in k8s? You will want to set an 'X-Forwarded-*' header with your original IP address and configure Traefik to use forwarded headers.
For example:
[entryPoints.http.forwardedHeaders]
trustedIPs = ["127.0.0.1/32", "192.168.1.7"]
http://docs.traefik.io/configuration/entrypoints/#whitelisting

Resources