I'm encountering a strange issue with docker-compose on one of my systems.
I have two TICK (Telegraf, InfluxDB, Chronograf, Kapacitor) docker-compose "projects" on the same machine. Using the following docker-compose.yml
Since both services are proxied behind the same NGINX SSL instance, they both join a common nginx_proxy external network.
Issue is that as soon as I start the second compose stack. Somehow the first stack starts misbehaving: a few (like 20%) requests from the first chronograf intance targeting the influxdb service via the influxdb hostname are somehow redirected to the influxb instance from the second stack.
I understand that since they are on the same nginx network, they can communicate, but how can I force the first instance to always target it's own service and not cross-compose? Tried to specify links but it did not work.
Any configuration I could setup to achieve this isolation without having to rename all my services?
I wouldn't expect the embedded DNS server to do what you want automatically as it seems ambiguous to me (the docker-compose file doesn't express what you expressed in natural language here, which is perfectly fine and non-ambiguous). In fact I'm even surprised it allows it to run (seems to be doing round-robin across the containers, which would explain why only some requests are misrouted).
To answer your question, I would simply use non-ambiguous aliases or just non-ambiguous service names, though there might be other solutions.
Related
tldr; does docker swarm have a forceful and centered proxy setting that explicitly proxies all internet traffic in all services that is hosted in the cluster? Or any other tip of how to go about using a global proxy solution in a swarm cluster...?
Obs! this is not a question about a reversed proxy.
I have a docker swarm cluster (moving to Kubernatives as a solution is off-topic)
I have 3 managers and 3 workers, I label the workers accordingly to the expected containers they can host. The cluster only deploys docker swarm services, when I write "container" in this writing I'm referring to a docker swarm service container.
One of the workers is labelless, though active, and therefore does not host any containers to any service. If I would label the worker to allow it to host any container, then I will suffer issues in different firewalls that I don't always control, because the IP simply is not allowed.
This causes the problem for me that I can't do horizontal scaling, because when I add a new worker to the cluster, I also add a new IP that the requests can originate from. To update the many firewalls that would need to be updated because of a horizontal scaling is quite large, and simply not an option.
In my attempt to solve this on my own, I did what every desperate developer does and googled for a solution... and there is a simple and official documentation to be able to achieve this: https://docs.docker.com/network/proxy/
I followed the environment variables examples on that page. Doing so did however not really help, none of the traffic goes through the proxy I configured. After some digging, I noticed that this is due to nodejs (all services are written using nodejs), ignoring the proxy settings set by the environment. To solve that nodejs can use these proxy settings, I have to refactor a lot of components in a lot of services... a workload that is quite trumendus and possibly dangerous to perform given the different protocols and ports I use to connect to different infrastructural services outside the cluster...
I expect there to be a better solution for this, I expect there to be a built in functionality that forces all internet access from the containers to go through this proxy, a setting I don't have to make in the code, in my implementations. I expect there to be a wrapping solution that I can control in a central manner.
Now reading this again, I think maybe I should have tested the docker client configuration on the same page to see if it has the desired effect I'm requiring, but I assume they both would have the same outcome, being described on the same page with no noticeable difference written in the documentation.
My question is, is there a solution, that I just don't seem to be able to find, that wraps the proxy functionality around all the services? or is it a requirement to solve these issues in the implementation itself?
My thought is to maybe depend on an image, that in its turn depends on the nodejs image that I use today - that is responsible for this wrapping functionality, though still on an implantation level. Doing so would however still force the inheriting of a distributed solution of this kind - if I need to change the proxy configurations, then I need to change them everywhere, and redeploy everything... given a less complex solution without an in common data access layer.
I'm trying to get Erlang and Docker Swarm to play together, and I'm having a hard time. The main issue appears to be that Erlang requires my nodes to have fully qualified domain names, or at least something that looks like it (hostnames without periods tend to cause errors).
Previously, I did use a docker overlay network, but not stack/services, and I would run containers manually named "xxx.yyy". Docker would resolve "xxx.yyy", also if nodes were on different hosts, and Erlang was happy. Now, I would like to switch to using docker stack, but in this case Docker no longer resolves by container name but by service name. There is just one issue: periods are illegal in service names, but if I try to name my Erlang nodes as 'nodename#hostname' (without periods) this causes problems on the Erlang side.
I noticed that I can set the "hostname:" property for a service, and there I can include periods, but this name only resolves inside the container itself, it is not recognized over the network.
Is there some way I can set a custom address for a Docker service, which can contain periods, which will resolve properly on nodes of other services, and which remains stable as the service replaces its containers?
NB: I have tried using Erlang's "-sname", but this seems to use the container id as hostname, which still needs to be included when connecting to other nodes over the network, and this is not stable as the service updates/restarts.
NB2: I also found info here that it might be possible to use the "links:" property, but it looks like that doesn't work on swarm (perhaps only on compose?).
We are moving some of our internal services to rely on Docker instead of direct installation on the host OS (good thing, right :).
We use docker stack command with compose file (as it felt to us it is the modern approach). But we are not sure about how to properly make our stacks modular, while allowing composition:
Let's imagine we have two stacks: stackA and stackB. Those two can perfectly be used in isolation, so for the moment we decided to host them in two separate repositories, each containing the docker-compose.yml of the corresponding stack.
Yet, there is also a mode where stackB can communicate with stackA to provide additional features. On some nodes, we might want to deploy both, and have them communicate.
By default, when we start both stacks on the same node with:
docker stack deploy -c stackA/ A-stack
docker stack deploy -c stackB/ B-stack
Both end up on different overlay networks, and cannot easily communicate.
It seems we are faced with a choice, for which we could only find 3 options at the moment:
We have seen ways to add external networks to stackB in its compose file, but that means now the stackB can only be deployed if stackA already runs (because it wants to join an external network)
We could define another compose file, manually merging both. But that leads us to maintain another repo, and duplicate changes.
We could have the stack communicate over the host network through exposed ports, but it might feel a bit weird.
Is there a best/recommended approach to keep different stacks modular, while allowing to easily compose them?
Or is it an implicit assumption that as soon as two containers are supposed to communicate, they have to be deployed from the same compose file?
I handle usually handle more than one stack in cases when I want handle them separate. Common situations are horizontal scaling of same web service image for different customer installations with different configurations f.e. databases.
The separated stacks allow me easy to shutdown them without any impact of other installations
I also like the standard naming conventions in multiple stack installations. Same services have same names beside the stack prefix.
To let the stack communicate over the boundaries thy only have to share the same network.
The first stack defines in my cases implizit a network and the other stack join that network by compose file configuration.
...
networks:
default:
external:
name: FIRST_STACK_NAME_default
...
I'm totally new to docker and started yesterday to do some tutorials. I want to build a small test application consisting of several different services (replicated and so on) that interact with each other and encountered a problem regarding 'service-discovery'. I started with the get-started tutorials on docker.com and at the moment i'm not really sure what's best practice in the world of docker to let the different containers in a network get to know each other...
As this is a rather vague 'problem description', i try to make this more precise. I want to use a few independent services (e.g. with stuff like postgre, mongodb, redis and rabbitmq...) together with a set of worker nodes to which work is assigned by a dedicated master node. Since it seems to be quite convenient, I wanted to use a docker-composer.yml file to define all my services and deploy them as a stack.
Moreover, I created a custom network and since it seems not to be possible to attach a stacked service to a bridge network, I created an attachable overlay network.
To finally get to the point: even though the services are deployed correctly, their actual container-name is random and without using somekind of service registry I'm not able to resolve their addresses.
A simple solution would be to use single containers with fixed container names - however this does not seem to be a best practice solution (even though it is actually just a docker-based DNS that is based on container names rather than domain names). Another problem are the randomly generated container names that contain underscores, and hence these names are not valid addresses that can be resolved...
best regards
Have you looked at something like Kubernetes? To quote from the home page:
It groups containers that make up an application into logical units for easy management and discovery.
I'm having great success so far using Mesos, Marathon, and Docker to manage a fleet of servers, and the containers I'm placing on them. However, I'd now like to go a bit further and start doing things like automatically linking an haproxy container to each main docker service that starts, or provide other daemon based and containerized services that are linked and only available to the single parent container.
Normally, I'd start up the helper service first with some name, then when I started the real service, I'd link it to the helper and everything would be fine. How does this model fit in to Marathon and Mesos though? It seems for now at least that the containerization assumes a single container.
I had one idea to start the helper service first, on whatever host it could find, then add a constraint to the real service that the hostname = helper service's hostname, but that seems like it'd cause issues with resource offers and race conditions for those resources.
I've also thought to provide an "embed", or "deep-link" functionality to docker, or to the executor scripts that start the docker containers.
Before I head down any of these paths, I wanted to find out if someone else had solved this problem, or if I was just horribly over thinking things.
Thanks!
you're wandering in uncharted territory! ☺
There are multiple approaches here; and none of them is perfect, but the situation will improve in future versions of Docker, thanks to orchestration hooks.
One way is to use good old service discovery and registration. I.E., when a service starts, it will figure out its publicly available address, and register itself in e.g. Zookeeper, Etcd, or even Redis. Since it's not trivial for a service to figure out its publicly available address (unless you adopt some conventions, e.g. always mapping port X:X instead of letting Docker assing random ports), you might want to do the registration from outside. That means that your orchestration layer (Mesos in that case) would start the container, then figure out the host and port, and put that in your service discovery system. I'm not extremely familiar with Marathon, but you should be able to register a hook for that. Then, other containers will just look up the endpoint address in the service discovery registry, plain and simple.
You could also look at Skydock, which automatically registers DNS names for your containers with Skydns. However, it's currently single-host, so if you like that idea, you'll have to extend it somehow to support multiple hosts, and maybe SRV records.
Another approach is to use "well-known entry points". This is actually is simplified case of service discovery. It means that you will make sure that your services will always run on pre-set hosts and ports, so that you can use those addresses statically. Of course, this is bad (because it will make your life harder when you will want to reproduce the environment for testing/staging purposes), but if you have no clue at all about service discovery, well, it could be a start.
You could also use Pipework to create one (or multiple) virtual network spanning across multiple hosts, and binding your containers together. Pipework will let you assign IP addresses manually, or automatically through DHCP. This approach is not recommended, though, but it's a good fit if you also want to plug your containers into an existing network architecture (e.g. VLANs...).
No matter which solution you decide to use, I highly recommend to "pretend" that you're using links. I.e. instead of hard-coding your app configuration to connect to (random example) my-postgresql-db:5432, use environment variables DB_PORT_5432_TCP_ADDR and DB_PORT_5432_TCP_PORT (as if it were a link), and set those variables when starting the container. That way, if you "fold down" your containers into a simpler environment without service discovery etc., you can easily fallback on links without efforts.