I have a small service that is split into 3 docker containers. One backend, one frontend and a small logging part. I now want to start them using coreOS and fleet.
I want to try and start 3 redundant backend containers, so the frontend can switch between them, if one of them fails.
How do i link them? If i only use one, it's easy, i just give it a name, e.g. 'back' and link it like this
docker run --name front --link back:back --link graphite:graphite -p 8080:8080 blurio/hystrixfront
Is it possible to link multiple ones?
The method you us will be somewhat dependent on the type of backend service you are running. If the backend service is http then there are a few good proxy / load balancers to choose from.
nginx
haproxy
The general idea behind these is that your frontend service need only be introduced to a single entry point which nginx or haproxy presents. The tricky part with this, or any cloud service, is that you need to be able to introduce backend services, or remove them, and have them available to the proxy service. There are some good writeups for nginx and haproxy to do this. Here is one:
haproxy tutorial
The real problem here is that it isn't automatic. There may be some techniques that automatically introduce/remove backends for these proxy servers.
Kubernetes (which can be run on top of coreos) has a concept called 'Services'. Using this deployment method you can create a 'service' and another thing called 'replication controller' which provides the 'backend' docker process for the service you describe. Then the replication controller can be instructed to increase/decrease the number of backend processes. You frontend accesses the 'service'. I have been using this recently and it works quite well.
I realize this isn't really a cut and paste answer. I think the question you ask is really the heart of cloud deployment.
As Michael stated, you can get this done automatically by adding a discovery service and bind it to the backend container. The discovery service will add the IP address (usually you'll want to bind this to be the IP address of your private network to avoid unnecessary bandwidth usage) and port in the etcd key-value store and can be read from the load balancer container to automatically update the load balancer to add the available nodes.
There is a good tutorial by Digital Ocean on this:
https://www.digitalocean.com/community/tutorials/how-to-use-confd-and-etcd-to-dynamically-reconfigure-services-in-coreos
Related
Scenario:
There is a container running with image version 1.0 and exposed port 8080 on localhost 80. The new version of the image is available, and there is a need to switch those versions. No, any orchestration tool is running ( Kubernetes, OpenShift etc...).
Is it possible to start a container with version 1.1 make it run without a problem
Please, keep in mind that I don't want to keep it simple, no replication, etc.
Simply docker container with the binded port to localhost.
Questions:
1. Is it possible to switch exposing of port between containers without downtime?
2. If not, is there is any mechanism implemented with docker (free edition) to do such switch?
Without downtime, you'd need a second replica of the service up an running, and a proxy in front of that service that's listening to user requests and routing from one to the other. Both Swarm Mode and Kubernetes provide this capability with similar tools, the port being exposed is indirectly connected to the app via either an application reverse proxy, or some iptables rules and ipvs entries in the kernel.
Out of the box, recent versions of docker include support for Swarm Mode with nothing additional to install. You can run a simple docker swarm init to start a single node swarm cluster in less than a second. And then instead of docker-compose up you switch to docker stack deploy -c docker-compose.yml $stack_name to manage your projects with almost the same compose file. For swarm mode, you'll want to be on version 3 of the compose file syntax.
For a v3 syntax compose file in swarm mode that has no outage on an update, you'll want healthcheck's defined in your image to monitor the application and report back when it's ready to receive requests. Then you'll want a deploy section of the compose file to either have multiple replicas for HA, or at least configure a single replica to have a "start-first" policy to ensure the new service is up before stopping the old one. See the compose docs for settings to adjust: https://docs.docker.com/compose/compose-file/#update_config
For an application based reverse proxy in docker, I really do like traefik, but more to allow me to run multiple http based container services with a single port opened. This allows me to mapping requests based off the hostname/path/http header to the right container, while at the same time giving features to migrate between different versions with weighting of which backend to use so you can do more than a simple round-robin load balancing during an upgrade.
There is no mechanism native to Docker that would allow you replace one container with another with no interruption. On the other hand, the duration of the interruption can probably be measured in milliseconds; whether or not this is really an issue for you depends entirely on your application.
You can get the behavior you want by introducing a dynamic reverse proxy such as Traefik into your configuration. The proxy binds to host ports and handles requests from remote systems, then distributes those requests to one or more backend containers.
You can create and remove backend containers as you please, and as long as at least one is running your application will be available. For your specific use case, this means that you can start the new version of your application first, then retire the old one, all without any interruption in service.
I'm interested in building microservices, but I'm getting a bit stuck on how service discovery should work when I've got multiple instances of a single microservice.
Suppose I've got an "OCR" app that reads text from an image.
Deploying that as 1 instance is easy, however, what if I want 50 instances of those?
I can run docker swarm to spin up get those 50 instances, but how do I send a request to any one of them, i.e. I don't want to have to know the exact container name of a specific instance, I don't care which one I get, as long as it's healthy, just send my request to any of the "OCR" containers.
How do I achieve this?
I've been looking into Consul and it seems very promising.
I especially like the HTTP api, (Although I'm a little unsure of how I would retrieve the url for the service I'm interested in. Would I need to do it before every request to make sure I'm pointing to a healthy instance?).
If I wanted to use consul, what would be the steps be in relation to docker swarm? Do I just need to register the service in consul when the container starts up, and it will automatically get de-registered if it fails right?).
After that, all of my containers just need to be aware of where consul is (And I guess I could stick a load balancer infront of it, incase I ever want to scale out consul itself to a bunch of instances?)
Please let me know if I'm going completely in the wrong direction.
If anyone could also suggest any articles or books on this topic I'd appreciate it.
Thanks.
When you're using Docker Swarm Mode, you get service discovery with load balancing for free.
DNSRR is in the key concept: https://docs.docker.com/engine/swarm/key-concepts/#load-balancing
Say you deploy OCR-app.
docker service create --network dev --name ORC-app --replicas 5 OCR-app:latest
The docker manager will deploy OCR-app in this case five times on nodes of your swarm network. Every other service which is part of the same docker network dev can request the OCR-app by it's name. E.g. GET http://OCR-app:4000/do/something.
Internally docker swarm uses round robin for forward the request automatically to one of the five services.
I want to find a way to deploy an etcd cluster as a Docker Swarm service that would automatically configure itself without any interaction. Basically, I think of something in spirit of this command:
docker service create --name etcd --replicas 3 my-custom-image/etcd
I'm assuming that overlay network is configured to be secure and provide both encryption and authentication, so I believe I don't need TLS, not even --auto-tls. Don't want an extra headache finding a way to provision the certificates, when this can be solved on the another layer.
I need an unique --name for each instance, but I can get that from an entrypoint script that would use export ETCD_NAME=$(hostname --short).
The problem is, I'm stuck on initial configuration. Based on the clustering guide there are three options, but none seems to fit:
The DNS discovery scenario is closest to what I'm looking for, but Docker doesn't support DNS SRV records discovery at the moment. I can lookup etcd and I will get all the IPs of my nodes' containers, but there are no _etcd-server._tcp records.
I cannot automatically build ETCD_INITIAL_CLUSTER because while I know the IPs, I don't know the names of the other nodes and I'm not aware about any way to figure those out. (I'm not going to expose Docker API socket to etcd container for this.)
There is no preexisting etcd cluster, and while supplying the initial configuration URI from discovery.etcd.io is a possible workaround I'm interested in not doing this. I'm aiming for "just deploy a stack from this docker-compose.yml and it'll automatically do the right thing, no questions asked" no-brainer scenario.
Is there any trick I can pull?
As you have correctly said you know the IPs of your nodes’ containers,
so the suggested trick is to simply build the required etcd names as derivatives of each node’s IP address.
inside each container etcd is named using this particular container's IP i.e. etcd-$ip
ETCD_INITIAL_CLUSTER is populated using other containers' IPs in a similar way
The names could be as simple as etcd-$ip or even better i.e. we could use the netmask to calculate the node’s IP on this network to make the names prettier.
In this case in a simple 3-nodes configuration one could end up having names like etcd-02 etcd-03 etc
No specific requirements exist for the name attribute, it just needs to be unique and human-readable. Although it indeed looks like a trick it might work
I'm learning Docker-Swarm with Consul and found some issues I don't really understand. Basically, I created a Docker-Swarm cluster (node-01 and node-02) with Consul Sevice Discovery. I then run a multi-container application (Express app with Mongo) and I can see it is running on node-02. In order to run it, I have to go in and find the IP address of my node-02 and then open the browser.
It works fine, it's just that I was expecting that I could just go to some virtual IP (or DNS) and that it the Consul service (or Swarm) would then translate it to the correct IP address of node-02 in this example.
Next item is that when I log into Consul web UI, I was expecting to see the nodes under the 'nodes' menu, but that seems not to be the case. I was also expecting to get an overview of the 'applications' or 'services' I was running on the node-01 and node-02, but that is also not the case.
My questions are:
Can someone explain why I would need to manually find out on which node in the cluster my app is running. Cannot imagine this is done in larger deployments.
Can someone address why I don't see the 'nodes' and 'services' in the Consul UI?
Note: I tried to be as short as possible though I have been documenting the full setup in a blog post (with screenshots) for those who want to see more details. Go to blog post
Question 1
I would like to access the service without having to use the Swarm agent's IP address
Solution
It is feasible, you just need to start up a reverse proxy such as nginx in a container (here are the official nginx images). At the start up of this container use the option --link with the name of the application. Thus the IP address of this container will added in the file /etc/hosts of the reverse proxy container (remember to use --name and --hostname). Run this reverse proxy container on a specific node.
So the solution to get rid of the IP address issue is to deploy another container on a specific node (and then specific IP address)? Yes! But using --link will make this issue scalable ;)
Question 2
I was expecting to see the nodes under the 'nodes' menu, but that seems not to be the case.
What you do mean? What did you expect? Do you need to query the k,v-storage DB?
Check this: https://github.com/vmudigal/microservices-sample
Microservices Sample Architecture
I am writing an application that is composed of a few spring boot based microservices with a zuul based reverse proxy in the front-
It works when I start the services on my machine, but for server rollout I'd like to use docker for the services, but this seems not to be possible right now.
Normally you would have a fixed "internal" port and randomized ports at the outside of the container. But the app in the container doesn't know the outside port (and IP).
The Netflix tools match what I would want to write an efficient microservice architecture and conceptually I really like docker.
As far as I can see it would be very troublesome to start the container, gather the outside port on the host and pass it to the app, because you can't simply change the port after the app is started.
Is there any way to use eureka with docker based clients?
[Update]
I guess I did a poor job explaining the problem. So maybe this clarifies it a bit more:
The eureka server itself can run in docker, as I have only one and the outside port doesn't matter. I can use the link feature to access it from the clients.
The problem is the URL that the clients register themselves with.
This is for example https://localhost:8080/ but due to dynamic port assignment it is really only accessible via https://localhost:54321/
So eureka will return the wrong URL for the services.
UPDATE
I have updated my answer below, so have a look there.
I have found a solution myself, which is maybe not the best solution, but it fits for me...
When you start docker with "--net=host" (host networking), then you use the hosts network stack directly. Then I just use 0 as port for spring-boot and spring randomizes the port for me and as it's using the hosts networking stack there is no translation to a different port (and IP).
There are some drawbacks though:
When you use host networking you can't use the link-feature for these containers as link source or target.
Using the hosts network stack leads to less encapsulation of the instance, which maybe a problem depending on your project.
I hope it helps
A lot of time has passed and I think I should elaborate this a little bit further:
If you use docker to host your spring application, just don't use a random port! Use a fixed port because every container gets his own IP anyway so every service can use the same port. This makes life a lot easier.
If you have a public facing service then you would use a fixed port anyway.
For local starts via maven or for example the command line have a dedicated profile that uses randomized ports so you don't have conflicts (but be aware that there are or have been a few bugs surrounding random ports and service registration)
if for whatever reason you want to or need to use host networking you can use randomized ports of course, but most of the time you shouldn't!
You can set up a directory for each docker instance and share it between the host and the instance and then write the port and IP address to a file in that directory.
$ instanceName=$(generate random instance name)
$ dirName=/var/lib/docker/metadata/$instanceName
$ mkdir -p $dirName
$ docker run -name $instanceName -v ${dirName}:/mnt/metadata ...
$ echo $(get port number and host IP) > ${dirName}/external-address
Then you just read /mnt/metadata/external-address from your application and use that information with Eureka.