I have a docker swarm running on 2 nodes created via Oracle VirtualBox installed on CentOS 7. I am able to deploy a stack running 6 containers equally distributed over the two machines.
However, I am unable to connect to the deployed application with ports exposed.
Here's the content of my Docker Compose File
version: "3"
services:
web:
image: <myusername>/friendlyhello:latest
deploy:
replicas: 6
resources:
limits:
cpus: "0.1"
memory: 50M
restart_policy:
condition: on-failure
ports:
- "80:80"
networks:
- webnet
networks:
webnet:
Here is the output of docker-machine ls:
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
myvm1 * virtualbox Running tcp://192.168.99.100:2376 v18.09.0
myvm2 - virtualbox Running tcp://192.168.99.101:2376 v18.09.0
Here is the error from my curl command
curl http://192.168.99.100/
curl: (7) Failed connect to 192.168.99.100:80; Connection refused
Even though my application seems to be running fine.
docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
i5nw2wcir9j3 getstartedlab_web replicated 6/6 harmanspall/friendlyhello:latest *:80->80/tcp
docker service ps getstartedlab_web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
dn7mrfo1yvg4 getstartedlab_web.1 harmanspall/friendlyhello:latest myvm1 Running Running 28 minutes ago
jxkr1psbvmpc getstartedlab_web.2 harmanspall/friendlyhello:latest myvm2 Running Running 28 minutes ago
jttd4t6b9gz5 getstartedlab_web.3 harmanspall/friendlyhello:latest myvm1 Running Running 28 minutes ago
zhs0c7ygj8cs getstartedlab_web.4 harmanspall/friendlyhello:latest myvm2 Running Running 28 minutes ago
mx6gykk3qocd getstartedlab_web.5 harmanspall/friendlyhello:latest myvm1 Running Running 28 minutes ago
pku7f60ij0bq getstartedlab_web.6 harmanspall/friendlyhello:latest myvm2 Running Running 28 minutes ago
My Docker network list, as seen from Swarm Manager:
NETWORK ID NAME DRIVER SCOPE
5c502a957a70 bridge bridge local
a3b1f749c09f docker_gwbridge bridge local
80nens8mmp6i getstartedlab_webnet overlay swarm
c9647a0f6c30 host host local
mj60zgzhiwjf ingress overlay swarm
5adba823ce78 none null local
Any pointers would be appreciated.
~~ EDIT ~~
This does not seem to be an issue with connectivity to VMs since I am able to ping my Virtualbox
ping 192.168.99.100 -c 5
PING 192.168.99.100 (192.168.99.100) 56(84) bytes of data.
64 bytes from 192.168.99.100: icmp_seq=1 ttl=64 time=0.246 ms
64 bytes from 192.168.99.100: icmp_seq=2 ttl=64 time=0.215 ms
64 bytes from 192.168.99.100: icmp_seq=3 ttl=64 time=0.226 ms
64 bytes from 192.168.99.100: icmp_seq=4 ttl=64 time=0.251 ms
64 bytes from 192.168.99.100: icmp_seq=5 ttl=64 time=0.262 ms
--- 192.168.99.100 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.215/0.240/0.262/0.017 ms
It also fails when I try curl from inside the VM
docker-machine ssh myvm1 "curl http://192.168.99.100/"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 192.168.99.100 port 80: Connection refused
exit status 7
docker-machine ssh myvm1 "curl http://localhost/"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 80: Connection refused
exit status 7
Related
I am trying to connect two locally developed projects running on docker-compose by using external networking.
From one side I have an 1st application intended to be exposed. Compose contains hosts: app and rabbit:
version: '3.4'
services:
app:
# ...
rabbit:
# ...
networks:
default:
driver: bridge
From other side I have second application expected to see 1st application:
version: '3.4'
services:
app:
# ...
networks:
- paymentservice_default
- default
networks:
paymentservice_default:
external: true
Reaching host rabbit.paymentservice_default is possible.
However service app (1st) conflicts with app (2nd):
root#6db86687229c:/app# ping app.paymentservice_default
PING app.paymentservice_default (192.168.80.6) 56(84) bytes of data.
root#6db86687229c:/app# ping app
PING app (192.168.80.6) 56(84) bytes of data.
In general from 2nd compose perspective hosts app and app.paymentservice_default shares same IP making app.paymentservice_default undiscoverable.
The question here is, do I have proper configuration and conflict can be avoided without changing service names app? Why this constraint? Taking consideration that every docker-compose configuration is shared across projects and can be developed in micro-services world.
$ docker-compose --version
docker-compose version 1.17.1, build unknown
$ docker --version
Docker version 19.03.4, build 9013bf583a
Thank you.
I use the following configuration on Docker Playground
paymentservice.docker-compose.yml
version: '3.4'
services:
app:
image: busybox
# keep container running
command: tail -f /dev/null
rabbit:
image: rabbitmq
networks:
default:
driver: bridge
other.docker-compose.yml
version: '3.4'
services:
app:
image: busybox
# keep container running
command: tail -f /dev/null
networks:
- paymentservice_default
- default
networks:
paymentservice_default:
external: true
Run both projects
$ COMPOSE_PROJECT_NAME=paymentservice docker-compose -f paymentservice.docker-compose.yml up -d
$ COMPOSE_PROJECT_NAME=other docker-compose -f other.docker-compose.yml up -d
Show Docker IPs
$ docker ps -q | xargs -n 1 docker inspect --format '{{ .Name }} {{range .NetworkSettings.Networks}} {{.IPAddress}}{{end}}' | sed 's#^/##';
I got
other_app_1 172.20.0.2 172.19.0.4
paymentservice_app_1 172.19.0.3
paymentservice_rabbit_1 172.19.0.2
and I pinged paymentservice_app_1 (172.19.0.3) from other_app_1 using app.paymentservice_default
$ docker exec -it other_app_1 ping -c 1 app.paymentservice_default
PING app.paymentservice_default (172.19.0.3): 56 data bytes
64 bytes from 172.19.0.3: seq=0 ttl=64 time=0.258 ms
--- app.paymentservice_default ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.258/0.258/0.258 ms
and I pinged other_app_1 (172.20.0.2) from other_app_1 using app
$ docker exec -it other_app_1 ping -c 1 app
PING app (172.20.0.2): 56 data bytes
64 bytes from 172.20.0.2: seq=0 ttl=64 time=0.054 ms
--- app ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.054/0.054/0.054 ms
As you can see, I can access the 1st app (of paymentservice.docker-compose.yml) from the 2nd app (of other.docker-compose.yml).
The same works in the other direction. I pinged other_app_1 (172.19.0.4) from paymentservice_app_1 using app.paymentservice_default
$ docker exec -it paymentservice_app_1 ping -c 1 app.paymentservice_default
PING app.paymentservice_default (172.19.0.4): 56 data bytes
64 bytes from 172.19.0.4: seq=0 ttl=64 time=0.198 ms
--- app.paymentservice_default ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.198/0.198/0.198 ms
I pinged paymentservice_app_1 (172.19.0.3) from paymentservice_app_1 using app
$ docker exec -it paymentservice_app_1 ping -c 1 app
PING app (172.19.0.3): 56 data bytes
64 bytes from 172.19.0.3: seq=0 ttl=64 time=0.057 ms
--- app ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.057/0.057/0.057 ms
As you can see, I can access app service of both projects. If I like to access the service of the same project, I use the default network of the project. If I'd like to access the service of another project, I use the external network shared between both projects.
Note: I would recommend to make this more explicit by creating the shared network outside of the projects using the command line
docker network create shared-between-paymentservice-and-other
and declaring it as external in both projects.
Note: There is still the limitation that service discovery may not work if you have 3 projects with the same service name (e.g. app) in the same (external) network (sort of a namespace). In that case, it might be a better idea to rename your services, use multiple external networks, define aliases or use a totally different approach to discover/identify the Docker containers.
Afterword
Has that been the requirement? I tried to reproduce your issue, but I'm not sure if I did the same as you. For example, I'm not sure, where you are running ping. Is root#6db86687229c the Docker host or a Docker container? Which container? I assumed it is the Docker container of service app of other.docker-compose.yml. Please comment if I'm missing something or misinterpreted your question and I will update my answer. Then I may explain in more detail or make another suggestion how to do service discovery between multiple Docker Compose projects.
Appendix
Cleanup
$ COMPOSE_PROJECT_NAME=other docker-compose -f other.docker-compose.yml down
$ COMPOSE_PROJECT_NAME=paymentservice docker-compose -f paymentservice.docker-compose.yml down
Versions
$ docker --version
Docker version 20.10.0, build 7287ab3
$ docker-compose --version
docker-compose version 1.26.0, build unknown
Solved at bottom
But why do I have to append :4000?
I'm following the docker get-started Guide here, https://docs.docker.com/get-started/part4/
I'm fairly certain I've done everything correctly, but am wondering why I can't connect to view the app after deploying it.
I've set my env to my VM, myvm1, for reference to following commands.
docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
099e16249604 beresj/getting-started:part2 "python app.py" 12 seconds ago Up 12 seconds 80/tcp getstartedlab_web.5.y0e2k1r1ev47u24e5iufkyn3i
6f9a24b343a7 beresj/getting-started:part2 "python app.py" 12 seconds ago Up 12 seconds 80/tcp getstartedlab_web.3.1pls3osj3uhsb5dyqtt4ts8j6
docker image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
beresj/getting-started <none> e290b6208c21 22 hours ago 131MB
docker stack ls
NAME SERVICES ORCHESTRATOR
getstartedlab 1 Swarm
docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
myvm1 * virtualbox Running tcp://192.168.99.100:2376 v18.09.6
myvm2 - virtualbox Running tcp://192.168.99.101:2376 v18.09.6
docker stack ps getstartedlab
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
vkxx79fh3h85 getstartedlab_web.1 beresj/getting-started:part2 myvm2 Running Running 3 minutes ago
qexbaa3wz0pd getstartedlab_web.2 beresj/getting-started:part2 myvm2 Running Running 3 minutes ago
1pls3osj3uhs getstartedlab_web.3 beresj/getting-started:part2 myvm1 Running Running 3 minutes ago
ucuwen1jrncf getstartedlab_web.4 beresj/getting-started:part2 myvm2 Running Running 3 minutes ago
y0e2k1r1ev47 getstartedlab_web.5 beresj/getting-started:part2 myvm1 Running Running 3 minutes ago
curl 192.168.99.100
curl: (7) Failed to connect to 192.168.99.100 port 80: Connection refused
docker info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 1
Server Version: 18.09.6
...
Swarm: active
NodeID: 0p9qrax9h3by0fupat8ufkfbq
Is Manager: true
ClusterID: 7vnqdk85n8jx6fqck9k7dv2ka
Managers: 1
Nodes: 2
Default Address Pool: 10.0.0.0/8
...
Node Address: 192.168.99.100
Manager Addresses:
192.168.99.100:2377
...
Kernel Version: 4.14.116-boot2docker
Operating System: Boot2Docker 18.09.6 (TCL 8.2.1)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 989.4MiB
Name: myvm1
I would expect to see what I was able to see when I just ran it on my local machine instead of on a VM in a swarm (I think I have the lingo correct?)
Not sure how to check open ports.
Again: this works if I simply remove the stack, unset the docker-machine environment, and just run:
docker stack deploy -c docker-compose.yml getstartedlab
not on the vm.
Thank you in advance. (Also, I'm new hence the get-started guide so I appreciate any help)
Edit
It works if I append :4000 to the VM IP in my url, ex: 192.168.99.100:4000 or 192.168.99.101:4000. It shows the two container Id's listed in 'docker container ls' for myvm1, and the other three are from myvm2. Could anyone tell me why I have to append 4000? Is it because I have ports: "4000:80" in my docker-compose.yml?
Not sure if this will help but if you use docker inspect <instance_id_here>, you can see what ports are exposed.
Exposed ports aren't open ports. You would need to bind a host port to a container port in the docker-compose.yml in order for it to be to be open.
When I print list
docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f15a180315d3 influxdb "/entrypoint.sh infl…" 2 hours ago Exited (128) 2 hours ago influxdb
7b753ba600df influxdb "/entrypoint.sh infl…" 3 hours ago Exited (0) 2 hours ago nervous_fermi
2ddc5d9af400 influxdb "/entrypoint.sh infl…" 3 hours ago Exited (0) 3 hours ago nostalgic_varahamihira
2e174a82d38d influxdb "/entrypoint.sh infl…" 3 hours ago Exited (0) 3 hours a modest_mestorf
But if I try restart
docker container restart influxdb
I get
Error response from daemon: Cannot restart container influxdb: driver failed programming external connectivity on endpoint influxdb (06ee4d738dffecd1a202840699a899286f4bbb88392e4eb227d65670108687a6): Error starting userland proxy: listen tcp 0.0.0.0:8086: bind: address already in use
netstat -nl -p tcp | grep 8086
tcp6 0 0 :::8086 :::* LISTEN 1985/influxd
How to restart docker container?
If I go for
docker kill influxdb
Error response from daemon: Cannot kill container: influxdb: Container f15a180315d38c2f5fac929b2d0b9be3e8ca2a09033648b5c5174c15a64c4d71 is not running
Problem
As indicated by the error message:
Error response from daemon: Cannot restart container influxdb: driver failed programming external connectivity on endpoint influxdb (06ee4d738dffecd1a202840699a899286f4bbb88392e4eb227d65670108687a6): Error starting userland proxy: listen tcp 0.0.0.0:8086: bind: address already in use
The port 8086 was already blocked ( therefore the address already in use part) by another process. Therefore the container was not able to run, because the container tried to start influxdb, but failed because of the already bound port.
Additionally the output of netstat provided the hint, which process occupies the port:
netstat -nl -p tcp | grep 8086
tcp6 0 0 :::8086 :::* LISTEN 1985/influxd
(see the last part: 1985/influxd)
Solution
Kill the other process (first check, if the process is busy and you should save data before stopping it), e.g. using the kill command:
kill 1985
I am trying to add a cluster with replicas using docker-compose scale graylog-es-slave=2 but for a version 3 Dockerfile unlike Docker compose and hostname
What I am trying to do ix figure out how to get the specific node in the replica set
Here is what I have tried
D:\p\liberty-docker>docker exec 706814bf33b2 ping graylog-es-slave -c 2
PING graylog-es-slave (172.19.0.4): 56 data bytes
64 bytes from 172.19.0.4: icmp_seq=0 ttl=64 time=0.067 ms
64 bytes from 172.19.0.4: icmp_seq=1 ttl=64 time=0.104 ms
--- graylog-es-slave ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.067/0.085/0.104/0.000 ms
D:\p\liberty-docker>docker exec 706814bf33b2 ping graylog-es-slave.1 -c 2
ping: unknown host
D:\p\liberty-docker>docker exec 706814bf33b2 ping graylog-es-slave_1 -c 2
ping: unknown host
The docker-compose.yml
version: 3
service:
graylog-es-slave:
image: elasticsearch:2
command: "elasticsearch -Des.cluster.name='graylog'"
environment:
ES_HEAP_SIZE: 2g
deploy:
replicas: 2 <-- this is ignored on docker-compose just putting it here for completeness
Instead of ., use _ (underscore), and add the prefix of the project name (the directory that holds your docker-compose.yml, I assume that it is liberty-docker_graylog):
ping liberty-docker_graylog-es-slave_1
You can see that doing network ls, search for the right network, then docker network inspect network_id.
When running docker containers a swarm cluster do the containers have access to all the IPs of the cluster nodes via ENV variables or otherwise?
I want to run an Elasticsearch instance on each node in my swarm the cluster. And they will discovery each other in unicast mode. Therefore each Elasticsearch instanc needs to be configured with the list of IPs in the cluster.
If you mean that container of one node can access container's IP of other node , then it is not possible . You have to use weave tool to connect container across different node or other tool .
If you are using latest Docker (1.13+) with a swam overlay network, you should be able to get all the cluster's node IPs through DNS round robin.(--endpoint-mode dnsrr)
1) Create an overlay network.
https://docs.docker.com/engine/swarm/networking/
docker network create \
--driver overlay \
my-network
2) Verify swam nodes:
docker#node1:~$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
5l07yt2itiee60xfq7g6c01e4 * node1 Ready Active Leader
pckn7qo3xpbxvs89ni6whyql3 node2 Ready Active
3) Create an alpine container on each nodes using "global" mode:
docker service create --mode global --endpoint-mode dnsrr --name testservice --detach=true --network my-network alpine ash -c "apk update;apk add drill; ping docker.com"
4) verify service is running:
docker#node1:~$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
lmy5s3flw763 testservice global 2/2 alpine:latest
5) Verify that containers were deployed on individual nodes:
$ docker-machine ssh node1 "docker ps"
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4c7055b01479 alpine:latest "ash -c 'apk updat..." 2 minutes ago Up 2 minutes testservice.5l07yt2itiee60xfq7g6c01e4.atvascigh3rvxvlzttaotkrua
$ docker-machine ssh node2 "docker ps"
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
28da546aa0d5 alpine:latest "ash -c 'apk updat..." 2 minutes ago Up 2 minutes testservice.pckn7qo3xpbxvs89ni6whyql3.ebjz4asni4w1f0srna0p3vj4a
6) Confirm individual virtual IP of each containers on node1 and node2:
| => docker-machine ssh node1 "docker exec 4c7055b01479 ash -c 'ip addr'|grep eth0"
349: eth0#if350: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
inet 10.0.0.2/24 scope global eth0
| => docker-machine ssh node2 "docker exec 28da546aa0d5 ash -c 'ip addr'|grep eth0"
319: eth0#if320: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
inet 10.0.0.3/24 scope global eth0
7) Get the container IP addresses for all containers in cluster using Drill dns tool :
| => docker-machine ssh node1 "docker exec 4c7055b01479 ash -c 'drill testservice'"
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 60920
;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; testservice. IN A
;; ANSWER SECTION:
testservice. 600 IN A 10.0.0.3
testservice. 600 IN A 10.0.0.2
;; AUTHORITY SECTION:
;; ADDITIONAL SECTION:
;; Query time: 0 msec
;; SERVER: 127.0.0.11
;; WHEN: Thu Jul 20 19:20:49 2017
;; MSG SIZE rcvd: 83
8) Verify that containers can ping each other:
docker-machine ssh node1 "docker exec 4c7055b01479 ash -c 'ping -c2 10.0.0.3'"
PING 10.0.0.3 (10.0.0.3): 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=0.539 ms
64 bytes from 10.0.0.3: seq=1 ttl=64 time=0.731 ms
--- 10.0.0.3 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.539/0.635/0.731 ms
docker-machine ssh node2 "docker exec 28da546aa0d5 ash -c 'ping -c2 10.0.0.2'"
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: seq=0 ttl=64 time=0.579 ms
64 bytes from 10.0.0.2: seq=1 ttl=64 time=0.736 ms
--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.579/0.657/0.736 ms