OS: window7
virtualization tool: virtualbox
virtual hypervisor: centos7
linux core as below
[root#localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Two host IPs:
192.168.100.101
192.168.100.102
The steps:
curl -L git.io/weave -o /usr/local/bin/weave
chmod a+x /usr/local/bin/weave
[root#localhost ~]# weave launch 192.168.100.102
WARNING: existing iptables rule
'-A FORWARD -j REJECT --reject-with icmp-host-prohibited'
will block name resolution via weaveDNS - please reconfigure your firewall.
cannot locate running docker daemon
Warning: unable to detect proxy TLS configuration. To enable TLS, launch the proxy with ‘weave launch’ and supply TLS options. To suppress this warning, supply the ‘–no-detect-tls’ option.
3227932d5be77917c4e0b780cafe1171287c1029637f2360ece580fe6239cb4f
[root#localhost ~]# weave status
Version: 2.1.1 (failed to check latest version - see logs; next check at 2017/11/28 19:18:07)
Service: router
Protocol: weave 1..2
Name: 06:e0:c4:68:0c:ae(localhost.localdomain)
Encryption: disabled
PeerDiscovery: enabled
Targets: 1
Connections: 1 (1 failed)
Peers: 1
TrustedSubnets: none
Service: ipam
Status: ready
Range: 10.32.0.0/12
DefaultSubnet: 10.32.0.0/12
Service: dns
Domain: weave.local.
Upstream: 135.251.4.190, 135.251.38.218, 192.168.1.1
TTL: 1
Entries: 0
Service: proxy
Address: unix:///var/run/weave/weave.sock
Service: plugin (legacy)
DriverName: weave
[root#localhost ~]# weave version
weave script 2.1.1
weave 2.1.1
As above , the Connections: 1 (1 failed) , the docker run on these two hosts can’t ping each other.
And when I type weave version, it only showed weave script 2.1.1 and weave 2.1.1, not like other articles said it will show weaveexec and plugin or weavedns and so on. Is this the difference between versions or some mistakes I have made ?
Please help , thank you very much!
my docker log as below
[root#localhost ~]# docker logs weave
INFO: 2017/11/29 01:08:07.807752 Command line options: map[dns-effective-listen-address:172.17.0.1 nickname:localhost.localdomain dns-listen-address:172.17.0.1:53 ipalloc-range:10.32.0.0/12 status-addr:127.0.0.1:6782 weave-bridge:weave H:[unix:///var/run/weave/weave.sock] host-root:/host http-addr:127.0.0.1:6784 port:6783 proxy:true resolv-conf:/var/run/weave/etc/resolv.conf datapath:datapath docker-bridge:docker0 plugin:true]
INFO: 2017/11/29 01:08:07.807841 weave 2.1.1
INFO: 2017/11/29 01:08:07.859209 Docker API on unix:///var/run/docker.sock: &[ApiVersion=1.24 GoVersion=go1.8.3 Os=linux BuildTime=2017-10-24T15:40:21.112972404+00:00 PkgVersion=docker-1.12.6-61.git85d7426.el7.centos.x86_64 Version=1.12.6 Arch=amd64 KernelVersion=3.10.0-693.5.2.el7.x86_64 GitCommit=85d7426/1.12.6]
INFO: 2017/11/29 01:08:07.859520 Using docker bridge IP for DNS: 172.17.0.1
INFO: 2017/11/29 01:08:07.863781 proxy listening on unix:///var/run/weave/weave.sock
INFO: 2017/11/29 01:08:08.940871 Bridge type is bridged_fastdp
INFO: 2017/11/29 01:08:08.940885 Communication between peers is unencrypted.
INFO: 2017/11/29 01:08:08.961891 Our name is 06:e0:c4:68:0c:ae(localhost.localdomain)
INFO: 2017/11/29 01:08:08.962058 Restart/resume detected - using persisted peer list: [192.168.100.102]
INFO: 2017/11/29 01:08:08.972210 Docker API on unix:///var/run/docker.sock: &[KernelVersion=3.10.0-693.5.2.el7.x86_64 PkgVersion=docker-1.12.6-61.git85d7426.el7.centos.x86_64 GoVersion=go1.8.3 Os=linux Arch=amd64 BuildTime=2017-10-24T15:40:21.112972404+00:00 Version=1.12.6 ApiVersion=1.24 GitCommit=85d7426/1.12.6]
INFO: 2017/11/29 01:08:08.974990 Checking for pre-existing addresses on weave bridge
INFO: 2017/11/29 01:08:09.009949 [allocator 06:e0:c4:68:0c:ae] Initialising with persisted data
INFO: 2017/11/29 01:08:09.034491 Listening for DNS queries on 172.17.0.1
INFO: 2017/11/29 01:08:09.086102 Sniffing traffic on datapath (via ODP)
INFO: 2017/11/29 01:08:09.114882 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:08:09.116392 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2017/11/29 01:08:09.116576 Listening for metrics requests on 127.0.0.1:6782
INFO: 2017/11/29 01:08:09.125917 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:08:09.151109 Waiting for Weave API Server...
INFO: 2017/11/29 01:08:09.159548 Finished waiting for Weave API Server
INFO: 2017/11/29 01:08:09.159706 Listening on /run/docker/plugins/weave.sock for global scope
INFO: 2017/11/29 01:08:09.159811 Listening on /run/docker/plugins/weavemesh.sock for local scope
INFO: 2017/11/29 01:08:09.159822 Creating default "weave" network
INFO: 2017/11/29 01:08:09.462160 Discovered local MAC 06:e0:c4:68:0c:ae
INFO: 2017/11/29 01:08:09.547179 Discovered local MAC 46:2b:0d:08:12:be
INFO: 2017/11/29 01:08:09.554830 Discovered local MAC 0e:46:f3:dd:57:96
INFO: 2017/11/29 01:08:11.612424 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:08:11.614477 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:08:13.980824 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:08:13.982289 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:08:18.124543 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:08:18.125556 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:08:23.294574 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:08:23.322022 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:08:37.070537 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:08:37.073928 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:08:39.222651 Error checking version: Get https://checkpoint-api.weave.works/v1/check/weave-net?arch=amd64&flag_docker-version=1.12.6&flag_kernel-version=3.10.0-693.5.2.el7.x86_64&os=linux&signature=fvXv9SDD9r8gjV6d2HrXkVdBv5U72%2BeXQ6NT2u0JkKc%3D&version=2.1.1: dial tcp: lookup checkpoint-api.weave.works on 135.252.166.21:53: read udp 192.168.100.101:34840->135.252.166.21:53: i/o timeout
INFO: 2017/11/29 01:08:46.009136 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:08:46.011168 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:09:16.169210 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:09:16.171278 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:09:42.294136 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:09:42.296081 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:10:28.752091 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:10:28.756481 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:12:03.755330 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:12:03.760374 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:14:30.481453 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:14:30.486632 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:17:41.166716 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:17:41.168341 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:22:38.820826 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:22:38.829815 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
INFO: 2017/11/29 01:23:08.968136 Expired MAC 06:e0:c4:68:0c:ae at 06:e0:c4:68:0c:ae(localhost.localdomain)
INFO: 2017/11/29 01:23:08.968199 Expired MAC 46:2b:0d:08:12:be at 06:e0:c4:68:0c:ae(localhost.localdomain)
INFO: 2017/11/29 01:23:08.968219 Expired MAC 0e:46:f3:dd:57:96 at 06:e0:c4:68:0c:ae(localhost.localdomain)
INFO: 2017/11/29 01:30:27.085406 ->[192.168.100.102:6783] attempting connection
INFO: 2017/11/29 01:30:27.089200 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host
but on 192.168.100.101 , I can ping 192.168.100.102
[root#localhost ~]# ping 192.168.100.102
PING 192.168.100.102 (192.168.100.102) 56(84) bytes of data.
64 bytes from 192.168.100.102: icmp_seq=1 ttl=64 time=1.19 ms
64 bytes from 192.168.100.102: icmp_seq=2 ttl=64 time=1.05 ms
64 bytes from 192.168.100.102: icmp_seq=3 ttl=64 time=0.906 ms
^C
--- 192.168.100.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.906/1.051/1.195/0.120 ms
The issue has been resolved and shared my experience here.
As I described in my question
[root#localhost ~]# weave launch 192.168.100.102
WARNING: existing iptables rule
'-A FORWARD -j REJECT --reject-with icmp-host-prohibited'
will block name resolution via weaveDNS - please reconfigure your firewall.
cannot locate running docker daemon
Warning: unable to detect proxy TLS configuration. To enable TLS, launch the proxy with ‘weave launch’ and supply TLS options. To suppress this warning, supply the ‘–no-detect-tls’ option.
3227932d5be77917c4e0b780cafe1171287c1029637f2360ece580fe6239cb4f
Then on both hosts ,I run command
[root#localhost ~]# iptables -F
And then stop and rm all docker container and stop weave and then restart docker daemon and then on host 192.168.100.101 ,run command
[root#localhost ~]# weave launch
[root#localhost ~]# eval $(weave env)
[root#localhost ~]# docker run --name bbox1 -itd busybox
[root#localhost ~]# docker run --name bbox2 -itd busybox
and on host 192.168.100.102 , run command
[root#localhost ~]# weave launch 192.168.100.101
[root#localhost ~]# eval $(weave env)
[root#localhost ~]# docker run --name bbox3 -itd busybox
And then test
[root#localhost ~]# docker exec bbox3 ping -c2 bbox1
PING bbox1 (10.32.0.1): 56 data bytes
64 bytes from 10.32.0.1: seq=0 ttl=64 time=0.940 ms
64 bytes from 10.32.0.1: seq=1 ttl=64 time=2.362 ms
--- bbox1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.940/1.651/2.362 ms
Done!
so it seemed that the iptables rule is the root cause. I will learn more iptables rules later so that avoid merely using "iptables -F".
Thanks for #Marc Carré very much.
it only showed weave script 2.1.1 and weave 2.1.1, not like other articles said it will show weaveexec and plugin or weavedns and so on
This is expected with your version of Weave Net.
In prior versions, Weave Net started different containers, but starting Weave Net 2.0, these have been merged together.
See also:
https://github.com/weaveworks/weave/blob/master/CHANGELOG.md#release-200
All of Weave Net now runs in one container
Previously we had three separate containers for routing, Docker API proxy and Docker plugin. Running everything in one simplifies start-up and removes the need to detect various error conditions. #1642,#2897,#2936,#2945,#2946,#2951,#2960
https://www.weave.works/blog/weave-net-2-released
In Weave Net 2.0, the various processes forming Weave Net were merged into a single process.
and what follows.
Would you mind pointing to the docs which caused the confusion, so that we could improve these?
Related
My VSCode in WSL:Ubuntu is unable to listen to the xdebug port, because it is blocked by some docker-proxy.
I was following this Solution, but trying VSCode to listen to the xdebug port, results in the following error:
Error: listen EADDRINUSE: address already in use :::9003
Can anyone help with connecting VSCode to xdebug?
Windows 11 says the port is already allocated by wslhost:
PS C:\WINDOWS\system32> Get-Process -Id (Get-NetTCPConnection -LocalPort 9003).OwningProcess
Handles NPM(K) PM(K) WS(K) CPU(s) Id SI ProcessName
------- ------ ----- ----- ------ -- -- -----------
285 47 2288 4748 0,05 19480 1 wslhost
Ubuntu tells, its allocated by some docker-proxy:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9003 0.0.0.0:* LISTEN 17210/docker-proxy
tcp6 0 0 :::9003 :::* LISTEN 17217/docker-proxy
docker-compose-version: docker-compose version 1.25.0
The xdebug.log says:
[Step Debug] INFO: Connecting to configured address/port: host.docker.internal:9003.
[Step Debug] ERR: Time-out connecting to debugging client, waited: 200 ms. Tried: host.docker.internal:9003 (through xdebug.client_host/xdebug.client_port) :-(
For sure as long as nothing is listening.
As to xdebug.client_host I'v tried:
host.docker.internal
xdebug://gateway and xdebug://nameserver refering to this: https://docs.google.com/document/d/1W-NzNtExf5C4eOu3rRQm1WlWnbW44u3ANDDA49d3FD4/edit?pli=1
setting the env-variable with docker-compose.yml: XDEBUG_CONFIG="client_host=..."
Removing the Expose directive from Dockerfile/docker-compose as in this comment doesn't remove the error neither.
Solved it. For others with this challenge:
Inside of wsl-ubuntu -> docker-containter host.docker.internal directs to the wrong ip.
In the wsl-distribution the file /etc/resolv.conf is the ip of the windows host.
To get the correct ip use this answer: How to get the primary IP address of the local machine on Linux and OS X?
My solution is to define an env-variable with this ip:
alias docker_compose_local_ip="ifconfig eth0 | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'"
export DOCKER_COMPOSE_LOCAL_IP=$(docker_compose_local_ip)
and configure the container with it:
services:
service-name:
environment:
- XDEBUG_CONFIG=client_host=${DOCKER_COMPOSE_LOCAL_IP} ...
I have an issue with the docker daemon installed on an Ubuntu 14.04 VM. The logs reveal that ipv6 is enabled hence the docker seems to be listening on this ip address. Essentially, this effects Clair. I have made sure that ipv6 is disabled on the following recommendation here. I also disabled ipv6 in daemon.json as specified in Docker documentation. My docker version is Docker version 17.06.1-ce, build 874a737.
Docker daemon logs :
time="2018-02-20T20:33:17.736203462+01:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 20 01:4860:4860::8844]"
Clair logs:
2018/02/20 20:43:51 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp [::]:6060: connect: cannot assign requested address"; Reconnecting to {[::]:6060 <nil>}
2018/02/20 20:46:14 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp [::]:6060: connect: cannot assign requested address"; Reconnecting to {[::]:6060 <nil>}
It's trying to make an IPv6 connection, but the address is wrong. [::] is IN6ADDR_ANY, not an actual address you can connect to. Provide the correct address in your config.yaml.
Did you mean to connect to localhost?
api:
# v3 grpc/RESTful API server address
addr: "[::1]:6060"
on OS X i started kafka docker image successfully,but it seems that i can't access it on localhost
➜ ~ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1f931da3d661 wurstmeister/zookeeper:3.4.6 "/bin/sh -c '/usr/..." About an hour ago Up About an hour 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp docker_zookeeper_1
8bc36bcf8fdf wurstmeister/kafka:0.10.1.1 "start-kafka.sh" About an hour ago Up About an hour 0.0.0.0:9092->9092/tcp docker_kafka_1
➜ ~ telnet 0.0.0.0:2181
0.0.0.0:2181: nodename nor servname provided, or not known
➜ ~ telnet 0.0.0.0 2181
Trying 0.0.0.0...
telnet: connect to address 0.0.0.0: Connection refused
telnet: Unable to connect to remote host
➜ ~ telnet 192.168.43.193 2181
Trying 192.168.43.193...
telnet: connect to address 192.168.43.193: Connection refused
telnet: Unable to connect to remote host
➜ ~ telnet 127.0.0.1 2181
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host
my docker file is here kafka.yml and use this command to up:
docker-compose -f src/main/docker/kafka.yml up -d
when i use
./mvnw
the console is:
2017-09-15 17:05:46.433 WARN 15871 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
how can i access the 2181 port
EDIT
docker logs 8bc36bcf8fdf
[2017-09-15 08:14:13,386] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/1001. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:393)
at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:379)
at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:70)
at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:51)
at kafka.server.KafkaServer.startup(KafkaServer.scala:270)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2017-09-15 08:14:13,393] INFO [Kafka Server 1001], shutting down (kafka.server.KafkaServer)
docker logs 1f931da3d661
2017-09-14 08:53:05,878 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x15e7ea74c8e0000, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2017-09-14 08:53:05,887 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#1007] - Closed socket connection for client /172.18.0.2:54222 which had sessionid 0x15e7ea74c8e0000
Have you tried using host networking as in this example? https://docs.confluent.io/current/cp-docker-images/docs/quickstart.html#zookeeper
That looks like it will simplify and solve this. I'd also recommend checking out these images instead of the custom ones it looks like you are using because these are being run in production for people so they are known to work well.
Update
I believe the culprit is the master who does not appear to be listening on port 7946. netstat shows that 7946 is listening on the nodes, but not the master. When I check the syslogs for the nodes I see the following error
level=error msg="Failed to join memberlist [10.0.0.12] on retry: 1 error(s) occurred:\n\n* Failed to join 10.0.0.12: dial tcp 10.0.0.12:7946: getsockopt: connection refused"
Original Post
I am running a three node Swarm Mode cluster in AWS; one master and two workers. This is swarm mode not to be confused with docker swarm from pre 1.12.
I created all of the services with docker-machine. Each machine is running Ubuntu 15.10 with Docker 1.12.3.
Linux swarm-master-01 4.2.0-42-generic #49-Ubuntu SMP Tue Jun 28 21:26:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Using the master node I have created a service with the following
docker service create --replicas 1 --name myapp -p 3000 myapp
When I run docker service ps myapp I get the following output
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
02awst8p9pezgpkfzqgz8z79t myapp.1 myapp:latest swarm-node-01 Running Running 19 minutes ago
The running task is deployed to swarm-node-01.
I checked the auto-selected port which was published publicly
$ docker service inspect myapp | jq .[].Endpoint.Ports[].PublishedPort
30000
According to the documentation:
External components, such as cloud load balancers, can access the service on the PublishedPort of any node in the cluster whether or not the node is currently running the task for the service. All nodes in the swarm route ingress connections to a running task instance.
But when I try to curl the nodes who do not have the task running I'm getting connection refused.
$ curl $(docker-machine ip swarm-node-01):30000/stats
{"uptime":"2016-11-09T14:48:35Z","requestCount":7,"statuses":{"200":7},"pid":1,"open_db_conns":0}
$ curl $(docker-machine ip swarm-node-02):30000/stats
curl: (7) Failed to connect to [the IP] port 30000: Connection refused
note: I scrubbed the IP of node-02
My Troubleshooting:
The nodes are both properly connected to the swarm
Scaling the service up to 5 (which inherently deploys the task to every node) makes curl work on every node, because the task is deployed to every node.
UPDATE 1
I initialized the swarm with
docker swarm init --advertise-addr 10.0.0.12:2377 --listen-addr 10.0.0.12:2377
I checked the syslogs from the nodes and I'm seeing the following errors
level=error msg="Failed to join memberlist [10.0.0.12] on retry: 1 error(s) occurred:\n\n* Failed to join 10.0.0.12: dial tcp 10.0.0.12:7946: getsockopt: connection refused"
I checked to see if the ingress port was listening and it doesn't seem to be
ubuntu#swarm-master-01:~$ sudo lsof -i :7946
ubuntu#swarm-master-01:~$ cat < /dev/tcp/10.0.0.12/7946
-bash: connect: Connection refused
-bash: /dev/tcp/10.0.0.12/7946: Connection refused
ubuntu#swarm-master-01:~$ cat < /dev/tcp/0.0.0.0/7946
-bash: connect: Connection refused
-bash: /dev/tcp/0.0.0.0/7946: Connection refused
I was able to get around the issue for now, but I don't know what initially caused it. The overlay network (port 7946) wasn't listening on swarm-master-01. I figured this out with netstat -nlt. I searched the syslogs and found these errors related to the port in the syslog.
Nov 8 20:28:20 ubuntu docker[23092]: time="2016-11-08T20:28:20.171385360Z" level=warning msg="2016/11/08 20:28:20 [ERR] memberlist: Failed TCP fallback ping: read tcp 10.0.0.85:54016->10.0.0.13:7946: i/o timeout"
Nov 9 18:26:17 swarm-node-01 docker[714]: time="2016-11-09T18:26:17.573441271Z" level=warning msg="2016/11/09 18:26:17 [ERR] memberlist: Failed to send indirect ping: write udp [::]:7946->10.0.0.38:7946: use of closed network connection"
For some reason docker refused to open this port and listen any more. Here is what I did (albeit undesirable) to circumvent the issue:
Created another node with docker-machine called swarm-master-02
Joined swarm-master-02 to the cluster as a master
Demoted master-01 which set master-02 as the leader
Restarted the docker daemon on each node (might not have been necessary)
Now all of the machines are working as expected except for swarm-master-01. One task is running on swarm-node-01 and curl works against all nodes by forwarding the traffic to the proper container on the proper node. However, swarm-master-01 refuses to listen on the overlay network and curl does not work against this node. I was only able to fix swarm-master-01 by completely removing it from the cluster, restarting the docker daemon, and joining it again as a master. Now 7946 is listening on that machine.
I installed docker on a new dedicated server (on a generic ubuntu 14.0 - linux kernel 3.13.0-71).
I installed an ubuntu docker image to test the environment. ( docker run -it ubuntu bash ) and I installed curl with openssl support.
When I try to get the content of an HTTP page, I have no problem. When I try to load an HTTPS page, my connection is refused:
root#835f01fef568:/# curl https://www.google.com
curl: (7) Failed to connect to www.google.com port 443: Connection refused
in verbose mode I have:
root#835f01fef568:/# curl -V https://www.google.com
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP
and if I try to log the trace in a file, I have:
== Info: Rebuilt URL to: https://www.google.com/
== Info: Hostname was NOT found in DNS cache
== Info: Trying 173.194.123.81...
== Info: connect to 173.194.123.81 port 443 failed: Connection refused
== Info: Trying 173.194.123.84...
== Info: connect to 173.194.123.84 port 443 failed: Connection refused
== Info: Trying 173.194.123.80...
== Info: connect to 173.194.123.80 port 443 failed: Connection refused
== Info: Trying 173.194.123.82...
== Info: connect to 173.194.123.82 port 443 failed: Connection refused
== Info: Trying 173.194.123.83...
== Info: connect to 173.194.123.83 port 443 failed: Connection refused
== Info: Trying 2607:f8b0:4006:80c::1013...
== Info: Immediate connect fail for 2607:f8b0:4006:80c::1013: Network is unreachable
== Info: Failed to connect to www.google.com port 443: Connection refused
== Info: Closing connection 0
I am a bit lost on what I can do :(
It is not a DNS problem since I can ping server or CURL http content on port 80. It only related to SSL connections.
Is there someone here with any idea about this issue?
Thanks
I found the source of the problem. Here it was related to an iptables issue of the main host
with the command iptables -L -t nat I discovered that there was a pre-routing activated on all https traffic redirected to the port 9092, used by another service.
I had the same problem. I found that setting the interface of the iptables rule to ‘eth0’ instead of ‘any’ solved the problem.
Here is an example that worked on the host for me:
iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 443 -j DNAT --to-destination 172.17.0.3:8443
Once the interface change to ‘eth0’ wget https://... worked again from within docker.
Hope this helps.
echo ipv4 >> ~/.curlrc
run this command on terminal. its working for me