Overlay driver not being applied when creating docker swarm network - docker

Question

I'm working with docker swarm and trying to create a network with the overlay driver.
Whenever I create the network, the driver is not attached.
If I try and attach a service to the network, the process just hangs infinitely.
If I create a service without attaching it to the network, it works right away.
pi#node3:~ $ docker network ls
NETWORK ID NAME DRIVER SCOPE
a1cc2e1f4f2b bridge bridge local
83597f713bcf docker_gwbridge bridge local
277f1166485e host host local
fs2vvjeuejxc ingress overlay swarm
5d0ce08c744c none null local
pi#node3:~ $ docker network create --driver overlay test
4bfkahhkhrblod2t79yd83vws
pi#node3:~ $ docker network ls
NETWORK ID NAME DRIVER SCOPE
a1cc2e1f4f2b bridge bridge local
83597f713bcf docker_gwbridge bridge local
277f1166485e host host local
fs2vvjeuejxc ingress overlay swarm
5d0ce08c744c none null local
4bfkahhkhrbl test swarm
I can't figure out why it's not adding the driver. I have a suspicion it has something to do with the ingress network settings, but I'm stuck as for troubleshooting here.
Relevant Info
Swarm:
pi#node3:~ $ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
ygcte2diochpbgu7bqtw41k70 node1 Ready Active 20.10.7
xbllxgfa35937rmvdi8mi8dlb node2 Ready Active 20.10.7
tvw4b53w6g3qv2k3919dg3a81 * node3 Ready Active Leader 20.10.7
Manager node:
pi#node3:~ $ docker node inspect node3
[
{
"ID": "tvw4b53w6g3qv2k3919dg3a81",
"Version": {
"Index": 165
},
"CreatedAt": "2021-07-10T16:41:23.043334654Z",
"UpdatedAt": "2021-07-11T00:27:25.807737662Z",
"Spec": {
"Labels": {},
"Role": "manager",
"Availability": "active"
},
"Description": {
"Hostname": "node3",
"Platform": {
"Architecture": "armv7l",
"OS": "linux"
},
"Resources": {
"NanoCPUs": 4000000000,
"MemoryBytes": 969105408
},
"Engine": {
"EngineVersion": "20.10.7",
"Plugins": [
{
"Type": "Log",
"Name": "awslogs"
},
{
"Type": "Log",
"Name": "fluentd"
},
{
"Type": "Log",
"Name": "gcplogs"
},
{
"Type": "Log",
"Name": "gelf"
},
{
"Type": "Log",
"Name": "journald"
},
{
"Type": "Log",
"Name": "json-file"
},
{
"Type": "Log",
"Name": "local"
},
{
"Type": "Log",
"Name": "logentries"
},
{
"Type": "Log",
"Name": "splunk"
},
{
"Type": "Log",
"Name": "syslog"
},
{
"Type": "Network",
"Name": "bridge"
},
{
"Type": "Network",
"Name": "host"
},
{
"Type": "Network",
"Name": "ipvlan"
},
{
"Type": "Network",
"Name": "macvlan"
},
{
"Type": "Network",
"Name": "null"
},
{
"Type": "Network",
"Name": "overlay"
},
{
"Type": "Volume",
"Name": "local"
}
]
},
"TLSInfo": {
"TrustRoot": "-----BEGIN CERTIFICATE-----\nMIIBajCCARCgAwIBAgIUFIx3NAw+jgaasNXCoi+QP4GxaOQwCgYIKoZIzj0EAwIw\nEzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMjEwNzEwMTYyMjAwWhcNNDEwNzA1MTYy\nMjAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH\nA0IABKyunnrZtfkOO+Cc/MX/qbyJjG12ee8es0IHB1HXF2MhqSfYOeUuBlTvuHuB\nxl8s8eQ4IMfjP0w5LYJNqypZp0KjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB\nAf8EBTADAQH/MB0GA1UdDgQWBBRq6yBEIFv03tQqBkohCh4A+mIZdTAKBggqhkjO\nPQQDAgNIADBFAiA5kKgC2WxcOMyfrmFr8fU6w1Mo1mq5GMKA4owTB7pcEQIhALZi\n9AH0vVyR+7NmmR7LfPO65CIJ9UVuPZBXRZ6pcmzX\n-----END CERTIFICATE-----\n",
"CertIssuerSubject": "MBMxETAPBgNVBAMTCHN3YXJtLWNh",
"CertIssuerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAErK6eetm1+Q474Jz8xf+pvImMbXZ57x6zQgcHUdcXYyGpJ9g55S4GVO+4e4HGXyzx5Dggx+M/TDktgk2rKlmnQg=="
}
},
"Status": {
"State": "ready",
"Addr": "0.0.0.0"
},
"ManagerStatus": {
"Leader": true,
"Reachability": "reachable",
"Addr": "10.0.0.93:2377"
}
}
Ingress network:
pi#node3:~ $ docker network inspect ingress
[
{
"Name": "ingress",
"Id": "fs2vvjeuejxcjxqivenb76kgj",
"Created": "2021-07-10T17:24:14.228552858-07:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.10.0.0/24",
"Gateway": "10.10.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": true,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "34003d042d395b90328ed90c8133505a6bec6df90065c5b47b47ee3853545c91",
"MacAddress": "02:42:0a:0a:00:02",
"IPv4Address": "10.10.0.2/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4096"
},
"Labels": {},
"Peers": [
{
"Name": "e2f4d4e6ba20",
"IP": "10.0.0.93"
},
{
"Name": "de3d98ce0f8d",
"IP": "10.0.0.25"
},
{
"Name": "b61722e30756",
"IP": "10.0.0.12"
}
]
}
]
Docker version:
pi#node3:~ $ docker --version
Docker version 20.10.7, build f0df350
Docker info:
pi#node3:~ $ docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 5
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: tvw4b53w6g3qv2k3919dg3a81
Is Manager: true
ClusterID: 4vf16jdlegf3ctys5k6wumcfc
Managers: 1
Nodes: 3
Default Address Pool: 10.10.0.0/24
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.0.0.93
Manager Addresses:
10.0.0.93:2377
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.17-v7+
Operating System: Raspbian GNU/Linux 10 (buster)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 924.2MiB
Name: node3
ID: A67O:SIT4:QOMH:SILY:WHAY:KSGQ:VWMF:QVEJ:VCOZ:KW32:PZRV:ZD4B
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
What I've tried:
Removing all the nodes and creating a new swarm
Removing the ingress network and creating a new one following the instructions here
Tried to go through the walkthrough here but can't get past Create the Services 2.
Rebooted all the nodes
Any advice or pointing in the right direction would be much appreciated! I've been stuck here for 48 hours.

Answer 1

Solved!
The issue ended up being:
The nodes were all on 10.0.0.x
I set the --default-addr-pool 10.10.0.0/24 when initializing the swarm
Any network I tried to create using the --driver overlay would end up without any subnet or gateway information.
How I resolved the issue:
I was able to solve it using the --subnet flag when creating a custom network.
pi#node3:~ $ docker network create --driver overlay --subnet 10.10.10.0/24 test
pi#node3:~ $ docker network ls
NETWORK ID NAME DRIVER SCOPE
55ab64773261 bridge bridge local
ce1a0f497e9d docker_gwbridge bridge local
7c85cac72cf8 host host local
o7iew29j70nl ingress overlay swarm
ca5fc5682911 none null local
plezwc8zahpl test overlay swarm
pi#node3:~ $ docker network inspect test
[
{
"Name": "test",
"Id": "plezwc8zahpl9gs8hbv64bbo3",
"Created": "2021-07-16T17:30:28.773110478Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.10.10.0/24",
"Gateway": "10.10.10.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": null,
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": null
}
]