docker-compose not setting gateway and IP address - docker

Question

I have a problem where docker-compose containers aren't able to reach the internet. Manually created containers via the docker cli or kubelet work just fine.
This is on an AWS EC2 node created using Kops with Calico overlay (I think that may be unrelated, however).
Here's the docker-compose:
version: '2.1'
services:
app:
container_name: app
image: "debian:jessie"
command: ["sleep", "99999999"]
app2:
container_name: app2
image: "debian:jessie"
command: ["sleep", "99999999"]
This fails:
# docker exec -it app ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
docker-compose container<->container works (as expected):
# docker exec -it app ping app2
PING app2 (172.19.0.2): 56 data bytes
64 bytes from 172.19.0.2: icmp_seq=0 ttl=64 time=0.098 ms
Manually created container works fine:
# docker run -it -d --name app3 debian:jessie sh -c "sleep 99999999"
# docker exec -it app3 ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=37 time=9.972 ms
So it seems like docker-compose containers can't reach the internet.
Here's the NetworkSettings from app3, which works:
"NetworkSettings": {
"Bridge": "",
"SandboxID": "54168ea912b9caa842b208f36dac80a588ebdc63501a700379fb1b732a41d3ac",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/54168ea912b9",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "cdddee0f3e25e7861a98ba6aff33652619a3970c061d0ed2a5dc5bd2b075b30d",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:02",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "46e8bc586d48c9a57e2886f7f35f7c2c8396f8084650fcc2bf1e74788df09e3f",
"EndpointID": "cdddee0f3e25e7861a98ba6aff33652619a3970c061d0ed2a5dc5bd2b075b30d",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:02"
}
}
}
From one of the docker-compose containers (fails):
"NetworkSettings": {
"Bridge": "",
"SandboxID": "6b79a6b45f099c65f89adf59eb50eadff2362942f316b05cf20ae1959ca9b88b",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/6b79a6b45f09",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"root_default": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"app2",
"4f48647ba5bb"
],
"NetworkID": "ffb540b2b9e2945908477a755a43d3505aea6ed94ef5fd944909a91fb104ce8e",
"EndpointID": "48aff2f00bb4bd670b5178b459a353ac45f7d3efbfb013c1026064022e7c4e59",
"Gateway": "172.19.0.1",
"IPAddress": "172.19.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:13:00:02"
}
}
}
So it seems like the major difference is that the docker-compose containers aren't created with an IPAddress or Gateway.
Some background info:
# docker version
Client:
Version: 1.12.6
API version: 1.24
Go version: go1.6.4
Git commit: 78d1802
Built: Tue Jan 10 20:17:57 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Go version: go1.6.4
Git commit: 78d1802
Built: Tue Jan 10 20:17:57 2017
OS/Arch: linux/amd64
# docker-compose version
docker-compose version 1.15.0, build e12f3b9
docker-py version: 2.4.2
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.1t 3 May 2016
# ip route
default via 10.20.128.1 dev eth0
10.20.128.0/20 dev eth0 proto kernel scope link src 10.20.140.184
100.104.10.64/26 via 10.20.136.0 dev eth0 proto bird
100.109.150.192/26 via 10.20.152.115 dev tunl0 proto bird onlink
100.111.225.192 dev calic6f21d462fc scope link
blackhole 100.111.225.192/26 proto bird
100.111.225.193 dev calief8dddb6a0d scope link
100.111.225.195 dev cali8ca1dd867c3 scope link
100.111.225.196 dev cali34426885f86 scope link
100.111.225.197 dev cali6cae60de42a scope link
100.111.225.231 dev calibd569acd2f3 scope link
100.115.17.64/26 via 10.20.148.89 dev tunl0 proto bird onlink
100.115.237.64/26 via 10.20.167.9 dev tunl0 proto bird onlink
100.117.246.128/26 via 10.20.150.249 dev tunl0 proto bird onlink
100.118.80.0/26 via 10.20.162.215 dev tunl0 proto bird onlink
100.119.204.0/26 via 10.20.135.183 dev eth0 proto bird
100.123.178.128/26 via 10.20.170.43 dev tunl0 proto bird onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.18.0.0/16 dev br-bd6445b00ccf proto kernel scope link src 172.18.0.1
172.19.0.0/16 dev br-ffb540b2b9e2 proto kernel scope link src 172.19.0.1
iptables are a bit long, so not posting for now (I would expect them to interfere with the non-docker-compose generated containers, so I think the iptables are unrelated).
Anyone know what's going on?