Can't connect to docker after resuming VM - docker

Question

For some reason whenever I suspend my VM and resume it, I can no longer connect to the docker container that is hosted within the VM. Usually, I pass -p 3000:3000 to the docker container so that I can access the rails instance within it and this works fine, but when I suspend the VM and resume it later, I can no longer connect to port 3000 even though it's listening within the docker image.
This results in me having to reboot the VM as service docker restart does not change anything.
Is there something else I should be looking at to resolve this issue? I've been suspending/resuming my VM with docker in it for quite awhile and have never run into this issue before.
EDIT
To reproduce this issue, I simply resumed my VM and tried connecting to localhost port 3000 from the VM itself (not within the docker image) and it cannot connect. However, below shows that port 3000 is listening:
[root:kali:~/app]# curl http://localhost:3000
curl: (56) Recv failure: Connection reset by peer
[root:kali:~/app]# netstat -antp | grep -i listen
tcp 0 0 127.0.0.1:43050 0.0.0.0:* LISTEN 84770/autossh
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 20478/sshd
tcp6 0 0 :::3000 :::* LISTEN 32731/docker-proxy
tcp6 0 0 :::3001 :::* LISTEN 32715/docker-proxy
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 :::22 :::* LISTEN 20478/sshd
From within docker, I can see that rails is working:
[root:77f444beafff:~/app]# rails s --binding 0.0.0.0
=> Booting Puma
=> Rails 5.2.3 application starting in development
=> Run `rails server -h` for more startup options
Puma starting in single mode...
* Version 3.12.1 (ruby 2.5.1-p57), codename: Llamas in Pajamas
* Min threads: 5, max threads: 5
* Environment: development
* Listening on tcp://0.0.0.0:3000
Use Ctrl-C to stop
And here's the netstat from within docker:
[root:77f444beafff:~/app]# netstat -antp | grep -i listen
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 478/redis-server *:
tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN 765/puma 3.12.1 (tc
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN -
tcp6 0 0 :::6379
If I curl from within the docker image, I can see it hits the rails app just fine:
[root:77f444beafff:~/app]# curl http://localhost:3000/ -I
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
Content-Type: text/html; charset=utf-8
ETag: W/"5078d30a6c1a5f6fc5cb7f9a82cd89f5"
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: _vspm_session=Cace%2FN0zB%2F6QJOiietbuHxTHOMZUMuRmEukYqQTNaHQ91hskaN%2BPJzev0KdGUAAtYx9a35Mqdkr8eRkPdH4qOl6vOaCcPU0gy8s7IMfkb9VhRGPPbecepmI%2F9leA2dnD694P8ctXSBklOCnjhN0%3D--SglWrWvx3BFEAI3z--IkylACdXbR6eF27Hgn0Cgg%3D%3D; path=/; HttpOnly
X-Request-Id: 29aa7251-f29a-4309-adec-6af479e7bd9b
X-Runtime: 12.241723

Answer 1

I'm having exactly the same issue with my VMWare virtual machine (VMWare running on Windows).
The only workaround that is working for me is:
docker stop $(docker ps -aq) && sudo systemctl restart NetworkManager docker
If I had to guess I would say it may be related to some firewall rules docker setup on start, maybe when you resume the virtual machine a change in the network configuration breaks those rules.
Similar issue: https://github.com/docker/for-mac/issues/1990 (Doesn't seem specific to docker for mac).

Answer 2

I was able to solve this issue with the hint given by lannox in the comment. It's necessary to mark the network interfaces of the docker containers as unmanaged by NetworkManager.
To do that, create a new file /etc/NetworkManager/conf.d/10-unmanage-docker-interfaces.conf with the following content:
[keyfile]
unmanaged-devices=interface-name:docker*;interface-name:veth*;interface-name:br-*;interface-name:vmnet*;interface-name:vboxnet*
This configures NetworkManager to ignore all interfaces with names docker*,
veth*,
br-*,
vmnet*, and
vboxnet* interfaces.
Then restart NetworkManager with sudo systemctl restart NetworkManager.
Next time the host suspends and resumes, the docker containers keep their network connectivity.

Answer 3

Several questions here that might help you solve this :
Is your docker container still running? Run docker ps and find your container
Since the -p 3000:3000 option is set I guess the port is exposed, but you might want to check you really have run your container with this option this time
Is your app really listening? Run lsof -np | grep listen and find your app listening on port 3000
Connect to your container with docker exec -it <your_container> bash and try running lsof -np | grep listen to see if this is a docker issue or your app

Answer 4

It seems that when you run netstat on your VM you get the following line :
tcp6 0 0 :::3000 :::* LISTEN 32731/docker-proxy
On Docker you get :
tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN 765/puma 3.12.1 (tc
There are two differences here:
:::3000 vs 0 0.0.0.0:3000, the first means it is listening on IPv6 and the second on IPv4 (found the info on this question).
tcp6 vs tcp, again IPv6 vs IPv4.
According to this other question, it seems you have to run rails with -b :: option.
The -b option binds Rails to the specified IP, by default it is localhost. You can run a server as a daemon by passing a -d option.

Answer 5

Please do
sudo docker ps
If you do not got your container do
sudo docker ps -a
Does your container is stopped?
If its true so do
sudo docker start CONTAINER_ID