I have been running a media cluster for sometime without any issues. I have everything networked into two different docker networks... the first network just bridges the docker instance to the local machine, the second network is a docker VPN container that I use for the other media services (an earlier version of what I am working on can be found here: https://github.com/Xander-Rudolph/MediaDocker)
The strangest thing happened today though. I ran the docker update for windows and now docker spools up without any errors or issues, however none of the services work outside of the machine running docker. Usually I have a poke through for a couple of the services in my router (namely wordpress/joomla which is on the bridge) and they work outside of my local network, but none of them are working anymore. I was able to confirm its not the DNS A record because I'm able to use the RDP ports I have mapped for my router, and when I test on another machine in the same network, it can't access the services via the internal IP (but it can RDP).
Anyone have any idea what could have changed to break this? I've already updated all my docker images and even rebuilt my VPN container (before I realized its a networking issue). What are some steps I can do to try to troubleshoot what is going wrong in docker to prevent access outside of localhost?
Update
I've been able to rule out the docker update as the root cause... I upgraded docker on my laptop (which was previously running the same version as my desktop) and its not having the same issue... this configuration must be localized to this desktop... No idea what the issue is... Will try a linux VM on the desktop instead of docker for windows...
Update 2
After a lot of screwing around in both a VM and in WSL, I'm still only able to access the docker services from localhost but not a different machine on my network or via the IP on the host machine (perhaps something similar to this: Can't access localhost via IP address). RDP does work so the computer is accessible but the services are not.
I'm not sure if this is a result of a docker networking config or a windows network config (I'm using WSL with docker installed on ubuntu 20.08) but I'm not seeing anything stick out. I'm going to remove the tag for docker windows but this is definitely an issue with networking and I suspect it has something to do with the fact that the containers are running behind a VPN... although I don't know why I would be able to access them on localhost but not the IP on another VM...
When I run
netstat -a -o
on WSL I can see the established ports on localhost... EX:
tcp 0 0 localhost:7878 localhost:37520 ESTABLISHED
but when I look on the host machine (for wsl) I don't see the connection. I tried to use netsh to create a firewall rule to see if that would help:
netsh advfirewall firewall add rule name="TCP Port 7878" dir=in localport=7878 protocol=TCP action=allow
but it didn't have any effect.
Any suggestions for ways to trace the network to see where/how its failing/getting blocked would be extremely helpful.
Your question: "...What are some steps I can do to try to troubleshoot what is going wrong in docker to prevent access outside of localhost?..."
Troubleshooting help for you, first do you have multiple networking adapters (Ethernet, Wi-Fi, etc.) present on the host. First ensure, the priority of these adapters needs to be configured in correct order so the Windows networking stack can correctly choose gateway routes.
Now, to fix this set your primary internet-connected networking adapter to have the lowest InterfaceMetric value, use can use these Powershell commands from an elevated console:
Get-NetIPInterface -AddressFamily IPv4 | Sort-Object -Property InterfaceMetric -Descending
Please ensure that the host's primary internet-connected network adapter has the lowest InterfaceMetric value.
// Use this command to make the change for e.g. lets say your
// primary adapter InterfaceAlias is 'Wi-Fi'
Set-NetIPInterface -InterfaceAlias 'Wi-Fi' -InterfaceMetric 3
Now step two, if your host's primary network adapter is bridged because you have an External virtual switch setup in Hyper-V, then you will set the external virtual switch to have the lowest InterfaceMetric value.
Lastly, confirm/verify your routing tables, when you run this, the last line should show the primary adapter's gateway address along with it's ifMetric value):
Get-NetRoute -AddressFamily IPv4
If you’re using Docker Toolbox then any port you publish with docker run -p will be published on the Toolbox VM’s private IP address.
docker-machine ip will tell you.
It is frequently
192.168.99.100
Taken from: https://forums.docker.com/t/cant-connect-to-container-on-localhost-with-port-mapping/52716/25
After several attempts using the references below, I was still not getting anywhere. The recommendation by #derple didn't get me anywhere (since I was in wsl) but the article he linked someone had said they switched to linux and uninstalled and reinstalled docker desktop... and for some stupid reason that works.
These are my exact steps I took to fix it:
Uninstall docker desktop
Install WSL and docker inside an ubuntu18.04 instance in wsl
Test docker in wsl with localhost (worked only on localhost still)
Uninstall WSL using windows add/remove features
reinstall docker desktop
Oddly the get-netipinterface and get-netroute look exactly the same as they did before I did the uninstall and reinstall but things seem to be working now... I have no idea why the above worked...
Related
I have a couple docker containers running on my local machine (pgadmin, jupyter notebooks etc) and have them mapped to various ports. I can happily navigate to localhost:10100 to get to the pgadmin web interface, for example. The issue is that when I connect to the work VPN I am unable to get to any locally running containers. I get an "ERR_CONNECTION_RESET" error on chrome.
With the VPN on I've tried:
localhost:10100 (also tried 127.0.0.1)
my-hostname:10100
192.168.0.X:10100 (the wifi interface address)
192.168.19.X:10100 (the VPN TUN interface address)
I can ping any of the above addresses and get a response and can successfully use them when the VPN is disabled. Using PulseVPN, Ubuntu 21.10, and fairly recent docker/docker-compose if that helps.
You can try to run the containers with the host network by adding the flag:
--network host
to the end of each command when you first start the containers.
And if that does not work, you can try it with:
--network none
Turns out that are a combination of issues that are causing problems. I haven't found a bullet proof solution yet but here are some breadcrumbs for someone else:
The default docker network subnet was overlapping with my work subnet.
The VPN route was set to have the lowest cost, therefor all traffic is being routed through it.
Changing the default subnet resulted in the containers working, for around 5 minutes. Then the low cost routing was discovered and my traffic went through there instead.
My guess is that I have to fiddle with my network routing so that the docker networks are separated from the work VPN. It's been a decade since my CCNA so I can't remember how to do this off hand...
I have a VM running Ubuntu 16.04, on which I want to deploy an application packaged as a Docker container. The application needs to be able to perform an HTTP request towards a server under VPN (e.g. server1.vpn-remote.com)
I successfully configured the host VM in order to connect to the VPN through openconnect, I can turn this connection on/off using a systemd service.
Unfortunately, when I run docker run mycontainer, neither the host nor the container are able to reach server1.vpn-remote.com. Weirdly enough, there is no error displayed in the VPN connection service logs, which is stuck to the openconnect messages confirming a successful connection.
If I restart the VPN connection after starting mycontainer, the host machine is able to access server1.vpn-remote.com, but not the container. Moreover, if I issue any command like docker run/start/stop/restart on mycontainer or any other container, the connection gets broken again even for the host machine.
NOTE: I already checked on the ip routes and there seems to be no conflict between Docker and VPN subnets.
NOTE: running the container with --net="host" results in both host and container being able to access the VPN but I would like to avoid this option as I will eventually make a docker compose deployment which requires all containers to run in bridge mode.
Thanks in advance for your help
EDIT: I figured out it is a DNS issue, as I'm able to ping the IP corresponding to server1.vpn-remote.com even after the VPN connection seemed to be failing. I'm going through documentation regarding DNS management with Docker and Docker Compose and their usage of the host's /etc/resolv.conf file.
I hope you don't still need help six months later! Some of the details are different, but this sounds a bit like a problem I had. In my case the solution was a bit disappointing: after you've connected to your VPN, restart the docker daemon:
sudo systemctl restart docker
I'm making some inferences here, but it seems that, when the daemon starts, it makes some decisions/configs based on the state of the network at that time. In my case, the daemon starts when I boot up. Unsurprisingly, when I boot up, I haven't had a chance to connect to the VPN yet. As a result, my container traffic, including DNS lookups, goes through my network directly.
Hat tip to this answer for guiding me down the correct path.
I cannot connect to the published port on the swarm that uses overlay networking. I am using Docker for Windows with Windows containers. Both Windows and Docker are fully upgraded. After Windows' 1709 update, I was hoping this issue would be resolved. I looked for information on the Internet to see if I was doing something wrong to no avail. I would like to know if anyone was successfully able to get it working.
On a side note, when I direct the port on my machine in docker run -p 80:80 without using swarm, "localhost" does not work as well. I think this is a known limitation though. Both issues work when I switch to Linux containers.
Expected behavior
I am running a dotnet kestrel web server service. I should be able to connect to my service using the published port.
Actual behavior
Firefox gives me timeout, opera straight away returns connection refused. Cannot telnet into it either. Container IP's assigned by the overlay network do not work either.
Information
docker service ls gives me this:
Ports cannot be seen there, is it because publish mode is host? Ports information is available in the output of docker service ps
And when I change the publish mode, I can scale it as well and the port information is seen in docker service ls albeit still cannot connect. the one below is without the publish mode=host parameter:
For more info, this is the output of the docker network ls I wonder if i need some sort of bridge network like in Linux.
Steps to reproduce the behavior
Initialise swarm
Start the service, in my case: a simple web service built using aspnetcore:latest image. I tried different parameters, even used a docker-stack.yml:
docker service create --name=web --publish mode=host,published=80,target=80 web:aspnetcorelatest in the case above, I was unable to scale it on the same machine, which is normal i guess
docker service create --name=web --publish published=85,target=80 web:aspnetcorelatest
Try to connect using one of http://localhost or another IP. I tried connecting over VPN, from another machine as well as Internet IP.
I was creating a container using microsoft/windowsservercore image. And then when I tried to ping google.com from inside the container, I got this error:
Ping request could not find host www.google.com. Please check the name
and try again.
Then I switched to Linux Container mode in docker for windows. Then tried the same in an ubuntu container but this time it worked fine. Then when I switched back to Windows Container mode and tried the same thing again, it worked this time. Although my issue was resolved but I still don't understand what caused this issue in the first place ?
Docker for windows and linux have different default network settings.
Typically, the default for linux is bridged mode while in windows you have NAT.
You can alter your configuration with Network Connection Settings for windows
See: https://docs.docker.com/docker-for-windows/#network
The first option for me is always to look at the network section when executing docker inspect *containername*. This command gives you information about your network settings for the container. Other options are to check your firewall settings.
In general I usually use ping 8.8.8.8 since www.google.com cannot be pinged even from my standard windows machine.
Currently, I'm working on a project where we are using ROS, Bebop_autonomy, and OpenCV to control a Parrot Bebop2 autonomously. The machines we use in the workspace are running Ubuntu 14.04.5, and I can start a container using an image
I created with "docker run -it --network=host username/image". After configuring everything inside the container, the bebop_autonomy node is fine and can communicate on the Bebop's network perfectly. When you run ip addr in both the container and host machine, they show the same address, as you'd expect.
However, when trying to run it on my Windows machine, the ip is different than the host machine, and I'm never receiving any ACK packets when I try to communicate with the Bebop. I'm assuming this is because the packets aren't being sent to the right ip, or they aren't being forwarded correctly.
I have tried creating my own network and setting the ip manually with "docker network create" and passing it in to the run command as an argument, but I can't seem to get it to work at all. I've also tried creating different switches in the Hyper-V manager, but nothing I've read in the last few days has helped me figure this out.
I've got a good handle on how docker works, but most of the reference material I see is talking about a host that already runs linux. If I can't figure this out, it's almost useless for us to continue with docker in the first place.
Is there any way to configure Docker for Windows to work in the same way that Docker works on Linux when providing --network=host?
I ended up achieving what I wanted by creating a separate network in Hyper-V manager, setting that network to only use an external wifi-adapter, and running the container using that network. There has to be a better way though.