I run a https web service from a docker container set up on Vbox. Here is my config:
Vbox
Docker
Unfortunatelly, https://127.0.0.1 is not accessible.
The output of the command docker run -it --rm --net=container:$cont_id --pid=container:$cont_id busybox netstat -lntp is:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 21/sshd
tcp 0 0 127.0.0.1:444 0.0.0.0:* LISTEN 319/node
tcp 0 0 127.0.0.1:8081 0.0.0.0:* LISTEN 315/python
tcp 0 0 :::22 :::* LISTEN 21/sshd
tcp 0 0 :::443 :::* LISTEN 319/node
I can't figure out where I'm getting wrong (I am still a beginner in port forwarding and networking). Any help appreciated, Thanks!
Related
Fairly new to Docker. Our containers work fine when hitting localhost with curl or a browser, but any external calls to http://[ip address] just time out. We're seeing the exact same behavior with Kong and also a basic whoami. The only way the containers are externally accessible is when we add --network host to the docker run command, but that's not an option for our production use.
The server itself and firewall are configured correctly; when I shut down docker and spun up a simple webserver it was reachable at the IP address. Essentially, any bridge-type network for Docker is inaccessible to the outside world and produces time-outs on any call to a port we set it to listen for (vs immediate connection refused for random unmapped ports).
The run commands we're using:
docker run -d -p 80:80 containous/whoami
docker run -d --name kongtest \
-p 0.0.0.0:80:8000 -p 0.0.0.0:443:8443 \
kong/kong-gateway:3.0.0.0-alpine
Output from docker ps:
88a4bf28bbcd kong/kong-gateway:3.0.0.0-alpine "/docker-entrypoint.…" 5 seconds ago Up 5 seconds (health: starting) 8001-8004/tcp, 8444-8447/tcp, 0.0.0.0:80->8000/tcp, 0.0.0.0:443->8443/tcp kongtest
netstat -lntup using default or custom Docker bridge network:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 384272/sshd: /usr/s
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 640858/docker-proxy
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 384297/systemd-reso
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 640845/docker-proxy
tcp6 0 0 :::22 :::* LISTEN 384272/sshd: /usr/s
udp 0 0 127.0.0.53:53 0.0.0.0:* 384297/systemd-reso
udp 0 0 140.82.10.213:68 0.0.0.0:* 384291/systemd-netw
netstat -lntup using --network=host:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 384272/sshd: /usr/s
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 384297/systemd-reso
tcp6 0 0 :::22 :::* LISTEN 384272/sshd: /usr/s
tcp6 0 0 :::80 :::* LISTEN 708481/whoami
udp 0 0 127.0.0.53:53 0.0.0.0:* 384297/systemd-reso
udp 0 0 140.82.10.213:68 0.0.0.0:* 384291/systemd-netw
I am running a network server under the jamq user in Docker.
[root#12af450e8259 /]# su jamq -c '/opt/jboss-amq-7-i0/bin/artemis-service start'
Starting artemis-service
artemis-service is now running (25)
I am then trying to list processes and their listening sockets using netstat as root, but for processes running as different user than me, I only see - instead of PID.
[root#12af450e8259 /]# netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1/sshd
tcp 0 0 0.0.0.0:1883 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8161 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5445 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:61613 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:61616 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN 1/sshd
I tried adding --privileged onto Docker command line, and that fixes the problem. I then wanted to use more granular capabilities, but I cannot find the right capability.
I tried
docker run --rm --cap-add=SYS_ADMIN --cap-add=NET_ADMIN -it myimage:latest bash
but that does not help.
The required capability is --cap-add=SYS_PTRACE. There are various reports in bugs that netstat needs this capability. For example, Bug 901754 - SELinux is preventing /usr/bin/netstat from using the 'sys_ptrace' capabilities.
The correct command therefore is
docker run --rm --cap-add=SYS_PTRACE -it myimage:latest bash
[root#f9c4b5fa7d1c /]# netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN 22/java
tcp 0 0 0.0.0.0:61613 0.0.0.0:* LISTEN 22/java
tcp 0 0 0.0.0.0:61616 0.0.0.0:* LISTEN 22/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 92/sshd
tcp 0 0 0.0.0.0:1883 0.0.0.0:* LISTEN 22/java
tcp 0 0 127.0.0.1:8161 0.0.0.0:* LISTEN 22/java
tcp 0 0 0.0.0.0:5445 0.0.0.0:* LISTEN 22/java
tcp6 0 0 :::22 :::* LISTEN 92/sshd
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name Path
I have following setup
Two VMs
created overlay network
created two docker swarm services
docker service create --name karaf1-service --replicas 1 --network karaf_net karaf1:2.0.0
docker service create --name karaf2-service --replicas 1 --network karaf_net karaf2:2.0.0
Now these containers open socket port at start, i observed some time it successfully able to create it lot of time it fails.
ServerSocketFactory.getDefault().createServerSocket(serverPort)
if both containers get start on one node its mostly successfull, but when containers get created on different node it almost fails every time.
before troubleshooting for network issue, container atleast should create sockets.
this container not able to open socket
root#bd48643080b2:/opt/apache/apache-karaf-4.1.5# netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8101 0.0.0.0:* LISTEN 61/java
tcp 0 0 127.0.0.1:1099 0.0.0.0:* LISTEN 61/java
tcp 0 0 0.0.0.0:41551 0.0.0.0:* LISTEN 61/java
tcp 0 0 127.0.0.11:44853 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:44444 0.0.0.0:* LISTEN 61/java
Following container able to create it on port 4550, but some times it vice versa
root#38d26c7dde1a:/opt/apache/apache-karaf-4.1.5# netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:37347 0.0.0.0:* LISTEN 61/java
tcp 0 0 0.0.0.0:8101 0.0.0.0:* LISTEN 61/java
tcp 0 0 0.0.0.0:4550 0.0.0.0:* LISTEN 61/java
tcp 0 0 127.0.0.11:37575 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:1099 0.0.0.0:* LISTEN 61/java
tcp 0 0 127.0.0.1:35321 0.0.0.0:* LISTEN 61/java
tcp 0 0 0.0.0.0:44444 0.0.0.0:* LISTEN 61/java
Root Cause Identified:
As i am creating two services so while creating first service i provide second service as hostname to first service to keep verifying status so java throwing error on hostname like "karaf2-service"
java.net.UnknownHostException: karaf2-service: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
Now i cant add entry of karaf2-service in etc/hosts so socket dont complain as i dont know which IP would be assign to docker-swarm service? in overlay network we mostly communicate with service names.
Any suggestions to resolve this???
The easiest way to do this, is to check on container startup if you can reach the other service, and if not, wait a few seconds then try again.
There are multiple tools to do this, such as wait-for-it: https://github.com/vishnubob/wait-for-it
I am trying to run a Docker image from inside Google Cloud Shell (i.e. on an courtesy Google Compute Engine instance) as follows:
docker run -d -p 20000-30000:10000-20000 -it <image-id> bash -c bash
Previous to this step, netstat -tuapn has reported the following:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:8998 0.0.0.0:* LISTEN 249/python
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:13080 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:13081 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:34490 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:13082 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:13083 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:13084 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:34490 127.0.0.1:48161 ESTABLISHED -
tcp 0 252 172.17.0.2:22 173.194.92.34:49424 ESTABLISHED -
tcp 0 0 127.0.0.1:48161 127.0.0.1:34490 ESTABLISHED 15784/python
tcp6 0 0 :::22 :::* LISTEN -
So it looks to me as if all the ports between 20000 and 30000 are available, but the run is nevertheless terminated with the following error message:
Error response from daemon: Cannot start container :
failed to create endpoint on network bridge: Timed out
proxy starting the userland proxy
What's going on here? How can I obtain more diagnostic information and ultimately solve the problem (i.e. get my Docker image to run with the whole port range available).
Opening up ports in a range doesn't currently scale well in Docker. The above will result in 10,000 docker-proxy processes being spawned to support each port, including all the file descriptors needed to support all those processes, plus a long list of firewall rules being added. At some point, you'll hit a resource limit on either file descriptors or processes. See issue 11185 on github for more details.
The only workaround when running on a host you control is to not allocate the ports and manually update the firewall rules. Not sure that's even an option with GCE. Best solution will be to redesign your requirements to keep the port range small. The last option is to bypass the bridge network entirely and run on the host network where there are no more proxies and firewall rules with --net=host. The later removes any network isolation you have in the container, so tends to be recommended against.
I got the docker gitlab for testing on mac but I can't access the UI.
$ docker pull sameersbn/gitlab:8.7.3
$ docker-compose up -d
Running docker exec -i -t dockersandbox_gitlab_1 netstat -tulpn gives
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 453/nginx -g daemon
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 455/sshd
tcp6 0 0 :::80 :::* LISTEN 453/nginx -g daemon
tcp6 0 0 :::22 :::* LISTEN 455/sshd
Running docker port dockersandbox_gitlab_1 gives me:
22/tcp -> 0.0.0.0:10022
80/tcp -> 0.0.0.0:10080