I have attached to a docker container and need to find out the number of sockets being open by java application . Unfortunately there is no lsof or netstat available in the container . There is no data in /proc/PID/net/tcp. Is there any way I can find this data?
I like netshoot for this. You can run a container in the same networking and even pid namespace, and use the tools in netshoot to analyze the other container's network:
$ docker run -d -p 8888:80 --name nginx-test nginx
d8a90f5c7d1744483ae6d26cc97dad222ed237b5c4211f711c9f15f88252897f
$ docker run --net container:nginx-test --pid container:nginx-test -it --rm nicolaka/netshoot
/ # netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1/nginx: master pro
/ # ps -ef
PID USER TIME COMMAND
1 root 0:00 nginx: master process nginx -g daemon off;
7 104 0:00 nginx: worker process
8 root 0:00 sh
15 root 0:00 ps -ef
Alternatively, you can see this: /proc/PID/net/tcp in the host machine as long as you are in the same box as the docker daemon. This is less elegant than #BMitch's answer.
What you need to do is find out the PID of your process outside the container (in the main pid namespace, technically speaking, your host).
ps aux | grep java
Inside your container, your java has a pid; but outside it has another pid that you can use to access to the information that you have requested: /proc/PID/net/tcp
Related
I have encountered a strange behaviour. I have running grafana container with tcp/3000 exposed
# docker ps -a | grep grafana
26fa0fff9220 registry:4000/prometheus/grafana 0/tcp, 0.0.0.0:3000->3000/tcp grafana
# docker inspect grafana -f '{{json .NetworkSettings.Ports }}'
{"0/tcp":null,"3000/tcp":[{"HostIp":"0.0.0.0","HostPort":"3000"}]}
# ss -tunlp | grep 3000
tcp LISTEN 0 4096 0.0.0.0:3000 0.0.0.0:* users:(("docker-proxy",pid=3051799,fd=4))
The problem is that although curl works correctly for localhost it doesn't work when i am using the address of the host machine's network interface.
For example
//works fine
root#host-1# curl localhost:3000
Found.
//doesn't work at all (10.25.200.111 is eth-interface ip address on this same host), curl hangs indefinitely
root#host-1# curl 10.25.200.111:3000
^C
If i make a request from another host to 10.25.200.111:3000 it works without problems.
root#host-2# curl 10.25.200.111:3000
Found.
What am i doing wrong or maybe missing?
I'm trying to run
/usr/bin/docker run --rm -v /var/data/redis:/data -v /var/data/conf/redis.conf:/usr/local/etc/redis/redis.conf --name redis -p 6379:6379 redis:5.0.3-alpine3.9
but I get:
/usr/bin/docker: Error response from daemon: driver failed programming external connectivity on endpoint redis (f16f19b7727a710fb6c96be566dac66ce26282982960d97faa28861c24fcf2fb): Bind for 0.0.0.0:6379 failed: port is already allocated.
When I try to check the ports used with netstat, I get:
[root#artik ~]# netstat -nlpute | grep 6379
tcp6 0 0 :::6379 :::* LISTEN 0 14384 2471/docker-proxy
I have no docker containers running right now.
I don't understand this issue, what should I do ?
[root#artik ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Steps I had to take to get everything working:
sudo service docker stop
sudo rm /var/lib/docker/network/files/local-kv.db
sudo service docker start
docker system prune
And then try again.
From your netstat output its clear that there is one process holding port 6379
[root#artik ~]# netstat -nlpute | grep 6379
tcp6 0 0 :::6379 :::* LISTEN 0 14384 2471/docker-proxy
docker-proxy processes are created when you do port forwarding in docker run which is true in your case -p 6379:6379.
For more info on docker-proxy check this out.
I suspect that you earlier ran a redis container which used port 6379, but that container was not properly deleted which kept process docker-proxy running and hence you got port is already allocated
Hope this helps.
As DannyMoshe suggested for anyone else.
Try this before you potentially mess up your whole setup::
sudo service docker stop
sudo service docker start
remove the ports - ... in the docker-compose file and let it assign by itself. or change the port mapping in the host from 6379:6379 to 6378:6379 that worked for me. Before doing this you may need to clear already started containers. docker rm -f $(docker ps -a -q)
I have started an nginx container bound on the host network as follows:
docker run --rm -d --network host --name mynginx nginx
However, when querying process information with the ss command, this seems to be a pure nginx but not a docker process:
$ ss -tuap 'sport = :80'
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp LISTEN 0 128 0.0.0.0:http 0.0.0.0:* users:(("nginx",pid=16563,fd=6),("nginx",pid=16524,fd=6))
why is that?
You configured the nginx process to run in the host networking namespace --net host. In that mode you do not setup port forwarding from the host to the container network (e.g. -p 80:80). Had you done the port forwarding, you would see a docker process on the host which is forwarding to the same port in the container namespace for the nginx process.
Keep in mind that containers are a method to run an application with kernel options for things like namespacing, it is not a VM running under a separate OS, so you will see processes running and ports opened directly on the host.
Here's an example of what it would look like if you forwarded the port instead of using the host network namespace, and how you can also look at the network namespace inside the container:
$ docker run --rm -d -p 8000:80 --name mynginx nginx
d177bc43166ad59f5cdf578eca819737635c43b2204b2f75f2ba54dd5a9cffbb
$ sudo ss -tuap 'sport = :8000'
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp LISTEN 0 128 :::8000 :::* users:(("docker-proxy",pid=25229,fd=4))
$ docker run -it --rm --net container:mynginx --pid container:mynginx nicolaka/netshoot ss -tuap 'sport = :80'
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp LISTEN 0 128 *:http *:* users:(("nginx",pid=1,fd=6))
The docker-proxy process there is the default way that docker forwards a port to the container.
I am afraid there is some misunderstanding here about so-called docker process.
First of all, ss command doesn’t show what kind of process it is. It may show the application name(nginx here). But we could not say it’s so-called pure nginx process.
You could try pwdx nginx_pid. Otherwise, each running container is a process which we could check with ps -ef on its host machine.
Above all, you could use ps -ef|grep nginx and pwdx nginx_pid to find out what kind of process it is.
I'm getting this strange error, when I try to run a docker with a name it gives me this error.
docker: Error response from daemon: service endpoint with name qc.T8 already exists.
However, there is no container with this name.
> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
> sudo docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 3
Server Version: 1.12.3
Storage Driver: aufs
Root Dir: /ahdee/docker/aufs
Backing Filesystem: extfs
Dirs: 28
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null bridge host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-101-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 64
Total Memory: 480.3 GiB
Is there anyway I can flush this out?
Just in case someone else needs this. As #Jmons pointed out it was a weird networking issue. So I solved this by forcing a removal
docker network disconnect --force bridge qc.T8
A
TLDR: restart your docker daemon or restart your docker-machine (if you're using that e.g. on a mac).
Edit: As there are more recent posts below, they answer the question better then mine. The Network adapter is stuck on the daemon. I'm updating mine as its possibly 'on top' of the list and people might not scroll down.
Restarting your docker daemon / docker service / docker-machine is the easiest answer.
the better answer (via Shalabh Negi):
docker network inspect <network name>
docker network disconnect <network name> <container id/ container name>
This is also faster in real time if you can find the network as restarting the docker machine/demon/service in my experience is a slow thing. If you use that, please scroll down and click +1 on their answer.
So the problem is probably your network adapter (virtual, docker thing, not real): have a quick peek at this: https://github.com/moby/moby/issues/23302.
To prevent it happening again is a bit tricky. It seems there may be an issue with docker where a container exits with a bad status code (e.g. non-zero) that holds the network open. You can't then start a new container with that endpoint.
docker network inspect <network name>
docker network disconnect <network name> <container id/ container name>
You can also try doing:
docker network prune
docker volume prune
docker system prune
these commands will help clearing zombie containers, volume and network.
When no command works then do
sudo service docker restart
your problem will be solved
docker network rm <network name>
Worked for me
Restarting docker solved it for me.
I created a script a while back, I think this should help people working with swarm. Using docker-machine this can help a bit.
https://gist.github.com/lcamilo15/7aaaebe71852444ea8f1da5c4c9c84b7
declare -a NODE_NAMES=("node_01", "node_02");
declare -a CONTAINER_NAMES=("container_a", "container_b");
declare -a NETWORK_NAMES=("network_1", "network_2");
for x in "${NODE_NAMES[#]}"; do;
docker-machine env $x;
eval $(docker-machine env $x)
for CONTAINER_NAME in "${CONTAINER_NAMES[#]}"; do;
for NETWORK_NAME in "${NETWORK_NAMES[#]}"; do;
echo "Disconnecting $CONTAINER_NAME from $NETWORK_NAME"
docker network disconnect -f $NETWORK_NAME $CONTAINER_NAME;
done;
done;
done;
You could try seeing if there's any network with that container name by running:
docker network ls
If there is, copy the network id then go on to remove it by running:
docker network rm network-id
This could be because an abrupt removal of a container may leave the network open for that endpoint (container-name).
Try stopping the container first before removing it.
docker stop <container-name>. Then docker rm <container-name>.
Then docker run <same-container-name>.
i think restart docker deamon will solve the problem
Even reboot did not help in my case. It turned out that port 80 to be assigned by the nginx container automatically was in use, even after reboot. How come?
root#IONOS_2: /root/2_proxy # netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:873 0.0.0.0:* LISTEN 1378/rsync
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 1565/systemd-resolv
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1463/nginx: master
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1742/sshd
tcp6 0 0 :::2377 :::* LISTEN 24139/dockerd
tcp6 0 0 :::873 :::* LISTEN 1378/rsync
tcp6 0 0 :::7946 :::* LISTEN 24139/dockerd
tcp6 0 0 :::5355 :::* LISTEN 1565/systemd-resolv
tcp6 0 0 :::21 :::* LISTEN 1447/vsftpd
tcp6 0 0 :::22 :::* LISTEN 1742/sshd
tcp6 0 0 :::5000 :::* LISTEN 24139/dockerd
No idea what nginx: master means or where it came from. And indeed 1463 is the PID:
root#IONOS_2: /root/2_proxy # ps aux | grep "nginx"
root 1463 0.0 0.0 43296 908 ? Ss 00:53 0:00 nginx: master process /usr/sbin/nginx
root 1464 0.0 0.0 74280 4568 ? S 00:53 0:00 nginx: worker process
root 30422 0.0 0.0 12108 1060 pts/0 S+ 01:23 0:00 grep --color=auto nginx
So I tried this:
root#IONOS_2: /root/2_proxy # kill 1463
root#IONOS_2: /root/2_proxy # ps aux | grep "nginx"
root 30783 0.0 0.0 12108 980 pts/0 S+ 01:24 0:00 grep --color=auto nginx
And the problem was gone.
I have a docker image in which I start jstatd -p 1099 and then my Java app. I also expose port 1099 in the dockerfile.
I have deployed this docker image to AWS ElasticBeanstalk and I can see from the EB logs that the port is exposed.
/var/log/docker-ps.log
-------------------------------------
'docker ps' ran at Fri Jun 17 04:23:02 UTC 2016:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d3199a65e216 8b9c53bb10b6 "/app/run.sh" 5 minutes ago Up 5 minutes 1099/tcp, 8080/tcp jolly_carson
I would now like to profile the app using VisualVM. How can I find the correct ip to connect to? Attempts to telnet to the app's domain name on port 1099 time out.
The container's port is not bound to the instance's port, which is good because you don't want to expose your debugging interface publicly. The IP address of the container can be found with:
$ sudo docker ps
$ sudo docker inspect --format '{{ .NetworkSettings.IPAddress }}' <container_id>
Start an SSH tunnel that tunnels from port 5005 locally to that IP address and port 5005 on the box.
$ ssh ec2-user#ec2-54-204-111-222.compute-1.amazonaws.com -L 5005:<ip>:5005 -N
or you can configure port forwarding over iptables
$ sudo docker inspect --format '{{ .NetworkSettings.IPAddress }}' stupefied_swartz
172.17.0.2
$ sudo iptables -t nat -A PREROUTING -p tcp --dport 5005 -j REDIRECT --to-destination 172.17.0.2:5005