Remote debugging JVM in docker on AWS EB - docker

I have a docker image in which I start jstatd -p 1099 and then my Java app. I also expose port 1099 in the dockerfile.
I have deployed this docker image to AWS ElasticBeanstalk and I can see from the EB logs that the port is exposed.
/var/log/docker-ps.log
-------------------------------------
'docker ps' ran at Fri Jun 17 04:23:02 UTC 2016:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d3199a65e216 8b9c53bb10b6 "/app/run.sh" 5 minutes ago Up 5 minutes 1099/tcp, 8080/tcp jolly_carson
I would now like to profile the app using VisualVM. How can I find the correct ip to connect to? Attempts to telnet to the app's domain name on port 1099 time out.

The container's port is not bound to the instance's port, which is good because you don't want to expose your debugging interface publicly. The IP address of the container can be found with:
$ sudo docker ps
$ sudo docker inspect --format '{{ .NetworkSettings.IPAddress }}' <container_id>
Start an SSH tunnel that tunnels from port 5005 locally to that IP address and port 5005 on the box.
$ ssh ec2-user#ec2-54-204-111-222.compute-1.amazonaws.com -L 5005:<ip>:5005 -N

or you can configure port forwarding over iptables
$ sudo docker inspect --format '{{ .NetworkSettings.IPAddress }}' stupefied_swartz
172.17.0.2
$ sudo iptables -t nat -A PREROUTING -p tcp --dport 5005 -j REDIRECT --to-destination 172.17.0.2:5005

Related

Can't connect to Docker container from local machine

I have encountered a strange behaviour. I have running grafana container with tcp/3000 exposed
# docker ps -a | grep grafana
26fa0fff9220 registry:4000/prometheus/grafana 0/tcp, 0.0.0.0:3000->3000/tcp grafana
# docker inspect grafana -f '{{json .NetworkSettings.Ports }}'
{"0/tcp":null,"3000/tcp":[{"HostIp":"0.0.0.0","HostPort":"3000"}]}
# ss -tunlp | grep 3000
tcp LISTEN 0 4096 0.0.0.0:3000 0.0.0.0:* users:(("docker-proxy",pid=3051799,fd=4))
The problem is that although curl works correctly for localhost it doesn't work when i am using the address of the host machine's network interface.
For example
//works fine
root#host-1# curl localhost:3000
Found.
//doesn't work at all (10.25.200.111 is eth-interface ip address on this same host), curl hangs indefinitely
root#host-1# curl 10.25.200.111:3000
^C
If i make a request from another host to 10.25.200.111:3000 it works without problems.
root#host-2# curl 10.25.200.111:3000
Found.
What am i doing wrong or maybe missing?

Is it possible to bind on another IP than 127.0.0.1 on MacOS?

On MacOS 12, using Docker 20.10, I'm not able to start a container on another IP:
% docker run -p 127.123.2.13:80:80 -d nginx
a9216ae29940f7357b9b4826ecddf041f1805c9ee48ba1336361277fc0dcb524
docker: Error response from daemon: Ports are not available: listen tcp 127.0.17.1:80: bind: can't assign requested address.
Is there any other way?
In order to bind to the ip, other than 0.0.0.0, you need to have an interface in your system with the desired ip. For example, watch docker failing to bind to a non-existent ip of 127.0.0.2:
docker run -p 127.0.0.2:80:80 -d nginx
cc79b1b60c9f5e245b326bbfcc17d4a1f1abe6fad6fd12f9677b66bbee972a12
docker: Error response from daemon: Ports are not available: listen tcp 127.0.0.2:80: bind: can't assign requested address.
Now I create an alias for my existing interface lo0:
sudo ifconfig lo0 alias 127.0.0.2 netmask 0xff000000
and try again:
docker run -p 127.0.0.2:80:80 -d nginx
05223ecb6ae99a25b7423f014b9b95422c621717705ce1c255bea04072c45263
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cc79b1b60c9f nginx "/docker-entrypoint.…" 2 minutes ago Created hardcore_haslett
05223ecb6ae9 nginx "/docker-entrypoint.…" 2 minutes ago Up 2 minutes 127.0.0.2:80->80/tcp pensive_bardeen

driver failed programming external connectivity on endpoint redis : Bind for 0.0.0.0:6379 failed: port is already allocated

I'm trying to run
/usr/bin/docker run --rm -v /var/data/redis:/data -v /var/data/conf/redis.conf:/usr/local/etc/redis/redis.conf --name redis -p 6379:6379 redis:5.0.3-alpine3.9
but I get:
/usr/bin/docker: Error response from daemon: driver failed programming external connectivity on endpoint redis (f16f19b7727a710fb6c96be566dac66ce26282982960d97faa28861c24fcf2fb): Bind for 0.0.0.0:6379 failed: port is already allocated.
When I try to check the ports used with netstat, I get:
[root#artik ~]# netstat -nlpute | grep 6379
tcp6 0 0 :::6379 :::* LISTEN 0 14384 2471/docker-proxy
I have no docker containers running right now.
I don't understand this issue, what should I do ?
[root#artik ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Steps I had to take to get everything working:
sudo service docker stop
sudo rm /var/lib/docker/network/files/local-kv.db
sudo service docker start
docker system prune
And then try again.
From your netstat output its clear that there is one process holding port 6379
[root#artik ~]# netstat -nlpute | grep 6379
tcp6 0 0 :::6379 :::* LISTEN 0 14384 2471/docker-proxy
docker-proxy processes are created when you do port forwarding in docker run which is true in your case -p 6379:6379.
For more info on docker-proxy check this out.
I suspect that you earlier ran a redis container which used port 6379, but that container was not properly deleted which kept process docker-proxy running and hence you got port is already allocated
Hope this helps.
As DannyMoshe suggested for anyone else.
Try this before you potentially mess up your whole setup::
sudo service docker stop
sudo service docker start
remove the ports - ... in the docker-compose file and let it assign by itself. or change the port mapping in the host from 6379:6379 to 6378:6379 that worked for me. Before doing this you may need to clear already started containers. docker rm -f $(docker ps -a -q)

Find open sockets in docker container

I have attached to a docker container and need to find out the number of sockets being open by java application . Unfortunately there is no lsof or netstat available in the container . There is no data in /proc/PID/net/tcp. Is there any way I can find this data?
I like netshoot for this. You can run a container in the same networking and even pid namespace, and use the tools in netshoot to analyze the other container's network:
$ docker run -d -p 8888:80 --name nginx-test nginx
d8a90f5c7d1744483ae6d26cc97dad222ed237b5c4211f711c9f15f88252897f
$ docker run --net container:nginx-test --pid container:nginx-test -it --rm nicolaka/netshoot
/ # netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1/nginx: master pro
/ # ps -ef
PID USER TIME COMMAND
1 root 0:00 nginx: master process nginx -g daemon off;
7 104 0:00 nginx: worker process
8 root 0:00 sh
15 root 0:00 ps -ef
Alternatively, you can see this: /proc/PID/net/tcp in the host machine as long as you are in the same box as the docker daemon. This is less elegant than #BMitch's answer.
What you need to do is find out the PID of your process outside the container (in the main pid namespace, technically speaking, your host).
ps aux | grep java
Inside your container, your java has a pid; but outside it has another pid that you can use to access to the information that you have requested: /proc/PID/net/tcp

Open docker daemon container to outside

The docker daemon container is isolated from outside when we run it below,
$ docker run -d --name test_container ubuntu/ping \
/bin/sh -c "while true do echo hello world; sleep 1; done"
$ docker inspect test_container | grep IPAddress
[ip of test_container]
$ ping [ip of test_container]
[timeout]
$ ifconfig docker0 | grep "inet addr"
[ip of docker bridge]
$ ping [ip of docker bridge]
[ok]
$ docker exec -it test_container /bin/bash
# ping [ip of test_container]
[ok]
# ping [ip of docker bridge]
[timeout]
How to open the ip address of the docker daemon container inside out?
By default docker daemon is running on a unix socket
You can enable to listen on tcp socket by doing :
docker daemon -H tcp://validIpOnYourHost:port
By default port is 2375 if you do not provide some.
cf this page for more explanation : https://docs.docker.com/v1.11/engine/reference/commandline/daemon/
Be careful, if you expose docker throught TCP, this is not security enabled.
Probably I replied to something else, after reading your question :
could you do a :
docker network inspect bridge
and paste the json output.
I had similar issues when the attribute :
"com.docker.network.bridge.enable_ip_masquerade"
was set to false

Resources