When I look at all running processes on my Linux machine, there are quite a few docker-proxy processes. It seems like every running container (port) results in one docker-proxy!
Problem is I cannot find any documentation which processes docker actually starts and how their relationship/usage is.
Does anyone know if there is any documentation on that?
A full explanation of the docker-proxy is available here.
The summary is that the proxy is used to handle connections originating from the local machine that might otherwise not pass through the iptables rules that Docker configures to handle port forwarding, or when Docker has been configured such that it does not manipulate iptables at all.
Related
I have a DDev project in WSL2. Whenever I try to start it I get an error:
Error response from daemon: Ports are not available: exposing port TCP 127.0.0.1:443 -> 0.0.0.0:0: listen tcp 127.0.0.1:443: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.'
Sometimes it's also port 80. But most importantly before starting the project none of those ports is occupied. Neither inside WSL nor on the Windows Host. I am also able to start another docker container exposing on those ports. I am even to manually start the router with
COMPOSE_PROJECT_NAME=ddev-project docker-compose -f /home/crs/.ddev/.router-compose-full.yaml -p ddev-router up -d
but I still can't access the project even though the router seems to be running.
ddev debug test also fails.
I tried updating and reinstalling both Docker Desktop and ddev.
I also tried changing the router_http_port and router_https_port to something else. Then it does seem to start the project but I still can't access anything through the ddev router.
The web containers seem to work fine, when not going through the router I can access the project.
Debugging for this is explained in the docs, but it's slightly trickier on WSL2, because the process that's giving trouble may be either on the Windows side or the WSL2 side.
As explained there, you can either find the competing process or change to use different ports in DDEV.
On WSL2, port 80 is often apache2, which some distros have by default, so you can stop it or uninstall it without any harm. Port 443 is something occupied by random poorly behaved processes on Windows, including sometimes virus checkers.
If you use the techniques there to check for competing ports you'll almost certainly solve this.
Another technique is to use curl localhost, curl -I localhost or curl https://localhost and curl -I https://localhost to see if the HTTP response gives you a clue what process is problematic.
Also note that sometimes Docker Desktop is poorly behaved if you're using it, and you may have to restart it.
But if changing the ports to, say, 8080 and 8443 didn't solve it for you then you have a connectivity problem, likely a firewall. That's a completely different problem and you'll want to walk through the troubleshooting instructions in docs and start with temporarily turning off firewall and VPN.
For more interactive help, join us in the DDEV Discord.
I've been wondering why docker installation does not enable by default port forwarding to containers.
To save you a click, what I mean is:
$ sysctl net.ipv4.conf.all.forwarding=1
$ sudo iptables -P FORWARD ACCEPT
I assume it is some sort of security risk, but I just wonder what the risk it is.
Basically I want to create some piece of code that enables this by default, but I want to know what is the bad that can happen.
I googled this and couldn't find anything.
Generally FORWARD ACCEPT seems to be considered too permissive (?)
If so, what can I change to make this more secure?
My network is rather simple, it is a bunch of pcs in a local lan (10.0.0.0/24) with an openvpn server and those pcs may deploy docker hosts (I'm doing this by hand, not using docker compose or swarm or anything because nodes change) that need to see each other. So no real outside access. Another detail is that I am not using network overlay which I could do without swarm, but the writer of the post warns it could be deprecated soon, so also wonder if I should just start using docker-swarm straight away.
EDIT: My question here is maybe more theoretical I guess than what it may seem at first. I want to know why they decided not to do this. I pretty much need/want full communication between docker instances, they need to be ssh'd into and open up a bunch of different ports to talk to each other (and this is the limitation of my networking knowledge, I don't know how this really works, I suppose they are all high ports, but are those also blocked by docker?). I am not sure docker-swarm would help me much here either. They aimed at micro-services I maybe need interactive sessions from time to time, but this is probably asking too much in a single question.
Maybe the simplest version of this question is: "if I put that code up there as a script to load each time my computer boots up, how can someone abuse it".
Each docker container runs on a local bridge network with IPs generally in the range of 172.1x.xx.xx. You can get the ip address running:
docker inspect <container name> | jq -r ".[].NetworkSettings.Networks[].IPAddress"
You should either run your container exposing and publishing the specific container ports on the host running the containers.
Alternatively, you can use iptables to redirect traffic to a specific port from outside:
iptables -t nat -I PREROUTING -i <incoming interface> -p tcp -m tcp --dport <host listening port> --j DNAT --to-destination <container ip address>:<container port>
Change tcp for udp if the port is listening on a udp socket.
If you want to redirect all traffic you can still use the same approach, but may need to specify a secondary ip address on your host (e.g., 192.168.1.x) and redirect any traffic coming to that address to your container.
I am struggling with a problem which is best described on a picture.
I need somehow log all urls requested when I run docker build command.
Can somebody help me how to achieve this?
You cannot monitor the connections called by build command. They are just normal connections pass through your network interface.
You may want to install tcpdump and use it to monitor a specific network interface or to filter specific HTTP requests. That is the best what you can do as far as I know.
UPDATE
If you want to monitor build process which happens inside docker container, you may use tcpdump as mentioned above to monitor the network interface of that Docker container using its binding IP address. By that you can see the connections flows in and out that Docker container only.
It is really easy to mount directories into a docker container. How can I just as easily "mount a port into" a docker container?
Example:
I have a MySQL server running on my local machine. To connect to it from a docker container I can mount the mysql.sock socket file into the container. But let's say for some reason (like intending to run a MySQL slave instance) I cannot use mysql.sock to connect and need to use TCP.
How can I accomplish this most easily?
Things to consider:
I may be running Docker natively if I'm using Linux, but I may also be running it in a VM if I'm on Mac or Windows, through Docker Machine or Docker for Mac/Windows (Beta). The answer should handle both scenarios seamlessly, without me as the user having to decide which solution is right depending on my specific Docker setup.
Simply assigning the container to the host network is often not an option, so that's unfortunately not a proper solution.
Potential solution directions:
1) I understand that setting up proper local DNS and making the Docker container (network) talk to it might be a proper, robust solution. If there is such a DNS service that can be set up with 1, max 2 commands and then "just work", that might be something.
2) Essentially what's needed here is that something will listen on a port inside the container and like a sort of proxy route traffic between the TCP/IP participants. There's been discussion on this closed Docker GH issue that shows some ip route command-line magic, but that's a bit too much of a requirement for many people, myself included. But if there was something akin to this that was fully automated while understanding Docker and, again, possible to get up and running with 1-2 commands, that'd be an acceptable solution.
I think you can run your container with --net=host option. In this case container will bind to the host's network and will be able to access all the ports on your local machine.
I am writing an application that is composed of a few spring boot based microservices with a zuul based reverse proxy in the front-
It works when I start the services on my machine, but for server rollout I'd like to use docker for the services, but this seems not to be possible right now.
Normally you would have a fixed "internal" port and randomized ports at the outside of the container. But the app in the container doesn't know the outside port (and IP).
The Netflix tools match what I would want to write an efficient microservice architecture and conceptually I really like docker.
As far as I can see it would be very troublesome to start the container, gather the outside port on the host and pass it to the app, because you can't simply change the port after the app is started.
Is there any way to use eureka with docker based clients?
[Update]
I guess I did a poor job explaining the problem. So maybe this clarifies it a bit more:
The eureka server itself can run in docker, as I have only one and the outside port doesn't matter. I can use the link feature to access it from the clients.
The problem is the URL that the clients register themselves with.
This is for example https://localhost:8080/ but due to dynamic port assignment it is really only accessible via https://localhost:54321/
So eureka will return the wrong URL for the services.
UPDATE
I have updated my answer below, so have a look there.
I have found a solution myself, which is maybe not the best solution, but it fits for me...
When you start docker with "--net=host" (host networking), then you use the hosts network stack directly. Then I just use 0 as port for spring-boot and spring randomizes the port for me and as it's using the hosts networking stack there is no translation to a different port (and IP).
There are some drawbacks though:
When you use host networking you can't use the link-feature for these containers as link source or target.
Using the hosts network stack leads to less encapsulation of the instance, which maybe a problem depending on your project.
I hope it helps
A lot of time has passed and I think I should elaborate this a little bit further:
If you use docker to host your spring application, just don't use a random port! Use a fixed port because every container gets his own IP anyway so every service can use the same port. This makes life a lot easier.
If you have a public facing service then you would use a fixed port anyway.
For local starts via maven or for example the command line have a dedicated profile that uses randomized ports so you don't have conflicts (but be aware that there are or have been a few bugs surrounding random ports and service registration)
if for whatever reason you want to or need to use host networking you can use randomized ports of course, but most of the time you shouldn't!
You can set up a directory for each docker instance and share it between the host and the instance and then write the port and IP address to a file in that directory.
$ instanceName=$(generate random instance name)
$ dirName=/var/lib/docker/metadata/$instanceName
$ mkdir -p $dirName
$ docker run -name $instanceName -v ${dirName}:/mnt/metadata ...
$ echo $(get port number and host IP) > ${dirName}/external-address
Then you just read /mnt/metadata/external-address from your application and use that information with Eureka.