My docker swarm has an API service that uses Nginx proxy to access Report service. Nginx proxy only doing proxy_pass and running in 2 replicas. I have problem accessing Nginx proxy by service name or service IP. Sometimes it works, but mainly i get this response:
# curl 'http://nginx-proxy:8000/v1/report?id=662867'
curl: (7) Couldn't connect to server
# curl 'http://10.0.17.13:8000/v1/report?id=662867'
curl: (7) Couldn't connect to server
On other hand if i access individual replicas by an IP or from outside docker network - it works just fine:
# curl 'http://10.0.17.14:8000/v1/report?id=662867'
0 662867 0 10 6 6 0.0 194 3 5 437
# curl 'http://10.0.17.18:8000/v1/report?id=662867'
0 662867 0 10 6 6 0.0 194 3 5 437
Feels like internal docker balancer gets overwhelmed by amount of requests and stops accepting new connections. There is no errors in nginx logs - every request with 200 status. But API logs show this:
INFO Cannot get online report: Get http://nginx-proxy:8000/v1/report?id=732743: dial tcp 10.0.17.13:8000: connect: cannot assign requested address caller=/go/src/api/src/reader.go:300 (*ReaderCursor).readOnline
INFO Cannot get online report: Get http://nginx-proxy:8000/v1/report?id=732703: dial tcp 10.0.17.13:8000: connect: cannot assign requested address caller=/go/src/api/src/reader.go:300 (*ReaderCursor).readOnline
I'm using official Nginx image, only modified worker_processes auto; from 1 to auto.
Any ideas what can be wrong or where to look?
Problem wasn't in docker at all. It was in an API code, which was running multiple processes with default 2 connections per post. Lots of requests used up all local ports to use for the client connection.
Fixed probelm with http.DefaultTransport.(*http.Transport).MaxIdleConnsPerHost = 100
Related
I'm trying to run a docker image on my windows 10 pro workstation, and I'm getting this error:
docker: Error response from daemon: Ports are not available: listen tcp 0.0.0.0:80: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
I'm running this command:
docker run -d -p 80:80 docker/getting-started
And getting this response back:
Unable to find image 'docker/getting-started:latest' locally
latest: Pulling from docker/getting-started
188c0c94c7c5: Pull complete
617561f33ec6: Pull complete
7d856acdaa9c: Pull complete
a0d3c6e28e6d: Pull complete
af69a9b963c8: Pull complete
0739f3815ad8: Pull complete
7c7b75d0baf8: Pull complete
Digest: sha256:b821569034e3b5fae03b40e64a866017067f3bf17effe185b782bdbf02179528
Status: Downloaded newer image for docker/getting-started:latest
7907f6de2b55cc2d66b5ed3a642ac1a97e5bb5ecda5fcf76ff60d7236e8fd32d
docker: Error response from daemon: Ports are not available: listen tcp 0.0.0.0:80: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
How can I run a Docker container and get past this problem?
The commands that helped me were:
net stop winnat
net start winnat
Taken from here.
Check if somthing is listening on port 80 by running PowerShell command:
Get-Process -Id (Get-NetTCPConnection -LocalPort 80).OwningProcess
If the response is something like this:
NPM(K) PM(M) WS(M) CPU(s) Id SI ProcessName
------ ----- ----- ------ -- -- -----------
0 0.20 4.87 0.00 4 0 System
Then the it is taken.
terminate the process or simply change the port
docker run -d -p 81:81 docker/getting-started 5a0b1202f48ef63c06d75c2f26be2a05f29aa84fe2fbdc5b66f989aa86df98f
docker: Error response from daemon: Ports are not available: listen tcp 0.0.0.0:80: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
I was also using Docker's Getting Started tutorial, ran
docker run -d -p 80:80 docker/getting-started, and got the same error.
I thought that Internet Information Services (IIS) might be using port 80 already. I verified this hunch by going to
Start > IIS > computer name > Sites > Default Web Site > highlight Default Web Site
In the right pane, I saw
Browse *.80 (http)
Yep, port 80 was already in use!
Note: To verify if any process is using port 80, run the command
Get-Process -Id (Get-NetTCPConnection -LocalPort 80).OwningProcess
and look for something like
NPM(K) PM(M) WS(M) CPU(s) Id SI ProcessName
------ ----- ----- ------ -- -- -----------
0 0.20 2.96 0.00 4 0 System
per #Tim Dunphy's answer.
Solution #1
Stop the Default Web site.
Either go to IIS per above > right pane > Manage Website > Stop or
Use the net stop http command, and stop services using the port.
Solution #2
You may not want to stop IIS or any other service running on port 80, so just change the local port! For example,
docker run -d -p 81:80 docker/getting-started
will likely do the trick, as long as you don't have services using port 81.
When I encountered this error there was nothing actively listening on the port.However, the following command showed the port had been reserved by something else
netsh interface ipv4 show excludedportrange protocol=tcp
See This article for more details. A reboot fixed the issue for me.
Most times, Windows IIS (Internet Information Service) or some other program may be using port 80, which is the default http port used by Laravel, Apache and other PHP development environments.
To resolve this issue, Open a new PowerShell window as administrator and simply run this command:
net stop http
A prompt listing all services using the http port is shown and you are given the option of stopping them. enter 'Y' and press Enter. All the services are stopped and port 80 is now free for whatever you want to use it for.
Very useful for testing multiple application environments locally on Windows without having to worry about port configuration.
in my case,the task manager show that a system process is occupy port 80.
when i dive deeper, i found a svchost.exe related to port 80, and it is based on world wide web service
so,just open service list and stop the world wide web service,then everything will be ok,the name of service maybe different, but should include keywords :world wide web
just do it
netsh http add iplisten ipaddress=::
(This has to be run in a command prompt with elevated rights!)
I'm running a setup of 3 Ubuntu virtual machines. Two running the Python production code base and the other has a Memcached Docker container. On the Memcached machine I ran docker run -dit --name production-memcached --publish 11211:11211 memcached:latest.
The code base gets the following error message when trying to interact with it:
"exception": "TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None)"
I have ran docker exec -it production-memcached memcached stats and get the error message below.
failed to listen on TCP port 11211: Address already in use
However I've ran netstat -plnt and get tcp6 0 0 :::11211 :::* LISTEN 35030/docker-proxy, which looks fine to me.
I was able to get this to work by opening port 80 and using iptables to forward incoming port 80 to port 11211 but would prefer to use the Memcached port number.
The Memcached client is created by the following line:
client = base.Client(("domain.co.uk", 11211))
Any help would be appreciated, thanks.
DigitalOcean doesn't allow in-traffic on port 11211 to its virtual machines. If you want a Memcached machine you'll also need a virtual machine to act a proxy between you and it. I hope this saves someone the headache it caused me!
I have a docker container running Caddy, and another web server.
In my Dockerfile I have
EXPOSE 80 10240 # 10240 is the port of the other webserver.
And I run docker like this (don't ask my why you need EXPOSE and -p).
docker run -p 80:80 -p 10240:1024 -it <hash>
This starts the two servers. On my host machine (it's a Mac btw) I can connect to localhost:10240 fine. However if I connect to localhost:80, I get an empty response (dropped connection).
Netstat in the docker container shows:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:10240 0.0.0.0:* LISTEN 19/node
tcp6 0 0 :::80 :::* LISTEN 9/caddy
Here's where it starts to get weird. If I curl -L localhost within the docker container, it works fine - I get the web page from the 10240 server.
If I curl -L 127.0.0.1 from within the docker container it returns 404 Site 127.0.0.1 is not served on this interface. Ok fine.
If I curl -L 127.0.0.1 from outside the container it also returns 404 Site 127.0.0.1 is not served on this interface. So somehow my requests are getting through, but Caddy drops localhost requests from outside the container, and it doesn't from inside it. I have logging enabled but it prints nothing.
Can anyone tell me what the hell is going on? All this docker port forwarding stuff is ridiculous.
Here's my Caddyfile (and I've tried about a billion other combinations of localhost, 127.0.0.1, etc.):
localhost:80
bind 0.0.0.0
proxy / 127.0.0.1:10240
I am not sure but I suspect this was because Docker for Mac's networking is kind of broken. I gave up on Caddy and tried to do the same think with Traefik, which also didn't work (though it gave a "Gateway error" instead of totally dropping the connection).
As soon as moved everything to Linux, it worked perfectly.
I am using a Dockerfile to hit our corporate Nexus (npm) server for 'npm install' commands. I am seeing:
* Hostname was NOT found in DNS cache
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 216.xxx.xxx.xxx...
* connect to 216.xxx.xxx.xxx port 443 failed: Connection refused
* Failed to connect to nexus.<something>.com port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to nexus.<something>.com port 443: Connection refused
I can resolve www.google.com. I can hit and and use our corporate NPM registery from my local box. It appears that only our internal dns names are the problem and only when attempting to access them from inside a docker container. I've googled and not been able to determine the changes I need to make to fix this problem.
Dockerfile (I've trimmed the irrelevant commands):
FROM node:6.3
RUN curl -k -v https://www.google.com
RUN curl -k -vv https://nexus.<something>.com/repository/npm-all/
The curl to google.com succeeds. The curl to our internal repo fails.
I am starting it with the command:
docker build .
Contents of /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
I am running Ubuntu 15.10.
Solution
#BMitch is correct. Modify the contents of /etc/resolv.conf by adding dns server addresses associated with your corporate network. In the case of Ubuntu 15+ (i am running gnome3) your config file will be overwritten by the Network Manager so it cannot be hand edited. Make the changes via the Network Manager gui. Open Network Settings, select the DNS tab and add servers.
The local dns address is the issue you're facing. The container can't connect to your localhost ip from inside the container. The solution is to pass an ip address of the DNS server in your docker run or update your /etc/resolv.conf to point to a non-loopback ip address.
From Docker's DNS documentation:
Filtering is necessary because all localhost addresses on the host are
unreachable from the container’s network. After this filtering, if
there are no more nameserver entries left in the container’s
/etc/resolv.conf file, the daemon adds public Google DNS nameservers
(8.8.8.8 and 8.8.4.4) to the container’s DNS configuration. If IPv6 is
enabled on the daemon, the public IPv6 Google DNS nameservers will
also be added (2001:4860:4860::8888 and 2001:4860:4860::8844).
Note: If you need access to a host’s localhost resolver, you must modify your DNS service on the host to listen on a non-localhost
address that is reachable from within the container.
I have multiple Flask based webapps running in docker containers (their processes need to be isolated from the host OS). To run these apps I use uWSGI servers inside the containers. Incoming requests should hit a uWSGI FastRouter with a subscription server (as described here: http://uwsgi-docs.readthedocs.org/en/latest/SubscriptionServer.html). When starting a container, the uWSGI should announce itself based on some internal configuration as a subdomain.
So the setup looks like this:
Request ---> FastRouter ----> container | myapp1 |
|
----> container | myapp2 |
I'm trying to test this on a single host running both the fast router as well as some docker containers.
The FastRouter is started using
uwsgi --fastrouter :1717 --fastrouter-subscription-server localhost:2626 --fastrouter-subscription-slot 1000
Question 1 Do I need to do anything else to get the subscription server running? Is this started together with the fastrouter process?
The containers have two ports mapped from the host to the container: 5000 (the webapp) and 2626 (to subscribe to the fast router).
So they're started like this:
docker run -d -p 5000:5000 -p 2626:2626 myImage $PATH_TO_START/start.sh
Where in start.sh the uWSGI is started as
uwsgi --http :5000 -M --subscribe-to 127.0.0.1:2626:/test --module server --callable env --enable-threads
The output looks good, it prints at the end:
spawned uWSGI master process (pid: 58)
spawned uWSGI worker 1 (pid: 73, cores: 1)
spawned uWSGI http 1 (pid: 74)
subscribing to 127.0.0.1:2626:/test
On the host I can do
curl localhost:5001
And I see the Webserver greeting me from inside the container. However, doing
curl localhost:1717/test
gets no response.
Question 2
Am I getting anything fundamentally wrong here? Should I test differently?
Question 3
How can I debug the FastRouter?
Edit:
Still struggling with this setup. I'm using a separate VPS now for the fastrouter. It is started using
uwsgi --uid fastrouter --master --fastrouter :80 --fastrouter-subscription-server :2626 --daemonize uwsgi.log --pidfile ./uwsgi.pid --chmod-socket --chown-socket fastrouter
WARNING: Think before copying above call for your project since it opens up the subscription service publicly - my plan is to secure it afterwards using the key signing facilities provided by uwsgi since the VPS doesn't have a private network available.
netstat -anp shows
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 843/uwsgi
udp 0 0 0.0.0.0:2626 0.0.0.0:* 843/uwsgi
unix 3 [ ] STREAM CONNECTED 9089 843/uwsgi
unix 3 [ ] STREAM CONNECTED 9090 843/uwsgi
unix 3 [ ] SEQPACKET CONNECTED 8764 843/uwsgi
unix 3 [ ] SEQPACKET CONNECTED 8763 843/uwsgi
Anyway, using uwsgi nodes with --http :5000 --module server --callable env --enable-threads --subscribe-to [Subscription-Server-IP-Address]2626:/test --socket-timeout 120 --listen 128 --buffer-size 32768 --honour-stdin still leads to the same result - uwsgi logs 'subscribing to', but http://[Subscription-Server-IP-Address]/test is not reachable. Is this kind of routing even possible? Every example I can find only assigns subdomains like [mysub].example.com, root domains, or root domains with some port number. This page includes a hint that the subscription server should be part of a routable address: http://projects.unbit.it/uwsgi/wiki/Example.
So I have a follow-up question:
Is the FastRouter even meant to let nodes announce new routes that haven't yet been set statically in a DNS zone file? I don't really care whether it's http://[key].example.com or http://example.com/[key], what's important is that these keys can be generated from inside a Docker container at setup time of the uwsgi server.
Generally the "dockered" apps run in a different network namespace, so loopback of a docker app is not the same of the fastrouter.
Use unix sockets for subscriptions, they are a great way for inter-namespace communication.
Your commands are good, the fastrouter is pretty verbose so if you do not see messages in its logs it is not working (and eventually you can strace it)