I've installed Haproxy 1.8 on a Kubernetes Container.
Whenever I make any request to /test, I always get 503 Service Unavailable response. I want to return the stats page when I get a request to /test
Following is my configuration file:
/etc/haproxy/haproxy.cfg:
global
daemon
maxconn 256
defaults
mode http
timeout connect 15000ms
timeout client 150000ms
timeout server 150000ms
frontend stats
bind *:8404
mode http
stats enable
stats uri /stats
stats refresh 10s
frontend http-in
bind *:8083
default_backend servers
acl ar1 path -i -m sub /test
use_backend servers if ar1
backend servers
mode http
server server1 10.1.0.46:8404/stats maxconn 32
# 10.1.0.46 is my container IP
I can access the /stats page using:
curl -ik http://10.1.0.46:8404/stats
But when I do:
curl -ik http://10.1.0.46:8083/test
I always get following response:
HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Content-Type: text/html
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
I started haproxy using:
/etc/init.d/haproxy restart
and then subsequently restart it using:
haproxy -f haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
Following is the output of netstat -anlp:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN 54/python3.5
tcp 0 0 0.0.0.0:8083 0.0.0.0:* LISTEN 802/haproxy
tcp 0 0 0.0.0.0:8404 0.0.0.0:* LISTEN 802/haproxy
tcp 0 0 10.1.0.46:8404 10.0.15.225:20647 TIME_WAIT -
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
Following is the output of ps -eaf:
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Jul22 ? 00:00:00 /bin/sh -c /bin/bash -x startup_script.sh
root 6 1 0 Jul22 ? 00:00:00 /bin/bash -x startup_script.sh
root 54 6 0 Jul22 ? 00:00:09 /usr/local/bin/python3.5 /usr/local/bin/gunicorn --bind 0.0.0.0:5000 runner:app?
root 57 54 0 Jul22 ? 00:02:50 /usr/local/bin/python3.5 /usr/local/bin/gunicorn --bind 0.0.0.0:5000 runner:app?
root 61 0 0 Jul22 pts/0 00:00:00 bash
root 739 0 0 07:02 pts/1 00:00:00 bash
root 802 1 0 08:09 ? 00:00:00 haproxy -f haproxy.cfg -p /var/run/haproxy.pid -sf 793
root 804 739 0 08:10 pts/1 00:00:00 ps -eaf
Why could I be getting 503 unavailable always?
Why do you use HAProxy 1.8 when a 2.2.x already exists?
You will need to adopt the path in the backend which can't be set on the server level.
backend servers
mode http
http-request set-path /stats
server server1 10.1.0.46:8404 maxconn 32
# 10.1.0.46 is my container IP
Related
I am trying to use https://github.com/markshust/docker-magento. Following the instructions, the setup includes a nginx container in the frontend, which is suppose to connect with a php-fpm in the backend.
Containers:
ubuntu#DESKTOP-HED9HVG:/mnt/c/Users/Me$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8db110d5737a markoshust/magento-nginx:1.18-4 "/docker-entrypoint.…" 2 hours ago Up 2 hours 80/tcp, 0.0.0.0:80->8000/tcp, 0.0.0.0:443->8443/tcp localdev_app_1
74ff9e11646a markoshust/magento-php:7.4-fpm-5 "docker-php-entrypoi…" 2 hours ago Up 2 hours 9000-9001/tcp localdev_phpfpm_1
As far as I can understand the code, the phpfpm is listening on the docker socket.
Nginx upstream:
upstream fastcgi_backend {
server unix:/sock/docker.sock;
}
php-fpm.conf:
:
listen = /sock/docker.sock
:
this is the docker-compose.yaml file
But I can't get it to work.
php-fpm is working:
app#74ff9e11646a:~/html$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
app 1 0.0 0.2 222828 33168 ? Ss 06:57 0:00 php-fpm: master process (/usr/local/etc/php-fpm.conf)
app 6 0.0 0.0 222828 10412 ? S 06:57 0:00 php-fpm: pool www
app 7 0.0 0.0 222828 10348 ? S 06:57 0:00 php-fpm: pool www
app 8 0.0 0.0 222828 10348 ? S 06:57 0:00 php-fpm: pool www
app 9 0.0 0.0 222828 10348 ? S 06:57 0:00 php-fpm: pool www
However,from nginx to phpfpm, there is ping, but there is no telnet:
/var/www/html # ping phpfpm
PING phpfpm (172.19.0.7): 56 data bytes
64 bytes from 172.19.0.7: seq=0 ttl=64 time=0.128 m
/var/www/html # telnet phpfpm 9000
telnet: can't connect to remote host (172.19.0.7): Connection refused
/var/www/html # telnet phpfpm 9001
telnet: can't connect to remote host (172.19.0.7): Connection refused
I am windows10 wsl2. Any idea what I should check?
You are telling PHP-FPM to use a socket for communication, that's why it's not listening on TCP / port 9000.
It can either listen on a TCP Port, OR use a socket, not both.
For some reason whenever I suspend my VM and resume it, I can no longer connect to the docker container that is hosted within the VM. Usually, I pass -p 3000:3000 to the docker container so that I can access the rails instance within it and this works fine, but when I suspend the VM and resume it later, I can no longer connect to port 3000 even though it's listening within the docker image.
This results in me having to reboot the VM as service docker restart does not change anything.
Is there something else I should be looking at to resolve this issue? I've been suspending/resuming my VM with docker in it for quite awhile and have never run into this issue before.
EDIT
To reproduce this issue, I simply resumed my VM and tried connecting to localhost port 3000 from the VM itself (not within the docker image) and it cannot connect. However, below shows that port 3000 is listening:
[root:kali:~/app]# curl http://localhost:3000
curl: (56) Recv failure: Connection reset by peer
[root:kali:~/app]# netstat -antp | grep -i listen
tcp 0 0 127.0.0.1:43050 0.0.0.0:* LISTEN 84770/autossh
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 20478/sshd
tcp6 0 0 :::3000 :::* LISTEN 32731/docker-proxy
tcp6 0 0 :::3001 :::* LISTEN 32715/docker-proxy
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 :::22 :::* LISTEN 20478/sshd
From within docker, I can see that rails is working:
[root:77f444beafff:~/app]# rails s --binding 0.0.0.0
=> Booting Puma
=> Rails 5.2.3 application starting in development
=> Run `rails server -h` for more startup options
Puma starting in single mode...
* Version 3.12.1 (ruby 2.5.1-p57), codename: Llamas in Pajamas
* Min threads: 5, max threads: 5
* Environment: development
* Listening on tcp://0.0.0.0:3000
Use Ctrl-C to stop
And here's the netstat from within docker:
[root:77f444beafff:~/app]# netstat -antp | grep -i listen
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 478/redis-server *:
tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN 765/puma 3.12.1 (tc
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN -
tcp6 0 0 :::6379
If I curl from within the docker image, I can see it hits the rails app just fine:
[root:77f444beafff:~/app]# curl http://localhost:3000/ -I
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
Content-Type: text/html; charset=utf-8
ETag: W/"5078d30a6c1a5f6fc5cb7f9a82cd89f5"
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: _vspm_session=Cace%2FN0zB%2F6QJOiietbuHxTHOMZUMuRmEukYqQTNaHQ91hskaN%2BPJzev0KdGUAAtYx9a35Mqdkr8eRkPdH4qOl6vOaCcPU0gy8s7IMfkb9VhRGPPbecepmI%2F9leA2dnD694P8ctXSBklOCnjhN0%3D--SglWrWvx3BFEAI3z--IkylACdXbR6eF27Hgn0Cgg%3D%3D; path=/; HttpOnly
X-Request-Id: 29aa7251-f29a-4309-adec-6af479e7bd9b
X-Runtime: 12.241723
I'm having exactly the same issue with my VMWare virtual machine (VMWare running on Windows).
The only workaround that is working for me is:
docker stop $(docker ps -aq) && sudo systemctl restart NetworkManager docker
If I had to guess I would say it may be related to some firewall rules docker setup on start, maybe when you resume the virtual machine a change in the network configuration breaks those rules.
Similar issue: https://github.com/docker/for-mac/issues/1990 (Doesn't seem specific to docker for mac).
I was able to solve this issue with the hint given by lannox in the comment. It's necessary to mark the network interfaces of the docker containers as unmanaged by NetworkManager.
To do that, create a new file /etc/NetworkManager/conf.d/10-unmanage-docker-interfaces.conf with the following content:
[keyfile]
unmanaged-devices=interface-name:docker*;interface-name:veth*;interface-name:br-*;interface-name:vmnet*;interface-name:vboxnet*
This configures NetworkManager to ignore all interfaces with names docker*,
veth*,
br-*,
vmnet*, and
vboxnet* interfaces.
Then restart NetworkManager with sudo systemctl restart NetworkManager.
Next time the host suspends and resumes, the docker containers keep their network connectivity.
Several questions here that might help you solve this :
Is your docker container still running? Run docker ps and find your container
Since the -p 3000:3000 option is set I guess the port is exposed, but you might want to check you really have run your container with this option this time
Is your app really listening? Run lsof -np | grep listen and find your app listening on port 3000
Connect to your container with docker exec -it <your_container> bash and try running lsof -np | grep listen to see if this is a docker issue or your app
It seems that when you run netstat on your VM you get the following line :
tcp6 0 0 :::3000 :::* LISTEN 32731/docker-proxy
On Docker you get :
tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN 765/puma 3.12.1 (tc
There are two differences here:
:::3000 vs 0 0.0.0.0:3000, the first means it is listening on IPv6 and the second on IPv4 (found the info on this question).
tcp6 vs tcp, again IPv6 vs IPv4.
According to this other question, it seems you have to run rails with -b :: option.
The -b option binds Rails to the specified IP, by default it is localhost. You can run a server as a daemon by passing a -d option.
Please do
sudo docker ps
If you do not got your container do
sudo docker ps -a
Does your container is stopped?
If its true so do
sudo docker start CONTAINER_ID
I want to access the docker daemon on another machine,but it always fails.
Both machines are virtual machines.
systemctl status docker service
[root#localhost ~]# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2019-04-16 07:38:52 EDT; 3s ago
Docs: https://docs.docker.com
Main PID: 5191 (dockerd)
Memory: 128.5M
CGroup: /system.slice/docker.service
└─5191 /usr/bin/dockerd -H unix://var/run/docker.sock -H tcp://0.0.0.0:2375
ps -ef | grep docker
[root#localhost ~]# ps -ef | grep docker
root 5191 1 1 07:38 ? 00:00:01 /usr/bin/dockerd -H unix://var/run/docker.sock -H tcp://0.0.0.0:2375
root 5800 5161 0 07:40 pts/0 00:00:00 grep --color=auto docker
netstat -tulp
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 localhost:smtp 0.0.0.0:* LISTEN 4373/master
tcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTEN 4134/sshd
tcp6 0 0 localhost:smtp [::]:* LISTEN 4373/master
tcp6 0 0 [::]:2375 [::]:* LISTEN 5191/dockerd
tcp6 0 0 [::]:2377 [::]:* LISTEN 5191/dockerd
tcp6 0 0 [::]:7946 [::]:* LISTEN 5191/dockerd
tcp6 0 0 [::]:ssh [::]:* LISTEN 4134/sshd
Result of access on another machine
[root#localhost dack]# docker -H tcp://192.168.233.150:2375 images
error during connect: Get http://192.168.233.150:2375/v1.39/images/json: dial tcp 192.168.233.150:2375: connect: no route to host
[root#localhost dack]# docker -H tcp://192.168.233.150:2375 info
error during connect: Get http://192.168.233.150:2375/v1.39/info: dial tcp 192.168.233.150:2375: connect: no route to host
Can you check for firewall, if 2375 port is allowed to connect from other servers.
Trying to use jupyter-notebook on a docker image (https://hub.docker.com/r/tensorflow/tensorflow), but having problem where using the port-forwarded address in browser just hangs with the (chrome) home page stuck saying Waiting for 127.0.0... until it just times out.
The docker command being run looks like
➜ ~ docker run -it -p 8888:8888 --rm tensorflow/tensorflow:latest-devel-gpu-py3 jupyter-notebook --ip 0.0.0.0 --no-browser --allow-root
[I 04:26:44.023 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
[I 04:26:44.042 NotebookApp] Serving notebooks from local directory: /root
[I 04:26:44.043 NotebookApp] The Jupyter Notebook is running at:
[I 04:26:44.043 NotebookApp] http://(f1afd4b163fd or 127.0.0.1):8888/?token=5a838cefbd58822ce3de5a9ab00ed724bc6f9e048017125a
[I 04:26:44.043 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 04:26:44.043 NotebookApp]
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://(f1afd4b163fd or 127.0.0.1):8888/?token=5a838cefbd58822ce3de5a9ab00ed724bc6f9e048017125a
(note, have also tried docker run -it -p 8888:8888 --rm tensorflow/tensorflow:latest-devel-gpu-py3 /run_jupyter.sh --allow-root to similar hanging results).
Checking docker ps shows
➜ ~ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2114609d6d9d tensorflow/tensorflow:latest-devel-gpu-py3 "jupyter-notebook --…" About a minute ago Up About a minute 6006/tcp, 0.0.0.0:8888->8888/tcp mystifying_liskov
Checking for a response via curl shows
➜ ~ curl -v http://127.0.0.1:8888/?token=5a838cefbd58822ce3de5a9ab00ed724bc6f9e048017125a
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET /?token=5a838cefbd58822ce3de5a9ab00ed724bc6f9e048017125a HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.47.0
> Accept: */*
>
<at this point just hangs until I ctl+C out>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
and examine the ports shows
➜ ~ sudo netstat -plnt
[sudo] password for me:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1512/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 2485/cupsd
tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN 2284/smbd
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 1502/mysqld
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 2284/smbd
tcp 0 0 127.0.0.1:5037 0.0.0.0:* LISTEN 8558/adb
tcp 0 0 127.0.0.1:6000 0.0.0.0:* LISTEN 1006/unicorn.rb --h
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 1954/monitorix-http
tcp6 0 0 :::22 :::* LISTEN 1512/sshd
tcp6 0 0 ::1:631 :::* LISTEN 2485/cupsd
tcp6 0 0 :::445 :::* LISTEN 2284/smbd
tcp6 0 0 :::8888 :::* LISTEN 32491/docker-proxy
tcp6 0 0 :::139 :::* LISTEN 2284/smbd
tcp6 0 0 :::80 :::* LISTEN 1846/apache2
Other post I've seen seem to be people simply not forwarding the port that jupyter expects to use, but that does not seem to be the problem here. This occurs regardless of what docker image is used (so not just that particular image). If anyone has any ideas of what it could be or any debugging advice it would be appreciated.
Resolved the problem.
Restarted the host machine (note, this was the first time restarting since installing docker, but still did not work until...)
Ran sudo /etc/init.d/docker restart (Did this purely based on a hunch when skimming the troubleshooting docs here: https://docs.docker.com/toolbox/faqs/troubleshoot/#configure-http-proxy-settings-on-docker-machines).
Then the docker run ... statement from the posted question worked and can now reach the forwarded port on host machine and can curl the address. An Ubuntu notification popped up saying "Wired connection established".
This is a bit of a lame answer, but it's what worked for me. Oddly, seems have to rerun the sudo /etc/init.d/docker restart statement sometimes to get docker containers to open. Will try to figure out a bit more about what exactly was going on here, but if anyone with more experience thinks they know what may have been happening, please do let us know.
Running docker-machine version 0.5.0, Docker version 1.9.0 on OS X 10.11.1.
I've a Couchbase image of my own (not the official one). From inside the entrypoint script, I'm running some curl commands to configure the Couchbase server and to load sample data. Problem is, curl fails with error message Failed to connect to localhost port 8091: Connection refused.
I've tried 127.0.0.1, 0.0.0.0, localhost, all without any success. netstat shows that port 8091 on localhost is listening. If I later log on to the server using docker exec and run the same curl commands, those work! What am I missing?
Error:
couchbase4 | % Total % Received % Xferd Average Speed Time Time Time Current
couchbase4 | Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8091: Connection refused
netstat output:
root#cd4d3eb00666:/opt/couchbase/var/lib# netstat -lntu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:21100 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:21101 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9998 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8091 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8092 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:41125 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:11209 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:11210 0.0.0.0:* LISTEN
tcp6 0 0 :::11209 :::* LISTEN
tcp6 0 0 :::11210 :::* LISTEN
Here is my Dockerfile:
FROM couchbase
COPY configure-cluster.sh /opt/couchbase
CMD ["/opt/couchbase/configure-cluster.sh"]
and configure-cluster.sh
/entrypoint.sh couchbase-server &
sleep 10
curl -v -X POST http://127.0.0.1:8091/pools/default -d memoryQuota=300 -d indexMemoryQuota=300
curl -v http://127.0.0.1:8091/node/controller/setupServices -d services=kv%2Cn1ql%2Cindex
curl -v http://127.0.0.1:8091/settings/web -d port=8091 -d username=Administrator -d password=password
curl -v -u Administrator:password -X POST http://127.0.0.1:8091/sampleBuckets/install -d '["travel-sample"]'
This configures the Couchbase server but still debugging how to bring Couchbase back in foreground.
Complete details at: https://github.com/arun-gupta/docker-images/tree/master/couchbase
It turns out that if I do the curls after restarting the server, those work. Go figure! That said, note that the REST API for installing sample buckets is undocumented as far as I know. arun-gupta's blog and his answer here are the only places where I saw any mention of a REST call for installing sample buckets. There's a python script available but that requires installing python-httplib2.
That said, arun-gupta's last curl statement may be improved upon as follows:
if [ -n "$SAMPLE_BUCKETS" ]; then
IFS=',' read -ra BUCKETS <<< "$SAMPLE_BUCKETS"
for bucket in "${BUCKETS[#]}"; do
printf "\n[INFO] Installing %s.\n" "$bucket"
curl -sSL -w "%{http_code} %{url_effective}\\n" -u $CB_USERNAME:$CB_PASSWORD --data-ascii '["'"$bucket"'"]' $ENDPOINT/sampleBuckets/install
done
fi
where SAMPLE_BUCKETS can be a comma-separated environment variable, possible values being combinations of gamesim-sample, beer-sample and travel-sample. The --data-ascii option keeps curl from choking on the dynamically created JSON.
Now if only there was an easy way to start the server in the foreground. :)