Nginx reverse proxy fails to access prometheus container - docker

Question

I've created this docker compose file which build + runs without errors
version: '3.7'
volumes:
grafana:
driver: local
prometheus-data:
driver: local
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
expose:
- 9090
volumes:
- prometheus-data:/prometheus
- ./prometheus:/etc/prometheus
restart: unless-stopped
command:
- "--config.file=/etc/prometheus/prometheus.yml"
grafana:
image: grafana/grafana-oss:latest
container_name: grafana
expose:
- 3000
volumes:
- grafana:/etc/grafana
restart: unless-stopped
nginx:
image: nginx
container_name: webserver
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- /opt/certs/:/etc/nginx/certs/
- ./nginx/gw-web/:/usr/share/nginx/html:ro
- ./nginx/nginx_proxy.conf:/etc/nginx/conf.d/default.conf
- ./nginx/nginx.conf:/etc/nginx/nginx.conf
command: [nginx-debug, '-g', 'daemon off;']
nginx has the following reverse-proxy configuration:
server {
listen 80;
listen [::]:80;
server_name 10.23.225.72;
location /prometheus {
proxy_pass http://prometheus:9090;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_max_temp_file_size 0;
proxy_read_timeout 3000;
}
location /grafana {
proxy_pass http://grafana:3000;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_max_temp_file_size 0;
proxy_read_timeout 3000;
}
location / {
root /usr/share/nginx/html;
}
}
The root page is accessible and works as expected but when I try to go to any of the containers I just get a '404: not found' error.
What I've also tried:
location /prometheus {
proxy_pass http://prometheus:9090/;
and
location /prometheus/ {
proxy_pass http://prometheus:9090/;
and
location /prometheus/ {
proxy_pass http://prometheus:9090;
I've also tried running nginx as a service on the host machine, no success there either and anyway my project requires me to containerize this.
For shits and giggles, I tried adding another nginx container to the docker compose file:
nginx2:
image: nginx
container_name: webserver2
restart: unless-stopped
expose:
- 8080
and nginx config:
location /webserver {
proxy_pass http://webserver2:8080/;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_max_temp_file_size 0;
proxy_read_timeout 3000;
}
But that also returns 404.
Is there something going on with the docker networking maybe?
I check the network configuration json by inspecting it from vscode
{
"Name": "frp-6_default",
"Id": "940872d1cf151f8d1bb060edbf95c1bcfe29a544c7ea9561091b79a6c5588510",
"Created": "2023-01-25T11:52:01.520893117Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"4a2c2b6ca8aec52f801ad554c6b2c14abbe616078393f1e8ef40bd82ad12aa2a": {
"Name": "webserver2",
"EndpointID": "86228e6678d82a9d6735b5440618be7fc281127e6d610d281d9ffc5bcb4f256f",
"MacAddress": "02:42:ac:12:00:05",
"IPv4Address": "172.18.0.5/16",
"IPv6Address": ""
},
"9e13cbc4805fedf4db05b5de989f726bc110b97596b99b933a93e701641294a9": {
"Name": "webserver",
"EndpointID": "1bbcfe058373b225e5f53e8ff4d907d39bed578d9ac14d514ccf8da2d3c9628a",
"MacAddress": "02:42:ac:12:00:04",
"IPv4Address": "172.18.0.4/16",
"IPv6Address": ""
},
"b846cc374fa09e90589c982613224f31135660a17a5f4585b24d876ea2abd53c": {
"Name": "grafana",
"EndpointID": "3c0dac6d612a48dc067a5c208334af4f90c8c1d9122e263ca464186c1cf19f35",
"MacAddress": "02:42:ac:12:00:03",
"IPv4Address": "172.18.0.3/16",
"IPv6Address": ""
},
"dbe81d7b88413f78a06ce94f86781242299a3f6eee0f96f607ebaf8fb0b17be6": {
"Name": "prometheus",
"EndpointID": "4792bbd19cc7b782ec522d148eef7dcb5e31b5e38d99e886ff728a3f519b4973",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "default",
"com.docker.compose.project": "frp-6",
"com.docker.compose.version": "2.15.1"
}
}
The following stand out to me:
"Internal": false,
"Attachable": false,
Can those be the culprits? If so, how do I change them?
For the record, I also tried putting the IP addresses for the containers in the nginx config like so
location /prometheus {
proxy_pass http://172.18.0.2:9090/;
still unsuccessful...

Answer 1

The problem here is in your prometheus configuration. Because you have:
location /prometheus {
proxy_pass http://prometheus:9090;
...
}
When you access http://yourhost/prometheus/, that request gets proxied to http://prometheus:9090/prometheus/, and by default prometheus doesn't know what to do with that /prometheus path.
You need to tell it that it's being served from a non-root path using the --web.external-url command line option. That might look something like:
services:
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus:/etc/prometheus
- prometheus-data:/prometheus
restart: unless-stopped
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --web.console.libraries=/usr/share/prometheus/console_libraries
- --web.console.templates=/usr/share/prometheus/consoles
- --web.external-url=http://localhost:8080/prometheus/
(I've preserved all the command line options that are used by default in the prometheus image; the only thing new thing here is the --web.external-url option.)
Before making this change:
$ curl -i http://localhost:8080/prometheus
HTTP/1.1 404 Not Found
Server: nginx/1.23.3
Date: Wed, 25 Jan 2023 13:17:42 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 19
Connection: keep-alive
X-Content-Type-Options: nosniff
404 page not found
After making this change:
$ curl -i http://localhost:8080/prometheus
HTTP/1.1 302 Found
Server: nginx/1.23.3
Date: Wed, 25 Jan 2023 13:18:19 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 40
Connection: keep-alive
Location: /prometheus/graph
Found.