How to solve: Grafana has failed to load its application files - docker

I want to run Grafana in a specific domain, but this problem is encountered
Grafana image version is 9.1.0
docker-compose:
version: "3.8"
services:
grafana:
container_name: ${CONTAINER_GRAFANA}
image: grafana/grafana:9.1.0
restart: unless-stopped
env_file: .env
environment:
TZ: Asia/Tehran
GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
volumes:
- grafana-data:/var/lib/grafana
- ./docker/grafana/grafana.ini:/etc/grafana/grafana.ini
#- /etc/grafana/provisioning
ports:
- "${GRAFANA_PORT}:3000"
grafana.ini:
[server]
# Protocol (http, https, h2, socket)
;protocol = http
# The ip address to bind to, empty will bind to all interface
;http_addr =
# The http port to use
;http_port = 3000
# The public facing domain name used to access grafana from a browser
domain = ${GRAFANA_DOMAIN}
# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false
# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = %(protocol)s://%(domain)s/
If you're seeing this Grafana has failed to load its application files
This could be caused by your reverse proxy settings.
If you host grafana under subpath make sure your grafana.ini root_url setting includes subpath. If not using a reverse proxy make sure to set serve_from_sub_path to true.
If you have a local dev build make sure you build frontend using: yarn start, yarn start:hot, or yarn build
Sometimes restarting grafana-server can help
Check if you are using a non-supported browser. For more information, refer to the list of supported browsers.

I would suspect, that you are missing a GRAFANA_DOMAIN: ${GRAFANA_DOMAIN} in your docker-compose.yml. You need to pass through this environment variable so it is available inside the container.

Related

Connecting to XDEBUG on remote Docker Image

I am trying to debug using NetBeans 11 as my client, Xdebug 3 on the Docker image. The Docker container is on a remote host. I am unable to make a connection. Indicator at the bottom of the NetBeans screen scrolls forever with "waiting for connection (netbeans-xdebug)". I am not sure what I am doing wrong. I have had this work in the past without Docker and with Xdebug 2, I am not sure if I messed up the Xdebug 3, the Docker or both.
My configurations:
Dockerfile adds Xdebug properly and I can see it in my container.
docker-compose.yml
---
services:
drupal:
container_name: intranet-finkenb2
ports:
- "8082:80"
- "9092:9003"
volumes:
- /home/finkenb2/intranet/custom_themes:/opt/drupal/web/themes/custom
- /home/finkenb2/intranet/custom_modules:/opt/drupal/web/modules/custom
environment:
XDEBUG_MODE: debug,develop
XDEBUG_SESSION: netbeans-xdebug
XDEBUG_CONFIG: >
client_host = localhost
client_port = 9003
discover_client_host=true
start_with_request=yes
db:
container_name: intranet-finkenb2-db
solr:
container_name: intranet-finkenb2-solr
ports:
- "8982:8983"
volumes:
public_files:
private_files:
site_settings:
SSH Tunnel via PuTTY: R9092 localhost:9092
NetBeans PHP Debug config:
- Debugger Port: 9092
- Session ID: netbeans-xdebug
- Maximum Data Length: 8192
- Check: Stop at first line
NetBeans Project config (run configuration):
- Run As: Remote Web Site
- Project URL: http://intranet-finkenb2.devel.lib.msu.edu
- index file: index.php
- remote connection
- hostname intranet8.devel.lib.msu.edu /*docker host server*/
- user/pwd correct
- initial directory /var/www/
This is not correct:
ports
- "9092:9003"
Xdebug connects to your IDE, so you don't need to expose the port. These ports are for external exposures anyway, and you're SSH-ing into the container with -R already, so this makes no sense.
This is not correct:
XDEBUG_CONFIG: >
client_host = localhost
client_port = 9003
discover_client_host=true
start_with_request=yes
You can't use all of these as part of the XDEBUG_CONFIG variable, start_with_request, for example.
discover_client_host with Docker doesn't work, as it gets the wrong IP through the gateway
localhost is normally also not correct, as it needs to be the IP/hostname of the machine where your IDE listens. But you're SSH-ing into that container, so that should fine fine.
With that SSH tunnel, you need to set client_port to the port of your remote SSH end point (9092).
If anything else is unclear, make a log file, or try to debug a page with xdebug_info() in it, and it will tell you what Xdebug has tried.

Certificate of K3S cluster

I'm using a K3S Cluster in a docker(-compose) container in my CI/CD pipeline, to test my application code. However I have problem with the certificate of the cluster. I need to communicate on the cluster using the external addres. My docker-compose script looks as follows
version: '3'
services:
server:
image: rancher/k3s:v0.8.1
command: server --disable-agent
environment:
- K3S_CLUSTER_SECRET=somethingtotallyrandom
- K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
- K3S_KUBECONFIG_MODE=666
volumes:
- k3s-server:/var/lib/rancher/k3s
# get the kubeconfig file
- .:/output
ports:
# - 6443:6443
- 6080:6080
- 192.168.2.110:6443:6443
node:
image: rancher/k3s:v0.8.1
tmpfs:
- /run
- /var/run
privileged: true
environment:
- K3S_URL=https://server:6443
- K3S_CLUSTER_SECRET=somethingtotallyrandom
ports:
- 31000-32000:31000-32000
volumes:
k3s-server: {}
accessing the cluster from python gives me
MaxRetryError: HTTPSConnectionPool(host='192.168.2.110', port=6443): Max retries exceeded with url: /apis/batch/v1/namespaces/mlflow/jobs?pretty=True (Caused by SSLError(SSLCertVerificationError("hostname '192.168.2.110' doesn't match either of 'localhost', '172.19.0.2', '10.43.0.1', '172.23.0.2', '172.18.0.2', '172.23.0.3', '127.0.0.1', '0.0.0.0', '172.18.0.3', '172.20.0.2'")))
Here are my two (three) question
how can I add additional IP adresses to the cert generation? I was hoping the --bind-address in the server command triggers taht
how can I fall back on http providing an --http-listen-port didn't achieve the expected result
any other suggestion how I can enable communication with the cluster
changing the python code is not really an option as I would like o keep the code unaltered for testing. (Fallback on http works via kubeconfig.
The solution is to use the parameter tls-san
server --disable-agent --tls-san 192.168.2.110

Traefik v2 reverse proxy to a local server outside Docker

I have a simple server written in Python that listens on port 8000 inside a private network (HTTP communication). There is now a requirement to switch to HTTPS communications and every client that sends a request to the server should get authenticated with his own cert/key pair.
I have decided to use Traefik v2 for this job. Please see the block diagram.
Traefik runs as a Docker image on a host that has IP 192.168.56.101. First I wanted to simply forward a HTTP request from a client to Traefik and then to the Python server running outside Docker on port 8000. I would add the TLS functionality when the forwarding is running properly.
However, I can not figure out how to configure Traefik to reverse proxy from i.e. 192.168.56.101/notify?wrn=1 to the Python server 127.0.0.1:8000/notify?wrn=1.
When I try to send the above mentioned request to the server (curl "192.168.56.101/notify?wrn=1") I get "Bad Gateway" as an answer. What am I missing here? This is the first time that I am in contact with Docker and reverse proxy/Traefik. I believe it has something to do with ports but I can not figure it out.
Here is my Traefik configuration:
docker-compose.yml
version: "3.3"
services:
traefik:
image: "traefik:v2.1"
container_name: "traefik"
hostname: "traefik"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./traefik.yml:/traefik.yml:ro"
traefik.yml
## STATIC CONFIGURATION
log:
level: INFO
api:
insecure: true
dashboard: true
entryPoints:
web:
address: ":80"
providers:
docker:
watch: true
endpoint: "unix:///var/run/docker.sock"
file:
filename: "traefik.yml"
## DYNAMIC CONFIGURATION
http:
routers:
to-local-ip:
rule: "Host(`192.168.56.101`)"
service: to-local-ip
entryPoints:
- web
services:
to-local-ip:
loadBalancer:
servers:
- url: "http://127.0.0.1:8000"
First, 127.0.0.1 will resolve to the traefik container and not to the docker host. You need to provide a private IP of the node and it needs to be accessible form the traefik container.
There is some workaround to make proxy to localhost:
change 127.0.0.1 to IP of docker0 interface
It should be 172.17.0.1
and then try to listen your python server on all interfaces (0.0.0.0)
if you use simple python http server nothing change... on default it listen on all interfaces

Adding Traefik StripPrefix middleware to docker-compose labels results in 504

I have developed several docker-ized full-stack web applications that I am trying to route requests to using Traefik. I want to take advantage of the dynamic configuration via docker-compose labels. I would like to apply the stripPrefix middleware option so I can use the same application routing as if each app were served at the root. However, once these rules are applied it results in a 504 Gateway Timeout response.
Here's my set up:
Traefik 2.0.1
Docker 19.03.2, Compose 1.24.1
NGINX:latest images
A global docker network on which the Traefik container runs
Multiple multi-container applications, each of which includes an NGINX web server
All applications have their own local networks and the NGINX containers are also on the global network.
Each application is configured to be listening at /
Here is the docker-compose.yml definition for the offending NGINX container:
nginx:
image: nginx:latest
container_name: "mps_nginx"
volumes:
- ./nginx/confs/nginx.conf:/etc/nginx/default.conf
- ./static:/www/static
restart: "always"
labels:
- traefik.http.routers.mps.rule=Host(`localhost`) && PathPrefix(`/mps`)
- traefik.http.middlewares.strip-mps.stripprefix.prefixes=/mps
- traefik.http.routers.mps.middlewares=strip-mps#docker
networks:
- default
- mps
The aggravating part is, when I comment out the middlewares labels, it runs just fine but cannot find a matching URL pattern.
Prior to this I tested my approach using the whoami container that is defined in the Traefik Quickstart Tutorial:
# Test service to make sure our local docker-compose network functions
whoami:
image: containous/whoami
labels:
- traefik.http.routers.whoami.rule=Host(`localhost`) && PathPrefix(`/whoami`)
- traefik.http.middlewares.strip-who.stripprefix.prefixes=/whoami
- traefik.http.routers.whoami.middlewares=strip-who#docker
A request to http://localhost/whoami returns (among other things)
GET / HTTP/1.1.
This is exactly how I expected my routing approaches to work for all my other applications. The Traefik dashboard shows green all around for every middleware I register and yet all I see is 504 errors.
If anyone has any clues I would sincerely appreciate it.
To summarize my solution to a similar problem
my-service:
image: some/image
ports: # no need to expose port, just to show that service listens on 8090
- target: 8090
published: 8091
mode: host
labels:
- "traefik.enable=true"
- "traefik.http.routers.myservice-router.rule=PathPrefix(`/myservice/`)"
- "traefik.http.routers.myservice-router.service=my-service"
- "traefik.http.services.myservice-service.loadbalancer.server.port=8090"
- "traefik.http.middlewares.myservice-strip.stripprefix.prefixes=/myservice"
- "traefik.http.middlewares.myservice-strip.stripprefix.forceslash=false"
- "traefik.http.routers.myservice-router.middlewares=myservice-strip"
Now details
Routing rule
... redirect all calls thats path starts with /myservice/
- "traefik.http.routers.myservice-router.rule=PathPrefix(`/myservice/`)"
... to service my-service
- "traefik.http.routers.myservice-router.service=my-service"``
... to port 8090
- "traefik.http.services.myservice-service.loadbalancer.server.port=8090"
... and uses myservice-strip middleware
- "traefik.http.routers.myservice-router.middlewares=myservice-strip"
... this middleware will strip /myservice from path
- "traefik.http.middlewares.myservice-strip.stripprefix.prefixes=/myservice"
... and will not add trailing slash (/) if path becomes an empty string
- "traefik.http.middlewares.myservice-strip.stripprefix.forceslash=false"
There is an issue with prefix without ending with '/'.
Test your config like this:
- "traefik.http.routers.whoami.rule=Host(`localhost`) && (PathPrefix(`//whoami/`) || PathPrefix(`/portainer`))"
- "traefik.http.middlewares.strip-who.stripprefix.prefixes=/whoami"
I had a similar problem that was solved with the addition of
- "traefik.http.middlewares.strip-who.stripprefix.forceslash=true"
It makes sure the strip prefix doesn't also remove the forward slash.
You can read more about the forceslash documentation https://docs.traefik.io/middlewares/stripprefix/

Traefik Bad Gateway

I've got some strange issue. I have following setup:
one docker-host running traefik as LB serving multiple sites. sites are most php/apache. HTTPS is managed by traefik.
Each site is started using a docker-compose YAML containing the following:
version: '2.3'
services:
redis:
image: redis:alpine
container_name: ${PROJECT}-redis
networks:
- internal
php:
image: registry.gitlab.com/OUR_NAMESPACE/docker/php:${PHP_IMAGE_TAG}
environment:
- APACHE_DOCUMENT_ROOT=${APACHE_DOCUMENT_ROOT}
container_name: ${PROJECT}-php-fpm
volumes:
- ${PROJECT_PATH}:/var/www/html:cached
- .docker/php/php-ini-overrides.ini:/usr/local/etc/php/conf.d/99-overrides.ini
ports:
- 80
networks:
- proxy
- internal
labels:
- traefik.enable=true
- traefik.port=80
- traefik.frontend.headers.SSLRedirect=false
- traefik.frontend.rule=Host:${PROJECT}
- "traefik.docker.network=proxy"
networks:
proxy:
external:
name: proxy
internal:
(as PHP we use 5.6.33-apache-jessie or 7.1.12-apache f.e.)
Additionally to above, some sites get following labels:
traefik.docker.network=proxy
traefik.enable=true
traefik.frontend.headers.SSLRedirect=true
traefik.frontend.rule=Host:example.com, www.example.com
traefik.port=80
traefik.protocol=http
what we get is that some requests end in 502 Bad Gateway
traefik debug output shows:
time="2018-03-21T12:20:21Z" level=debug msg="vulcand/oxy/forward/http: Round trip: http://172.18.0.8:80, code: 502, Length: 11, duration: 2.516057159s"
can someone help with that?
it's completely random when it happens
our traefik.toml:
debug = true
checkNewVersion = true
logLevel = "DEBUG"
defaultEntryPoints = ["https", "http"]
[accessLog]
[web]
address = ":8080"
[web.auth.digest]
users = ["admin:traefik:some-encoded-pass"]
[entryPoints]
[entryPoints.http]
address = ":80"
# [entryPoints.http.redirect] # had to disable this because HTTPS must be enable manually (not my decission)
# entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[retry]
[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "example.com"
watch = true
exposedbydefault = false
[acme]
email = "info#example.com"
storage = "acme.json"
entryPoint = "https"
onHostRule = true
[acme.httpChallenge]
entryPoint = "http"
Could the issue be related to using the same docker-compose.yml?
Another reason can be that you might be accidentally mapping to the vm's port instead of the container port.
I made a change to my port mapping on the docker-compose file and forgot to update the labeled port so it was trying to map to a port on the machine that was not having any process attached to it
Wrong way:
ports:
- "8080:8081"
labels:
- "traefik.http.services.front-web.loadbalancer.server.port=8080"
Right way
ports:
- "8080:8081"
labels:
- "traefik.http.services.front-web.loadbalancer.server.port=8081"
Also in general don't do this, instead of exposing ports try docker networks they are much better and cleaner. I made my configuration documentation like a year ago and this was more of a beginner mistake on my side but might help someone :)
For anyone getting the same issue:
After recreating the network (proxy) and restarting every site/container it seems to work now.
I still don't know where the issue was from.
If you see Bad Gateway with Traefik chances are you have a Docker networking issue. First have a look at this issue and consider this solution. Then take a look at providers.docker.network (Traefik 2.0) or, in your case, the docker.network setting (Traefik 1.7).
You could add a default network here:
[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "example.com"
watch = true
exposedbydefault = false
network = "proxy"
Or define/override it for a given service using the traefik.docker.network label.
Got the same problem and none of the above mentioned answers solved it for me. In my case a wrong loadbalancer was added. Removing the label or changing it to the correct port made the trick.
- "traefik.http.services.XXX.loadbalancer.server.port=XXX"
In your example you don't have traefik enabled:
traefik.enable=false
Make sure to enable it first and then test your containers.
The error "bad gateway" is returned when the web server in the container doesn't allow traffic from traefik e.g. because of wrong interface binding like localhost instead of 0.0.0.0.
Take Ruby on Rails for example. Its web server puma is configured by default like this (see config/puma.rb):
port ENV.fetch("PORT") { 3000 }
But in order to allow access from traefik puma must bind to 0.0.0.0 like so:
bind "tcp://0.0.0.0:#{ ENV.fetch("PORT") { 3000 } }"
This solved the problem for me.
Another cause can be exposing a container at a port that Traefik already uses.
I forgot to expose the port in my Dockerfile thats why traefik did not find a port to route to. So expose the port BEFORE you start the application like node:
#other stuff before...
EXPOSE 3000
CMD ["node", "dist/main" ]
Or if you have multiple ports open you have to specify which port traefik should route the domain to with:
- "traefik.http.services.myservice.loadbalancer.server.port=3000"
Or see docs
I faced very close issue to this exception my problem was not related to network settings or config, after time we figured out that the exposed port from the backend container is not like the port we mapping to to access form outside the service port was 5000 and we mapped 9000:9000 the solution was to fix the port issue first 9000:5000.

Resources