I have Prometheus setup as a standalone service in a VM. I am able to go to localhost:9090 and move freely through the UI - graph, status, alerts, etc. I am also able to curl the metrics page and get a list of metrics. The issue I’m having is I get a permission denied error when trying to scrape metrics from my targets (localhost:9090/targets), including the same Prometheus instance.
The specific error message I get is “Get “http://localhost:9090/metrics": dial tcp [::1]:9090: connect: permission denied”. The error shows itself in all of the surrounding services - node exporter & alertmanager.
Prometheus target error: Get "http://localhost:9090/metrics": dial tcp [::1]:9090: connect: permission denied
Node target error: Get "http://localhost:9100/metrics": dial tcp [::1]:9100: connect: permission denied
Alertmanager error: msg="Error sending alert" err="Post "http://localhost:9093/api/v2/alerts": dial tcp [::1]:9093: connect: permission denied"
Does anyone know where this error message originates from? What permissions do I not have?
Edit: I've tried opening all ports listed above.
Related
I have Docker installed via Snap on Ubuntu 20.04. From time to time, Portainer (which I'm using as graphical UI to manage the containers) stops responding - in the sense, the UI accepts interaction, but the list of containers won't load, or the list of volumes, I won't be able to set up a new container, etc.
When I ran snap logs docker just now, I got the following:
2022-10-07T09:38:59+03:00 docker.dockerd[770]: time="2022-10-07T09:38:59.500434820+03:00" level=error msg="Error replicating health state for container 5e7ed995ca45945035048596539293a9bb11ee0b4e30e7c4956eec953077806b: open /var/snap/docker/common/var-lib-docker/containers/5e7ed995ca45945035048596539293a9bb11ee0b4e30e7c4956eec953077806b/.tmp-config.v2.json632473701: too many open files"
2022-10-07T09:38:59+03:00 docker.dockerd[770]: time="2022-10-07T09:38:59.542152208+03:00" level=error msg="Error replicating health state for container f876ad961153cbd2815cc7715983987bc072c631a378cf3d2ba7e248aae27423: open /var/snap/docker/common/var-lib-docker/containers/f876ad961153cbd2815cc7715983987bc072c631a378cf3d2ba7e248aae27423/.tmp-config.v2.json471368832: too many open files"
2022-10-07T09:38:59+03:00 docker.dockerd[770]: time="2022-10-07T09:38:59.542155861+03:00" level=error msg="Error replicating health state for container dabe3abe216046ef5b84e3169f7faf90b6e3050cef75a8ca23e940e516ecff20: open /var/snap/docker/common/var-lib-docker/containers/dabe3abe216046ef5b84e3169f7faf90b6e3050cef75a8ca23e940e516ecff20/.tmp-config.v2.json801890271: too many open files"
2022-10-07T09:39:00+03:00 dockerd[770]: http: Accept error: accept unix /var/run/docker.sock: accept4: too many open files; retrying in 1s
2022-10-07T09:39:01+03:00 dockerd[770]: http: Accept error: accept unix /var/run/docker.sock: accept4: too many open files; retrying in 1s
2022-10-07T09:39:02+03:00 dockerd[770]: http: Accept error: accept unix /var/run/docker.sock: accept4: too many open files; retrying in 1s
2022-10-07T09:39:02+03:00 docker.dockerd[770]: time="2022-10-07T09:39:02.822036736+03:00" level=error msg="Error replicating health state for container 98bee8456d114bff5ee423f46e7e0892dfdf5af694c8edc522841ce5e5976b1f: open /var/snap/docker/common/var-lib-docker/containers/98bee8456d114bff5ee423f46e7e0892dfdf5af694c8edc522841ce5e5976b1f/.tmp-config.v2.json624827314: too many open files"
2022-10-07T09:39:03+03:00 dockerd[770]: http: Accept error: accept unix /var/run/docker.sock: accept4: too many open files; retrying in 1s
2022-10-07T09:39:04+03:00 dockerd[770]: http: Accept error: accept unix /var/run/docker.sock: accept4: too many open files; retrying in 1s
Could you help me figure out what's causing this and how to avoid it in the future, please?
Thank you!
EDIT (31 Oct): the same problem has been occurring numerous times since I last wrote this post. I've tried the suggestions by IamK below, but they don't seem to work.
More specifically, even though I edited etc/sysctl.conf, the current soft and hard limits remain 1024 and 1048576, respectively.
I'd grealy appreciate further help. Thank you!
I've recently been learning Docker and Kubernetes. I think i now understand docker pretty well so ive been trying to make the kubernetes work. I'm on Windows btw.
I have installed kubernetes with the standard windows docker dashboard and done everything that was in the tutorials I've found to get the kubernetes dashboard.
Now I have my dashboard and I'm running a simple nginx pod.
with a service :
but when I go on localhost:88 it tells me that localhost hasnt sent any data.
I have also tried :
kubectl port-forward nginx-67d74d8fbc-qnzgf :88
and when I tried in the port that was assigned automatically it said inaccessible website and did this error in the cmd:
Handling connection for 58193
Handling connection for 58193
E0523 12:49:36.846094 16200 portforward.go:406] an error occurred forwarding 58193 ->
88: error forwarding port 88 to pod 764194941a96bf6dc7b3d491ffe910cd34885e20a0b98690392b9a4c4feca310, uid : exit status 1:
2022/05/23 10:49:36 socat[7405] E connect(16, AF=2 127.0.0.1:88, 16): Connection refused
E0523 12:49:36.847735 16200 portforward.go:406] an error occurred forwarding 58193 88: error forwarding port 88 to pod 764194941a96bf6dc7b3d491ffe910cd34885e20a0b98690392b9a4c4feca310, uid : exit status 1:
2022/05/23 10:49:36 socat[7406] E connect(16, AF=2 127.0.0.1:88, 16): Connection refused
E0523 12:49:36.884999 16200 portforward.go:234] lost connection to pod
Handling connection for 58193
E0523 12:49:36.885522 16200 portforward.go:391] error copying from local connection to
remote stream: EOF
E0523 12:49:36.888639 16200 portforward.go:346] error creating error stream for port
58193 -> 88: EOF
I don't know what to do to access the nginx app I created in my browser and i cant find any doc.
OK I have found what the problem was, I had opened docker and kubernetes on my admin session but was accessing it from a browser on my normal session.
solana account <address>
When I get account info, I have this error:
Error: RPC request error: cluster version query failed: error sending request for url (http://localhost:8899/): error trying to connect: tcp connect error: Connection refused (os error 111)
The error is indicating that the CLI RpcClient can not communicate with the Solana validator.
This is usually caused by not having solana-test-validator running in another terminal. Many make the mistake of thinking that the localhost is running the validator all the time... it's not.
In one terminal do: solana-test-validator which will startup up the local validator
Open a second terminal and do solana account - This will return account info for the default keypair
It's a network connection problem.
I use ubuntu 20.04, I was using windscribe VPN because my location and got the error, now I'm using psiphon VPN and it's working fine.
during OpenShift installation from a local mirror registry, after I started the bootstrap machine i see the following error in the journal log:
release-image-download.sh[1270]:
Error: error pulling image "quay.io/openshift-release-dev/ocp-release#sha256:999a6a4bd731075e389ae601b373194c6cb2c7b4dadd1ad06ef607e86476b129":
unable to pull quay.io/openshift-release-dev/ocp-release#sha256:999a6a4bd731075e389ae601b373194c6cb2c7b4dadd1ad06ef607e86476b129: unable to pull image:
Error initializing source docker://quay.io/openshift-release-dev/ocp-release#sha256:999a6a4bd731075e389ae601b373194c6cb2c7b4dadd1ad06ef607e86476b129:
(Mirrors also failed: [my registry:5000/ocp4/openshift4#sha256:999a6a4bd731075e389ae601b373194c6cb2c7b4dadd1ad06ef607e86476b129: Error reading manifest
sha256:999a6a4bd731075e389ae601b373194c6cb2c7b4dadd1ad06ef607e86476b129 in my registry:5000/ocp4/openshift4: manifest unknown: manifest unknown]):
quay.io/openshift-release-dev/ocp-release#sha256:999a6a4bd731075e389ae601b373194c6cb2c7b4dadd1ad06ef607e86476b129: error pinging docker registry quay.io:
Get "https://quay.io/v2/": dial tcp 50.16.140.223:443: i/o timeout
Does anyone have any idea what it can be?
The answer is here in the error:
... dial tcp 50.16.140.223:443: i/o timeout
Try this on the command line:
$ podman pull quay.io/openshift-release-dev/ocp-release#sha256:999a6a4bd731075e389ae601b373194c6cb2c7b4dadd1ad06ef607e86476b129
You'll need to be authenticated to actually download the content (this is what the pull secret does). However, if you can't get the "unauthenticated" error then this would more solidly point to some network configuration.
That IP resolves to a quay host (you can verify that with "curl -k https://50.16.140.223"). Perhaps you have an internet filter or firewall in place that's blocking egress?
Resolutions:
fix your network issue, if you have one
look at doing an disconnected /airgap install -- https://docs.openshift.com/container-platform/4.7/installing/installing-mirroring-installation-images.html has more details on that
(If you're already doing an airgap install and it's your local mirror that's failing, then your local mirror is failing)
I have installed CUPS on a new linux chromebook. It recognized my printers, and I used
sudo system-config-printer
to install drivers and test. The printers all worked. But I was unable to access CUPS using the localhost:631 web page, so I rebooted the chromebook. After that, the printers still work and I still can't access the localhost interface, but now the sudo system-config-printer fails with
No protocol specified
Unable to init server: Could not connect: Connection refused
No protocol specified
Unable to init server: Could not connect: Connection refused
No protocol specified
Unable to init server: Could not connect: Connection refused
(system-config-printer.py:658): Gtk-WARNING **: 08:11:20.173: cannot open display: :0
However, the simple system-config-printer command, without sudo, does bring up the CUPS gui window, although most functionality is unavailable. This feels like a permissions problem, but I can't find it.
Any help out there?