No connection to locally hosted portainer after restart - docker

Question

I had portainer working yesterday when I installed it. After turning the server off in the evening and starting it again today I can't access it anymore (ERR_CONNECTION_TIMED_OUT).
I think there may be a problem with some configuration of docker itself as I tried to configure docker swarm / portainer agent yesterday but failed.
What I tried so far:
sudo docker rm -f portainer
sudo docker volume rm -f portainer_data
sudo docker run -d --privileged -p 9000:9000 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest
I still had ERR_CONNECTION_TIMED_OUT. To check any firewall issues:
sudo ufw allow 9000
and then
nc -l 9000
When I now try to access portainer on port 9000, I get the following response:
GET / HTTP/1.1
Host: 192.168.178.46:9000
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: _pk_id.1.15bd=2476c3cd963cb84b.1667402782.
I don't know what exactly that means, but to me it looks like my request gets to the server but for any reason portainer doesn't answer it.
I also checked the logs of portainer, that showed:
sudo docker logs portainer
2022/11/03 09:45AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:530 > encryption key file not present | filename=portainer
2022/11/03 09:45AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:549 > proceeding without encryption key |
2022/11/03 09:45AM INF github.com/portainer/portainer/api/database/boltdb/db.go:124 > loading PortainerDB | filename=portainer.db
2022/11/03 09:45AM INF github.com/portainer/portainer/api/internal/ssl/ssl.go:80 > no cert files found, generating self signed SSL certificates |
2022/11/03 09:45:47 server: Reverse tunnelling enabled
2022/11/03 09:45:47 server: Fingerprint cf:97:21:21:46:a0:bf:ef:aa:e0:7c:66:f9:77:86:67
2022/11/03 09:45:47 server: Listening on 0.0.0.0:8000...
2022/11/03 09:45AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:789 > starting Portainer | build_number=24674 go_version=1.18.3 image_tag=linux-amd64-2.16.0 nodejs_version=14.20.1 version=2.16.0 webpack_version=5.68.0 yarn_version=1.22.19
2022/11/03 09:45AM INF github.com/portainer/portainer/api/http/server.go:337 > starting HTTPS server | bind_address=:9443
2022/11/03 09:45AM INF github.com/portainer/portainer/api/http/server.go:322 > starting HTTP server | bind_address=:9000

Answer 1

Okay, I managed to fix this.
TL;DR:
I removed the "insecure" line inside of /etc/docker/daemon.json
As I already mentioned I tried to get docker swarm / portainer agent running (to be able to look inside of the volumes I created).
By trying to create a Docker Swarm via portainer it asks for an environment address, where I tried LOCALHOST:9000 (Connection test failed).
I then tried to figure out why that fails and found, that I needed to add
{ "insecure-registries" : ["LOCALHOST-IP:9000"] }
to /etc/docker/daemon.json. That didn't solve the issue and I forgot about that. It turns out that removing this line (and restarting the server) gave me access to portainer again.
Still need to figure out how I can look inside volumes from inside portainer but thats a chapter of another book.