Where does Prometheus store metrics data (in container)? - docker

Question

I am searching for the ./data folder described in Storage section pf Prometheus Documentation.
I run a basic Prometheus Docker container prom/prometheus on Kubernetes. If I execute a shell inside the container, the working directory is /prometheus and it contains the wal directory, but it does not have the structure mentioned in the documentation and I can not find any metric data.
Where are the metrics stored which I can query over Prometheus GUI?

Answer 1

As you can check the Prometheus Dockerfile on Github (https://github.com/prometheus/prometheus/blob/master/Dockerfile#L24), the working directory is /prometheus and that where you will find all the metrics and data.
Below is the data present in /prometheus directory
/prometheus $ ls -l
total 8
-rw-r--r-- 1 nobody nogroup 0 May 28 12:39 lock
-rw-r--r-- 1 nobody nogroup 20001 May 28 12:45 queries.active
drwxr-xr-x 2 nobody nogroup 4096 May 28 12:39 wal
And this is the *Time Series Database * which you can't decode . If you will run more exporters, probably you can see more folders like chunks etc.
Ref: https://prometheus.io/docs/prometheus/1.8/storage/

Answer 2

I have figured out the problem now. The structure, which was mentioned in the documentation, need several hours to build up.
./data
├── 01BKGTZQ1HHWHV8FBJXW1Y3W0K
│ └── meta.json
├── 01BKGV7JC0RY8A6MACW02A2PJD
│ ├── chunks
│ │ └── 000001
│ ├── tombstones
│ ├── index
│ └── meta.json
└── wal
├── 00000002
└── checkpoint.000001
The first of those folders starting with 01... appeared after 4 hours. The next one took an other 2 hours. It seems to vary according to the amount of metrics you are pulling.
Sidenote: As nischay goyal wrote in his answer, Prometheus TSDB can not be decoded. You can query the Prometheus API with a script to export metrics.