Issue with Loki storing on AWS S3 - docker

Question

As per title: I've an issue with Loki (running on Docker) storing its chunks & C. on a bucket of AWS S3.
Loki is running fine, simply it stores its logs in the filesystem rather than in the bucket, and infact the bucket is empty.
What is wrong in my configuration?
In AWS, IAM, I've created a user with programmatic access, and I've give it the following policy ...
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "LokiStorage",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::__myBucket__",
"arn:aws:s3:::__myBucket__/*"
]
}
]
}
The policy seems sufficient, since I've use it to push in the bucket some files present in the filesystem.
The relevant part of the docker compose...
version: "3.8"
volumes:
loki_data: {}
services:
loki:
image: grafana/loki:2.1.0
networks:
- my-overlay
ports:
- 3100:3100
volumes:
- ./loki/loki-config.yml:/etc/loki/local-config.yml
- loki_data:/loki
command: -config.file=/etc/loki/local-config.yaml
... Seems fine too: Loki - as container and service - runs smoothly.
The Loki's configuration file, "loki-config.yaml", seems also fine...
---
auth_enabled: false
ingester:
chunk_idle_period: 3m
chunk_block_size: 262144
chunk_retain_period: 1m
max_transfer_retries: 0
lifecycler:
ring:
kvstore:
store: inmemory
replication_factor: 1
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
compactor:
working_directory: /loki/boltdb-shipper-compactor
shared_store: aws
schema_config:
configs:
- from: 2020-07-01
store: boltdb-shipper
object_store: aws
schema: v11
index:
prefix: loki_index_
period: 24h
server:
http_listen_port: 3100
storage_config:
aws:
s3: s3://__myAccessKey__:__mySecretAccessKey__#eu-west-1/__myBucket__
boltdb_shipper:
active_index_directory: /loki/index
shared_store: s3
cache_location: /loki/boltdb-cache
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
... But, infact, there is something wrong and/or missing somewhere, somehow.
Suggestions?

Answer 1

I've found the culprit: there was an error in the docker-compose:
volumes:
- ./loki/loki-config.yml:/etc/loki/local-config.yml
### Here: "yml", and not "yaml" ──────────────────┘
- loki_data:/loki
command: -config.file=/etc/loki/local-config.yaml
### Here: "yaml", and not "yml" ─────────────┘
Now the compose is correct, and the Loki's configuration file is the following:
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h
max_chunk_age: 1h
chunk_target_size: 1048576
chunk_retain_period: 30s
max_transfer_retries: 0
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: s3
aws:
s3forcepathstyle: true
bucketnames: __myBucket__
region: eu-west-1
access_key_id: __myAccessKey__
secret_access_key: __mySecretAccessKey__
compactor:
working_directory: /loki/compactor
shared_store: s3
compaction_interval: 5m
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
ingestion_rate_mb: 16
ingestion_burst_size_mb: 32
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
ruler:
storage:
type: s3
s3:
s3forcepathstyle: true
bucketnames: __myBucket__
region: eu-west-1
access_key_id: __myAccessKey__
secret_access_key: __mySecretAccessKey__
rule_path: /loki/rules-temp
alertmanager_url: http://localhost:9093
ring:
kvstore:
store: inmemory
enable_api: true
The configuration seems fine, since - at last! - the bucket of S3 is populated and Loki works fine.
Maybe.
Oddly enough: the bucket (as said) is populated, but I see that Loki also stores its chunks etc. in the local filesystem:
root#ip-aaa-bbb-ccc-ddd:/var/lib/docker/volumes/monit_loki_data# tree
.
└── _data
├── compactor
│   ├── index_19375
│   └── index_19376
├── index
│   └── uploader
│   └── name
├── index_cache
│   └── index_19376
│   ├── 5a22562e87b2-1674059149901907563-1674116100.gz
│   └── compactor-1674116450.gz
├── rules
└── tmprules
10 directories, 3 files
Hence the question: Loki really works as intended? That is: storing both in the bucket and in the filesystem it is its normal behaviour?