OpenTelemetry sidecar wait for client data flush - docker

Question

I use open telemetry for tracing and metrics. I have a pretty standard setup - there is a service that produces metrics/traces and an open telemetry sidecar that collects these metrics and pushes them to AWS:
services:
service:
build:
context: .
image: service
container_name: service
ports:
- "3000:3000"
depends_on:
- aws-otel-collector
aws-otel-collector:
image: public.ecr.aws/aws-observability/aws-otel-collector:latest
container_name: aws-otel-collector
ports:
- "4317:4317"
Service flushes metrics and shuts down an exporter on service shutdown:
shutdown, err := initMetricProvider(ctx)
if err != nil {
log.Fatal(err)
}
defer func() {
log.Printf("Shutting down metric provider")
if err := shutdown(ctx); err != nil {
log.Fatal(fmt.Errorf("failed to shutdown metric provider: %w", err))
}
}()
meter := global.MeterProvider().Meter("service")
counter, err := meter.SyncInt64().Counter("test")
From time to time I am getting errors during restarts, caused by an inability to push metrics on a shutdown, smth like:
max retry time elapsed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4317: connect: connection refused"
This happens because the otel collector sidecar is getting suspended before metrics are flushed in service.
Question: how does one guarantees the sidecar waits until metrics are flushed? is there a way to have a delay on sidecar shutdown? (it didn't manage to find this in otel documentation)