sidecar --min-time not honoured when object storage not configured

[milan-holub]

Thanos, Prometheus and Golang version used:

thanos, version 0.34.1 (branch: HEAD, revision: 4cf1559998bf6d8db3f9ca0fde2a00d217d4e23e)
  build user:       root@61db75277a55
  build date:       20240219-17:13:48
  go version:       go1.21.7
  platform:         linux/amd64
  tags:             netgo

prometheus, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
  build user:       root@42454fc0f41e
  build date:       20230725-12:31:24
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Object Storage Provider:
None

What happened:
Running sidecar with --min-time=-3h flag without object storage configured:

  - args:
    - sidecar
    - --min-time=-3h
    - --prometheus.url=http://127.0.0.1:9090/
    - --grpc-address=:12901
    - --http-address=:12902
    - --log.level=info
    - --log.format=json

Having query configured against this sidecar:

   ...
    --endpoint=dnssrv+_grpc-sidecar-2._tcp.observability-thanos-discovery.monitoring.svc

(verified the dnssrv+ is correctly resolved to the sidecar)
When sidecar is started it correctly sets the relative minimum time as it can be then seen from the query UI /stores endpoint

and the queried data are limited to that time.
Unfortunately the minimum time is never updated and as the time passes data can be still queried up to the fixed point in time of <start of the sidecar - 3h> even after days.

What you expected to happen:

• I'd expect that the --min-time flag is honoured as per documentation:

Start of time range limit to serve. Thanos sidecar will serve only metrics, which happened later than this value.

• to get only the last 3 hours of data anytime

How to reproduce it (as minimally and precisely as possible):
configure sidecar and query as per above

Full logs to relevant components:
No specific log records identified.

Anything else we need to know:
Based on this comment in the code it seems to be like a forgotten TODO?

Bumped into this for the use-case of having multiple sidecars per prometheus pod for performance/sizing reasons (separate sidecar for uploads to object storage and another one for grpc query).

[dosubot]

Hey @milan-holub! I'm here to assist you with bugs, questions, and becoming a contributor 🤖

The issue you're experiencing with the --min-time flag not updating dynamically in the Thanos sidecar when object storage is not configured is due to the fact that the minimum timestamp is never adjusted if shipping is disabled. This behavior is noted in the code as a "TODO" item, indicating that it's a known limitation:

// TODO: minimum timestamp is never adjusted if shipping is disabled.

When object storage is not configured, uploads are disabled, which means the dynamic adjustment of the --min-time flag does not occur. The flag is primarily used in conjunction with object storage to manage data sharding and retrieval, and without object storage, this dynamic behavior is not applicable ^[1][2].

To continue talking to Dosu, mention @dosu.

[milan-holub]

Great summary from dosubot but I think that there are also use-cases out of the object storage upload area. Limiting query results is one of those. Any ideas? Is there a chance that it will be implemented?

[GiedriusS]

Is it the same with the newest version?