Idea: Cache subqueries the same way as range queries

[julienduchesne]

What is the problem you are trying to solve?

When querying an aggregated subquery as an instant query:

max_over_time({{query}}[7d:1h])

or the same ``{{query}}` as a range query, there is a dramatic different in performance. In my own use case, the aggregated instant query takes around 1m50s and the range query takes ~1-5s, while they are querying the same data (minus a max operation which should take microseconds)

Which solution do you envision (roughly)?

This difference in runtime is due to the fact that range queries are incrementally cached, while subqueries (as part of instant queries) are not cached at all (see #3815)

Using the range query cache for subqueries could lead to performance improvements, especially for ruler queries which often do aggregations over subqueries and are all instant queries

Have you considered any alternatives?

No response

Any additional context to share?

No response

How long do you think this would take to be developed?

Not sure

What are the documentation dependencies?

No response

Proposer?

@julienduchesne

[julienduchesne]

Note that, following investigation, this is pretty tricky to do because we cannot arbitrarily inject precomputed results into promql queries. The range queries can be easily cached because this is done at the root level of the query based on the query time range.

The instant query subqueries are embedded within the queries. That means that to inject data, one would need to do something similar to the sharded querying done in range queries where a special selector replaces the actual query: https://github.com/grafana/mimir/blob/main/pkg/frontend/querymiddleware/astmapper/embedded.go#L15-L32. It may be possible but I'm not entirely sure