Add profiling labels to ingester's read path

[colega]

What this PR does

Adds labels to profiling that identify the read requests.

We're already adding the trace ID when the request is sampled, this would also add the method, userID and matchers of that read request.

This should help debugging issues when ingesters are busy running heavy
queries.

Checklist

• [na] Tests updated.
• [na] Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• [na] about-versioning.md updated with experimental features.

[simonswine]

Unfortunately we cannot handle that amount cardinality for any profiles label. We currently handle the span_id label differently than any other label, so that assumption is not true:

We're already adding the trace ID when the request is sampled, this would also add the method, userID and matchers of that read request.

If that PR should go forward I suggest to be able to control its behaviour at runtime: While we might be able to support e.g. this being enabled for a few tenants, enabling it across the board for all cells will exceed the cardinality we can handle at present.

[colega]

Is there any way you (Pyroscope) could just not store those labels? Maybe we can make that configurable?

I don't really need (it would be nice, but definitely not a requirement to get meaningful results) to aggregate these labels over time. During an incident it's often enough to take a look at one or maybe 2 profiles to understand what's going on. If you skipped those labels, we could still do a good-old /debug/pprof/profile call and visualize them (once go tool pprof properly renders, this of course, as there's a bug handling quotes right now, which is the real blocker for this PR)

[simonswine]

As a first step I am working on relabeling incoming profiles on our distributors. This will allow us to drop extra labels and potentially control selctively which ones to keep.

This could in the future also allow us to do some more to it, like convert the total profile sample values as a metric. I think your usecase is very valid and something we should incorporate into Pyroscope.

[dimitarvdimitrov]

The CHANGELOG has just been cut to prepare for the next Mimir release. Please rebase main and eventually move the CHANGELOG entry added / updated in this PR to the top of the CHANGELOG document. Thanks!