RFC: Link Application Metrics with Chaos Experiments #38
Conversation
Signed-off-by: STRRL <[email protected]>
Signed-off-by: STRRL <[email protected]>
STRRL
changed the title
text/2022-02-14-link-application-metrics-with-chaos-experiments.md
Outdated
Show resolved
Hide resolved
text/2022-02-14-link-application-metrics-with-chaos-experiments.md
Outdated
Show resolved
Hide resolved
text/2022-02-14-link-application-metrics-with-chaos-experiments.md
Outdated
Show resolved
Hide resolved
| This feature is all about Chaos Dashboard, would not affect anything with other | ||
| components and CRD. | ||
|
|
||
| We would use Grafana as the source of application metrics, and use Grafana Panel |
There was a problem hiding this comment.
This seems weird. If the goal is to correlate chaos experiments with user applications, then why we don't just correlate with the metrics directly? Grafana itself is the UI/display layer.
There was a problem hiding this comment.
I will try to answer all of the following questions together, but my understanding may not be as accurate as @STRRL and @cwen0, feel free to @ them.
Overall, this is more of an exploration. The goal is to bring users an integrated interface that includes experiments, workflows, and panels rendered from data obtained through the grafana API.
To achieve this, we want to be able to fetch the grafana panel data through the grafana API and then customize the rendering in the chaos dashboard. We will take the panel as the smallest unit and store it as a resource so that any chaos can associate the panel and combine their active time with the original panel data.
Users can simply click Share this panel, paste the link into the chaos dashboard's interface, and associate anything Chaos Mesh related with the panel's data.
It's an all-in-one experience where users no longer need to install plugins, configure the API address, and complicate their panels.
But that doesn't mean that we're going to abandon grafana datasource, we want to make it more of a community-driven project, and any good ideas and suggestions we have will be added to the datasource plugin if any.
There was a problem hiding this comment.
But I still want to make the point that it is extremely difficult to get close to the grafana ecosystem by way of plugins, and as someone who has had hands-on experience, writing plugins is painful (there is no extensive documentation, and I can only try to write them by referring to the source code of some plugins). I think this will make it fundamentally difficult to attract more people to participate in plugin development.
| draw charts by ourself. The most important problem is that we also provide the | ||
| feature about "exploring the query" and "design panels and dashboards" like | ||
| Grafana, for user to tuning their charts. | ||
| - Choose other Grafana alternative integration, like Datadog, Elastic stack, New |
There was a problem hiding this comment.
Grafana is the only frontend UI. But we are comparing it with Datadog, Elastic, which are the whole observability stack, including UI, metrics collecting, storage, etc. I don't think these are good examples here. Does it make sense to compare between these?
text/2022-02-14-link-application-metrics-with-chaos-experiments.md
Outdated
Show resolved
Hide resolved
|
|
||
| We already have a grafana plugin as mark annotations on grafana dashboard. There | ||
| are also feature overlapping between that grafana plugin and this new feature. | ||
| It might take more effort to keep behaviors consistent. |
There was a problem hiding this comment.
Why not add the link feature in the Grafana datasource? Like currently how trace, exemplars, metrics and logs correlations are implemented? This seems less effort and can do the same thing.
Co-authored-by: Siyu Chen <[email protected]> Signed-off-by: STRRL <[email protected]>
STRRL
force-pushed
the
link-application-metrics
branch
from
a64133b
to
0eab044
Compare
Signed-off-by: STRRL <[email protected]>
Signed-off-by: STRRL <[email protected]>
|
|
||
| Pros: | ||
|
|
||
| - It would prevent data loss when the Pod chaos-dashboard is deleted/recreated. |
There was a problem hiding this comment.
Maybe it's time to reconsider how to treat our data in chaos-dashboard?
Was it designed as persisted or volatile? @g1eny0ung
There was a problem hiding this comment.
It's hard for me to give an idea right now because it seems that both approaches have their advantages and disadvantages. I'll give it some more thought. 🤔
Signed-off-by: STRRL <[email protected]>
Signed-off-by: STRRL <[email protected]>
Signed-off-by: STRRL [email protected]
rendered markdown