Improve UX for plots debugging

[mnrozhkov]

Why it's important?

I spent some time helping a few teams with plot debugging. I helped multiple teams debug their plots. When users manually configure plots in dvc.yaml, two types of errors are likely to appear:

1. Provided data and configuration are not compatible with the plot template (e.g pass str to numerical data type)
2. Typos in file names

In every case, both DVC/DVCLive are "silent". We discussed here iterative/dvclive#750 and plan to improve.

Eventually, users open VSCode extension and don't see plots at all! This issue aims to find a better way to notify users about problems with plots and metrics visualization in VSCode.

AS IS

(I don't consider VSCode extension logs, because they are not appropriate for end users)

1. Error messages on hovering top-level tiles
At this moment we provide little means for understanding the error and debugging. As an example, out internal discussion in Slack

It seems that the best means we have now is error messages in tiles with branch/exp name:

However

• It's difficult to find (found this feature accidentally when trying to find smth)
• it was not expected that tiles would show additional information for ! sign
• for a customer who used VSCode via Dev Container to run/debug code in the remote machine, there was no message on hover ! - but only copy icon. Is it expected?

2. Error messages on hovering side menu

• I was not aware of them
• Takes too long to show the message on hover

DISCUSSION

1. Can we show an "empty" plot for every buggy plot specified in dvc.yaml and show an error message on it?

2. Is there another way to help with plot debugging?
3. Can we improve documentation to mention existing features for error messages?
4. When people open Show Plots for the first time, there is nothing inside because no experiments are selected
   • can we select "workspace" by default? I guess it's needed most of the time.

[shcheklein]

Related or even can absorb #4333 .

Can we be related to DVC not providing enough information.

[julieg18]

2. Error messages on hovering side menu

• I was not aware of them
• Takes too long to show the message on hover

We have no control over the side menu tooltips design but we can alter the plots webview.

1. Can we show an "empty" plot for every buggy plot specified in dvc.yaml and show an error message on it?

Possibly if we have errors that are plot-specific. But we wouldn't be able to show an "empty" plot as that would be difficult to render. We may be able to show a "placeholder" plot. May seem a bit noisy though.

When people open Show Plots for the first time, there is nothing inside because no experiments are selected
can we select "workspace" by default? I guess it's needed most of the time.

This is doable but it adds a layer of complexity to the experiment selection code, so it's probably not worth implementing.

Is there another way to help with plot debugging?
Can we improve documentation to mention existing features for error messages?

As discussed synchronously, we can improve error messaging in the plots webview since currently all we have is the! in ribbon blocks which is pretty subtle. I'll post some ideas.

[julieg18]

Tested some designs ideas on updating the ribbon blocks, section titles, and adding plot placeholders.

Plot Ribbons

We can update the icon into an error icon instead of an exclamation mark.

We can also fill the text as red but that may get a bit noisy with a project wide error:

Section Title

Similar to ribbon blocks, we can adjust text and icons to show there are errors within the section. Hovering over would show a tooltip message with the error.

Plot Error Placeholders

Unlike the previous examples which are rather simple to create, I believe this would take some more time since we would need to change how our backend sends plots.

What to Implement

I think we should

• update the ribbon blocks, making the text red and adding the error icon
• update the section title icon to an error, showing the error in the tooltip that appears on hover

What does everyone else think? cc @shcheklein

[mnrozhkov]

Thank you @julieg18! I really like your suggestions! 🔥 👍

[julieg18]

As discussed in #5145, we're going to leave the ribbon as is and focus on adding error messaging in the plot section.

Can we take the idea from Studio (@sroy3 is working on errors / warnings now) - show errors in the header of the section, show a sign (yellow, green, red) nearby the section title (similar to setup screen)?
@shcheklein in #5145 (comment)

Here's an example of what Studio errors look like:

I believe that Studio doesn't mention what commit the error came from, just plot name and error message. I'm assuming we want to show revision as well as plot name, but that does make errors more complicated.

Here's one design idea on what our error message would look like (open to suggestions 😅):

This error block would also have a collapsible option so the message doesn't get too long. What do we think? cc @shcheklein

[shcheklein]

I like it! :) thanks, @julieg18

[julieg18]

As discussed in #5145, we're going to leave the ribbon as is and focus on adding error messaging in the plot section.

While working on this, I realized that it's not currently doable due to dvc plots diff not giving us the plot type of errored-out plots (see #5152 (comment)). Since we don't have the plot type, we don't know what plot section the error actually belongs in.

Discussed this issue with @sroy3, and we came up with a new idea that's similar to Studio's current design. We could add a banner to the ribbon, notifying the user of errors, and once clicked on, it would open up a popup for viewing errors:

This idea is not section-specific so we can show it even if we don't know broken plot types and keeps the possibly lengthy errors from blocking the screen. What do we think? cc @shcheklein

[shcheklein]

@julieg18 yes, that's totally fine to start with, thanks!

[julieg18]

#5165 is now merged, making errors more obvious for users. Since we are unable to detect what plot types broken plots are (dvc plots diff does not give us this information), we currently can't implement other solutions like section messaging or "plot placeholders".

With improving documentation covered in #5057, I think we're good to close this issue for now.

[mnrozhkov]

Thank you @julieg18 🤝