Generate better bug reports for users

[tgodzik]

Inspired by #4902 I think we try to identify more places where we could save a snapshot of the workspace for the users to be able to file better issues.

We should:

1. Looks at the places where any logs are produced and see if we can find more information about the current state of Metals.
2. Save that data to .metals/reports
3. Enable user to have a report.zip easily created via command (can be added as a button to treeview) with all the sensible data anonymized (we can replace workspace by $workspace etc.)
4. Add a step in the bug report template for the users to upload the report zip

Opinions?

@kpodsiad was thinking of an additional red status bar icon for VS Code, which maybe could also invoke the command?

[keynmol]

Just as a thought, but specifically for this usecase I've been developing a snapshot/replay functionality in Langoustine tracer.

The idea is that a user can "restart with tracing" and capture the problematic usecase, and then download a snapshot in viewable JSON format.

@ckipp01 already played with this idea in nvim-metals: scalameta/nvim-metals#514

This won't capture Metals' internal state, of course, but it can go quite far.

Tracer itself can even be embedded into Metals.

[tgodzik]

@keynmol that could be cool, though I think here we should tackle the cases where users might not be able to reproduce the issue or we might want to add some specific internal data.

For langoustine, could it read the plain trace files that we have in Metals? We already trace whenever .metals/lsp.trace.json is created, so this could be that Restart with tracing option. We could adjust the format for langoustine if needed, since we are not tied to a particular one.

[keynmol]

if lsp.trace.json becomes actual json it can certainly be possible - I'd be happy to work on a format that fits both.

Current message format is similar to this:

{
  "type": "Message",
  "rm": {
    "timestamp": 1671655999047,
    "raw": {
      "jsonrpc": "2.0",
      "method": "textDocument/didOpen",
      "params": {
        "textDocument": {
          "text": "worsts = (5 + 1)\nx = (worsts + 11)\nz = 17\ntest = (x * 1)\nh = (x * (11 + worsts))\nhs = (x * (z + worsts))\nts = 25\na = 11\nr = (x + a)\n\n",
          "languageId": "quickmaffs",
          "version": 0,
          "uri": "file:///Users/velvetbaldmime/projects/quickmaffs/examples/test.qmf"
        }
      }
    },
    "decoded": {
      "type": "Notification",
      "generatedId": "generated-bd54046b-4430-44de-a93f-d90b5072ff1d",
      "method": "textDocument/didOpen",
      "direction": {
        "type": "ToServer"
      }
    }
  }
}

But it's only this way because that's how it's dumped from case classes - I can certainly amend it to be a lot denser.

With regards to extra information - client can just send a special metals/debug command or whatever and the result (internal state) will be captured in the request/response view - don't even have to change anything in tracing for that.

[keynmol]

• the whole type: Message is there to differentiate what each json line means - can add something like type: Metadata which will just carry any JSON payload (=internal state) you want rendered in the viewer.

[keynmol]

We can call it LSPTP :D Language Server Protocol Trace Protocol. More protocols!

[matthughes]

Could you buffer the last N LSP/BSP messages in memory and when touching the file (or at user request), dump them to disk?

[matthughes]

Not sure but seems like a good feature to add to scribe: outr/scribe#394

[jdegoes]

I think this is an excellent idea!

I would suggest that in addition to log files (and perhaps tracing), the following data could be collected that would prove useful in diagnosing certain classes of problems:

• Thread dumps, which can be useful to catch deadlocks and other concurrency issues
• Fiber dumps (although this would apply more to Bloop if migrated to CE3 or ZIO)
• JMX metrics, which can help diagnose a wide range of low-level issues (memory/CPU leaks)

I believe that making it easy to submit diagnostic information will increase the number of reports. But I also have a slight concern about security: in gathering logs, at minimum, path information will be captured, including the names of files that occur in commercial code bases; and including a personal identifier (home path) for the user generating the report.

It could be useful to do scrubbing on all logs and traces to remove such PII to avoid leaking information that could compromise an individual submitter or a company employing such an individual.

[SethTisue]

@smarter mentioned the existence of https://github.com/lampepfl/lsp-viewer — when he was developing the old Dotty LSP Server, EPFL students experiencing trouble could enable logging which he'd analyze in the viewer. (without anonymization since it was students doing assignments and it was opt-in)

[tgodzik]

Reporting has been added, there will be also an opt in report gathering in the near future.