Add Go error tracking

[marctc]

This PR adds the ability to automatically correct Go errors happening during an HTTP request.
The BPF probe only collects the last error, which is reported using fmt.Errorf, happened during
the request.

The stack trace and error message are used to add an exception event to the Span of the request:
https://opentelemetry.io/docs/specs/semconv/exceptions/exceptions-spans/

The env var BEYLA_TRACES_REPORT_EXCEPTION_EVENTS allows to enable this feature, which is
disabled by default. In a follow up PR i will add docs for config option and details of the feature.

Special thanks to @tpaschalis @inkel @rlankfo which contributed to this project.

Fixes #874

[grcevski]

Very cool! I think this will generally work! I left a few comments.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 67.34694% with 32 lines in your changes missing coverage. Please review.

Project coverage is 65.69%. Comparing base (e09e0df) to head (0dec945).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/internal/ebpf/common/spanner.go	42.10%	9 Missing and 2 partials ⚠️
pkg/internal/ebpf/gotracer/gotracer.go	46.66%	7 Missing and 1 partial ⚠️
pkg/internal/discover/attacher.go	33.33%	3 Missing and 1 partial ⚠️
pkg/internal/goexec/instructions.go	33.33%	4 Missing ⚠️
pkg/internal/ebpf/common/pids.go	82.35%	3 Missing ⚠️
pkg/internal/ebpf/httptracer/httptracer.go	0.00%	1 Missing ⚠️
pkg/internal/ebpf/tctracer/tctracer.go	0.00%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (e09e0df) and HEAD (0dec945). Click for more details.

HEAD has 2 uploads less than BASE

Flag	BASE (e09e0df)	HEAD (0dec945)
k8s-integration-test	1	0
integration-test	1	0

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1004       +/-   ##
===========================================
- Coverage   81.06%   65.69%   -15.38%     
===========================================
  Files         145      145               
  Lines       14558    14652       +94     
===========================================
- Hits        11802     9625     -2177     
- Misses       2188     4481     +2293     
+ Partials      568      546       -22     

Flag	Coverage Δ
integration-test	?
k8s-integration-test	?
oats-test	34.37% <54.08%> (-0.05%)	⬇️
unittests	51.81% <36.73%> (-0.18%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[grcevski]

There's one concern I came with thinking more about the symbol table. We essentially need to keep the symbol table live for the duration of the executable being instrumented. We should figure out how much memory is this for something reasonably large, like the OTel Collector, Alloy or Prometheus. If we were to instrument 10 applications, if this symbol table memory footprint is very large, we'll easily balloon the overall Beyla memory usage.

I'm not sure if there's a solution, but perhaps this can be optionally enabled for certain applications, if it's large memory footprint.

[mariomac]

It mostly LGTM, thanks for the great work!

I have a proposal for some improvements with respect to memory (as previously stated by @grcevski ). They don't have to be done in this PR but I'd suggest to do them in another PR.

1. Instead of directly storing the symtabs map in the PIDs filter, create an interface like:

type symtabs struct {
   Get(pid) *Symtab
   Put(pid, execPath, *Symtab)
   Delete(pid)
}

Then, provide two implementations, one using a hash map (to be used when BEYLA_TRACES_REPORT_EXCEPTION_EVENTS=true) and another that does nothing (to be used when BEYLA_TRACES_REPORT_EXCEPTION_EVENTS=false). That would prevent unnecessarily storing the symtabs when they we don't need them.

POST-EDIT: ah, I just saw that you already nil the symtabs if error reporting is disabled!

2. In the previous definition of Put, I added an executable Path, or something that uniquely identifies each symtab. That would allow to store only one symtab per executable. If I understood the code correctly, if Beyla instruments 4 gunicorn processes, it will store 4 copies of the gunicorn symtab, right? Then the Delete method would need to have a reference count for each symtab, to remove it only when all the pids pointing to it have been removed.

If I misunderstood the way the code works, please forget point 2.

[grcevski]

I think we need to check how this works with multiple processes launched from the same executable. I'm wondering if the symbol table will be meant for the address range of the initial executable which launched. What I mean by this is that I think you must know the executable base memory address to be able to do the PC to symbol translation, because technically the symbol table has only offsets from the executable start, but the PC that comes from eBPF is the absolute memory address of the instruction pointer.

Let's take the following example:

1. Imagine we have a symbol table with one function foo() at offset 0x100.
2. We launch the executable once and its base code virtual address is 0x20000.
3. We launch the executable one more time and this one starts at virtual address 0x30000.
4. We get exception at 0x200105 and then at 0x300105. Technically that's exception at two different addresses, but it's the same exception in two different processes.

We'll only parse the symbol table once, since we detect that the executable is already instrumented.

I think this code translation must take into account the base memory address of the executable, I think the symbol table is not sufficient.