[ext] Add instruction tracing to Orbetto

See #10.

.github/workflows/build.yml

@@ -8,7 +8,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - run: sudo apt-get update
-      - run: sudo apt-get install -y libusb-1.0-0-dev libzmq3-dev meson libsdl2-dev libdwarf-dev libdw-dev libelf-dev libcapstone-dev python3-pip ninja-build protobuf-compiler
+      - run: sudo apt-get install -y libusb-1.0-0-dev libzmq3-dev meson libsdl2-dev libdwarf-dev libdw-dev libelf-dev libcapstone-dev python3-pip ninja-build protobuf-compiler cmake
       - run: sudo pip3 install meson==1.2.0
       - uses: actions/checkout@v3
       - run: meson setup ./build
@@ -27,7 +27,7 @@ jobs:
     name: Build and Test Orbetto on macOS
     runs-on: macos-13
     steps:
-      - run: brew install zmq sdl2 libelf protobuf meson ninja capstone dwarfutils
+      - run: brew install zmq sdl2 libelf protobuf meson ninja capstone dwarfutils cmake
       - uses: actions/checkout@v3
       - run: meson setup ./build
         working-directory: ext/orbetto

.gitignore

@@ -8,6 +8,10 @@ dist/
 /ext/orbetto/subprojects/orbuculum
 /ext/orbetto/subprojects/libdwarf*
 /ext/orbetto/subprojects/packagecache
+/ext/orbetto/subprojects/CRoaring
 /ext/orbetto/*.perf
 /ext/orbetto/*.gz
 /ext/orbetto/*.debug
+/ext/orbetto/*.zip
+/ext/orbetto/*.roar
+/ext/orbetto/trace.swo

ext/orbetto/meson.build

@@ -15,6 +15,17 @@ perfbuf_sources = cpp_gen.process(proto_sources, preserve_path_from: meson.curre
 # Compile and link the protobuf source files
 perfbuf_lib = static_library('perfbuf', sources: perfbuf_sources, dependencies: [dependency('protobuf')])
 
+# Specify the include directory for Roaring Bitmap
+cmake = import('cmake')
+rm = cmake.subproject('croaring')
+# Fetch the dependency object
+message('CMake targets:\n - ' + '\n - '.join(rm.target_list()))
+rm_dep1 = rm.dependency('roaring-headers-cpp')
+rm_dep2 = rm.dependency('roaring-headers')
+rm_dep = rm.dependency('roaring')
+rm_inc = rm.get_variable('roaring_headers_cpp_inc') + rm.get_variable('roaring_headers_inc')
+
+
 # Clone the orbuculum git repository
 orbuculum = subproject('orbuculum')
 liborb = orbuculum.get_variable('liborb')
@@ -24,8 +35,8 @@ git_version_info_h = orbuculum.get_variable('git_version_info_h')
 # Compile and link everything together
 executable('orbetto',
     sources: ['src/orbetto.cpp', 'src/device.cpp', git_version_info_h],
-    include_directories: ['src'] + perfbuf_lib.private_dir_include() + orbinc,
-    dependencies: [dependency('protobuf')] + orbuculum.get_variable('dependencies'),
+    include_directories: ['src'] + perfbuf_lib.private_dir_include() + orbinc + rm_inc,
+    dependencies: [dependency('protobuf')] + orbuculum.get_variable('dependencies') + [rm_dep] + [rm_dep1] + [rm_dep2],
     link_with: [liborb, perfbuf_lib],
     install: true,
 )

ext/orbetto/metrics/METRICS.md

@@ -0,0 +1,68 @@
+# Trace Metrics with Plotly in Dash
+
+As explained in [TRACE.md](../src/TRACE.md), Perfetto is a very useful tool for
+displaying trace data, but is limited in data size and display variability.
+Therefore, as a proof of concept, some metrics are calculated in Python. We use
+the Protobuf file generated by Orbetto, the SQL trace processor from Perfetto
+and a program counter bitmap generated by CRoaring in Orbetto.
+
+
+## Visualising metrics
+
+To start the local dash server you need to pass a Protobuf file when analyzing
+ITM data or a bitmap for code coverage data. Depending on which argument you
+pass, different metrics can be displayed. Use `-h` to see all options. If you
+are using the `-diff` flag, you will need to provide a second Protobuf file to
+display the metrics based on the difference between the between the two files.
+This is very useful for comparing different versions or settings between two
+runs.
+
+```sh
+# cd orbetto/metrics
+# two metrics
+python3 metrics.py -f /path/to/orbetto.perf -wq -dwq
+# difference between two traces on one metric
+python3 metrics.py -f /path/to/orbetto_1.perf -f2 /path/to/orbetto_1.perf -hp -diff
+# Code coverage for pc bitmap (beta not completely accurate)
+python3 metrics.py -bm /path/to/bitmap.roar -elf /path/to/elf_file.elf -cc
+```
+
+Once executed, copy the URL into your browser and use the plotly interface to
+display the desired plot.
+
+> [!NOTE]  
+> Some metrics take a while to calculate (mainly code coverage).
+
+
+## Advanced metrics and goal
+
+There are already some simple statistical techniques included that can detect
+outliers, but the bigger picture of this work is to do regression testing and
+fuzzing with statistics derived from the instruction trace. For example:
+
+- [x] Regularity of sensor readings.
+- [ ] Comm link throughputs.
+- [x] Scheduling latency vs timeouts.
+- [ ] Thread progress vs semaphores.
+- [x] Callstack changes (code coverage).
+
+All these metrics need to be combined into a large latent space that can be
+diffed (a kind of distance measure) between different versions of PX4 to detect
+bugs in the code before it flies.
+
+
+## GCOV
+
+There has been a quick investigation to use a more advanced and also tested
+tool. The most common is gcov, which is a test coverage program from GCC. Two
+files are required for gcov to display code coverage:
+
+- `.gcno`, which is an advanced source file you get when compiling with the
+   GCC `-ftest-coverage` option.
+- `.gcda`, which is created when the GCC `-fprofile-arcs` option is run.
+
+It should be simple to get the .gcno file as the flag is already a compile
+option in PX4. Theoretically it should be possible to reverse engineer
+the `.gcda` file just from the PC values, ([documentation can be found here]
+(https://github.com/gcc-mirror/gcc/blob/master/gcc/gcov-io.h)) However, it is
+unclear how long this will take and how robust it will be.