"Local profile (diff)" command fails (with valgrind 3.19.0)

[RalfJung]

I am trying to run the "Local profile (diff)" command given on this page. Unfortunately it doesn't work:

$ ./target/release/collector profile_local cachegrind +3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa --rustc2 +94b77248c5aa991f2597b4a1c1a49e297ac2f3e2 --include ctfe-stress-5 --profiles Debug --scenarios Full
Running with 1 job(s)
detecting the channel of the `3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa` toolchain...
downloading <https://ci-artifacts.rust-lang.org/rustc-builds/3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa/rustc-nightly-x86_64-unknown-linux-gnu.tar.xz>...
75.26 MB / 75.26 MB [===============================================================================================================================================================================================================================================] 100.00 % 11.18 MB/s 
downloading <https://ci-artifacts.rust-lang.org/rustc-builds/3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz>...
27.80 MB / 27.80 MB [===============================================================================================================================================================================================================================================] 100.00 % 11.16 MB/s 
toolchain `3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa` is successfully installed!
Profiling 3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa with Cachegrind
Executing benchmark ctfe-stress-5 (1/1)
Preparing ctfe-stress-5
Running ctfe-stress-5: Debug + [Full]
Finished benchmark ctfe-stress-5 (1/1)
collector error: Failed to profile 'ctfe-stress-5' with Cachegrind, recorded: expected success, got exit status: 101

stderr=   Compiling ctfe-stress-5 v0.1.0 (/tmp/.tmprorwj9)
==92466== Cachegrind, a cache and branch-prediction profiler
==92466== Copyright (C) 2002-2017, and GNU GPL'd, by Nicholas Nethercote et al.
==92466== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==92466== Command: /home/r/.rustup/toolchains/3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa/bin/rustc --crate-name ctfe_stress_5 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=60c11a1ed113cdf7 -C extra-filename=-60c11a1ed113cdf7 --out-dir /tmp/.tmprorwj9/target/debug/deps -L dependency=/tmp/.tmprorwj9/target/debug/deps -Ctarget-cpu=native -Adeprecated -Aunknown-lints -Zincremental-verify-ich
==92466== 
--92466-- warning: L3 cache found, using its data for the LL simulation.
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x23
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x23
==92466== Valgrind: debuginfo reader: ensure_valid failed:
==92466== Valgrind:   during call to ML_(img_get)
==92466== Valgrind:   request for range [2334864704, +4) exceeds
==92466== Valgrind:   valid image size of 2821480 for image:
==92466== Valgrind:   "/home/r/.rustup/toolchains/3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa/bin/rustc"
==92466== 
==92466== Valgrind: debuginfo reader: Possibly corrupted debuginfo file.
==92466== Valgrind: I can't recover.  Giving up.  Sorry.
==92466== 
thread 'main' panicked at 'command did not complete successfully: MALLOC_CONF="dirty_decay_ms:0,muzzy_decay_ms:0" RUSTC_FORCE_INCR_COMP_ARTIFACT_HEADER="rustc-perf" RUSTC_FORCE_RUSTC_VERSION="rustc-perf" "valgrind" "--tool=cachegrind" "--cache-sim=no" "--branch-sim=no" "--cachegrind-out-file=cgout" "/home/r/.rustup/toolchains/3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa/bin/rustc" "--crate-name" "ctfe_stress_5" "src/lib.rs" "--error-format=json" "--json=diagnostic-rendered-ansi,artifacts,future-incompat" "--crate-type" "lib" "--emit=dep-info,metadata,link" "-C" "embed-bitcode=no" "-C" "debuginfo=2" "-C" "metadata=60c11a1ed113cdf7" "-C" "extra-filename=-60c11a1ed113cdf7" "--out-dir" "/tmp/.tmprorwj9/target/debug/deps" "-L" "dependency=/tmp/.tmprorwj9/target/debug/deps" "-Ctarget-cpu=native" "-Adeprecated" "-Aunknown-lints" "-Zincremental-verify-ich"', collector/src/bin/rustc-fake.rs:24:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: could not compile `ctfe-stress-5` (lib)


 stdout=

detecting the channel of the `94b77248c5aa991f2597b4a1c1a49e297ac2f3e2` toolchain...
downloading <https://ci-artifacts.rust-lang.org/rustc-builds/94b77248c5aa991f2597b4a1c1a49e297ac2f3e2/rustc-nightly-x86_64-unknown-linux-gnu.tar.xz>...
73.95 MB / 73.95 MB [================================================================================================================================================================================================================================================] 100.00 % 8.94 MB/s 
downloading <https://ci-artifacts.rust-lang.org/rustc-builds/94b77248c5aa991f2597b4a1c1a49e297ac2f3e2/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz>...
27.78 MB / 27.78 MB [================================================================================================================================================================================================================================================] 100.00 % 8.36 MB/s 
toolchain `94b77248c5aa991f2597b4a1c1a49e297ac2f3e2` is successfully installed!
Profiling 94b77248c5aa991f2597b4a1c1a49e297ac2f3e2 with Cachegrind
Executing benchmark ctfe-stress-5 (1/1)
Preparing ctfe-stress-5
Running ctfe-stress-5: Debug + [Full]
Finished benchmark ctfe-stress-5 (1/1)
collector error: Failed to profile 'ctfe-stress-5' with Cachegrind, recorded: expected success, got exit status: 101

stderr=   Compiling ctfe-stress-5 v0.1.0 (/tmp/.tmpyrVw1y)
==92495== Cachegrind, a cache and branch-prediction profiler
==92495== Copyright (C) 2002-2017, and GNU GPL'd, by Nicholas Nethercote et al.
==92495== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==92495== Command: /home/r/.rustup/toolchains/94b77248c5aa991f2597b4a1c1a49e297ac2f3e2/bin/rustc --crate-name ctfe_stress_5 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=60c11a1ed113cdf7 -C extra-filename=-60c11a1ed113cdf7 --out-dir /tmp/.tmpyrVw1y/target/debug/deps -L dependency=/tmp/.tmpyrVw1y/target/debug/deps -Ctarget-cpu=native -Adeprecated -Aunknown-lints -Zincremental-verify-ich
==92495== 
--92495-- warning: L3 cache found, using its data for the LL simulation.
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x23
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x23
==92495== Valgrind: debuginfo reader: ensure_valid failed:
==92495== Valgrind:   during call to ML_(img_get)
==92495== Valgrind:   request for range [2334864704, +4) exceeds
==92495== Valgrind:   valid image size of 2821480 for image:
==92495== Valgrind:   "/home/r/.rustup/toolchains/94b77248c5aa991f2597b4a1c1a49e297ac2f3e2/bin/rustc"
==92495== 
==92495== Valgrind: debuginfo reader: Possibly corrupted debuginfo file.
==92495== Valgrind: I can't recover.  Giving up.  Sorry.
==92495== 
thread 'main' panicked at 'command did not complete successfully: MALLOC_CONF="dirty_decay_ms:0,muzzy_decay_ms:0" RUSTC_FORCE_INCR_COMP_ARTIFACT_HEADER="rustc-perf" RUSTC_FORCE_RUSTC_VERSION="rustc-perf" "valgrind" "--tool=cachegrind" "--cache-sim=no" "--branch-sim=no" "--cachegrind-out-file=cgout" "/home/r/.rustup/toolchains/94b77248c5aa991f2597b4a1c1a49e297ac2f3e2/bin/rustc" "--crate-name" "ctfe_stress_5" "src/lib.rs" "--error-format=json" "--json=diagnostic-rendered-ansi,artifacts,future-incompat" "--crate-type" "lib" "--emit=dep-info,metadata,link" "-C" "embed-bitcode=no" "-C" "debuginfo=2" "-C" "metadata=60c11a1ed113cdf7" "-C" "extra-filename=-60c11a1ed113cdf7" "--out-dir" "/tmp/.tmpyrVw1y/target/debug/deps" "-L" "dependency=/tmp/.tmpyrVw1y/target/debug/deps" "-Ctarget-cpu=native" "-Adeprecated" "-Aunknown-lints" "-Zincremental-verify-ich"', collector/src/bin/rustc-fake.rs:24:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: could not compile `ctfe-stress-5` (lib)


 stdout=

Cannot open results/cgout-3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa-ctfe-stress-5-Debug-Full for reading
collector error: Cannot run cg_diff

Caused by:
    failed to generate cachegrind diff
Diffs:
collector error: 3 benchmarks failed

Am I doing something wrong?

This is on Debian testing with all apt-updates installed.

$ valgrind --version
valgrind-3.19.0

Is that valgrind too old, or so?

[Kobzol]

You will indeed need a newer Valgrind version, which understands the DWARF version used by Rust. This is sadly a common error, because such a version of Valgrind is not yet available in commonly used distributions, AFAIK.

I usually just use a trunk version of Valgrind:

$ git clone https://sourceware.org/git/valgrind.git
$ cd valgrind
$ ./autogen.sh
$ ./configure
$ make
($ sudo make install)

I'm not sure which Valgrind version is "new enough" for it to work, maybe 3.20 or 3.21. @nnethercote might know.

[RalfJung]

Yeah I found rust-lang/rust#98746. I don't have the time to mess with building valgrind from source though, I'll just have to hope someone else can get the perf diffs for me when it comes up in my PRs...

[Kobzol]

In my experience, building Valgrind is quite straightforward and has worked fine without any issues, but YMMV, as always.

Here is the diff from your PR description:

Diff

--------------------------------------------------------------------------------
Files compared:   results/cgout-3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa-ctfe-stress-5-Debug-Full; results/cgout-94b77248c5aa991f2597b4a1c1a49e297ac2f3e2-ctfe-stress-5-Debug-Full
Command:          /home/kobzol/.rustup/toolchains/3071e0aef6dfd0a150c3fb1da0abad4ec86ca0aa/bin/rustc --crate-name ctfe_stress_5 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=60c11a1ed113cdf7 -C extra-filename=-60c11a1ed113cdf7 --out-dir /tmp/.tmpsWH5k3/target/debug/deps -L dependency=/tmp/.tmpsWH5k3/target/debug/deps -Adeprecated -Aunknown-lints -Zincremental-verify-ich; /home/kobzol/.rustup/toolchains/94b77248c5aa991f2597b4a1c1a49e297ac2f3e2/bin/rustc --crate-name ctfe_stress_5 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=60c11a1ed113cdf7 -C extra-filename=-60c11a1ed113cdf7 --out-dir /tmp/.tmptkic2g/target/debug/deps -L dependency=/tmp/.tmptkic2g/target/debug/deps -Adeprecated -Aunknown-lints -Zincremental-verify-ich
Data file:        /tmp/.tmpdQxwTM
Events recorded:  Ir
Events shown:     Ir
Event sort order: Ir
Thresholds:       0.1
Include dirs:     
User annotated:   
Auto-annotation:  on

--------------------------------------------------------------------------------
Ir          
--------------------------------------------------------------------------------
755,673,586  PROGRAM TOTALS

--------------------------------------------------------------------------------
Ir             file:function
--------------------------------------------------------------------------------
  955,939,620  ???:<rustc_const_eval::interpret::eval_context::InterpCx<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>>::storage_live_for_always_live_locals
 -824,300,678  ???:<rustc_const_eval::interpret::eval_context::InterpCx<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>>::push_stack_frame
  540,667,487  ???:<rustc_const_eval::interpret::eval_context::InterpCx<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>>::statement
  104,233,168  ???:<rustc_const_eval::interpret::eval_context::InterpCx<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>>::pass_argument::<core::iter::adapters::filter::Filter<core::iter::adapters::zip::Zip<core::slice::iter::Iter<rustc_const_eval::interpret::terminator::FnArg>, core::slice::iter::Iter<rustc_target::abi::call::ArgAbi<rustc_middle::ty::Ty>>>, <rustc_const_eval::interpret::eval_context::InterpCx<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>>::eval_fn_call::{closure#2}>>
  -69,366,384  ???:<rustc_const_eval::interpret::eval_context::InterpCx<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>>::eval_place
   21,495,808  ???:<rustc_const_eval::interpret::place::PlaceTy as rustc_const_eval::interpret::projection::Projectable<rustc_middle::mir::interpret::AllocId>>::to_op::<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>
   18,260,032  ???:<rustc_const_eval::interpret::eval_context::InterpCx<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>>::terminator
    6,973,377  ???:<rustc_const_eval::interpret::eval_context::InterpCx<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>>::eval_fn_call
    3,631,214  ???:rustc_const_eval::const_eval::eval_queries::eval_to_allocation_raw_provider
   -2,202,083  ???:<rustc_const_eval::const_eval::machine::CompileTimeInterpreter as rustc_const_eval::interpret::machine::Machine>::find_mir_or_eval_fn
    1,441,368  ???:<rustc_const_eval::interpret::eval_context::InterpCx<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>>::eval_operand
   -1,310,450  ./string/../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:__memcpy_avx_unaligned_erms
      845,150  ???:<rustc_type_ir::sty::TyKind<rustc_middle::ty::context::TyCtxt> as rustc_serialize::serialize::Decodable<rustc_metadata::rmeta::decoder::DecodeContext>>::decode

--------------------------------------------------------------------------------
The following files chosen for auto-annotation could not be found:
--------------------------------------------------------------------------------
  ./string/../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S

Let me know e.g. on Zulip (Jakub Beránek) if you want me to run any diffs for you, anytime!

[RalfJung]

Here is the diff from your PR description:

Thanks a ton. :-) <3

[nnethercote]

I think Valgrind 3.20 (from October 2022) should be new enough.

I don't have the time to mess with building valgrind from source though

Jakub gave the instructions above. It's quick to build, just a minute or two. If you don't want to make install you can run the vg-in-place binary that gets built in the repo root dir.

[RalfJung]

Yeah make install is something I'll definitely not do, I've messed up my system too many times with commands like that. dpkg is the only tool allowed to write to /usr. So I'd have to get ./target/release/collector to somehow use the not-in-PATH valgrind or so?

[Kobzol]

You can do the good ol' local install + PATH override:

$ ./configure --prefix=my-install-dir
$ make install

and then

$ PATH="my-install-dir/bin:${PATH}" ./target/release/collector