Gem commonmarker 1.x fails to compile native extensions with rust #3396

rwstauner · 2024-01-18T23:28:00Z

Using the latest commit and a jvm-ce build

truffleruby 24.1.0-dev-00307c24, like ruby 3.2.2, GraalVM CE JVM [x86_64-linux]

with the latest gem version (1.0.4)
installation fails due to several items missing from RbConfig::CONFIG:

error: failed to run custom build command for `rb-sys v0.9.85`

Caused by:
  process didn't exit successfully: `/home/spin/.gem/truffleruby/3.2.2/gems/commonmarker-1.0.4/ext/commonmarker/target/release/build/rb-sys-4475e29d236d2359/build-script-main` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=RUBY
  cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING
  cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION
  cargo:rerun-if-env-changed=RBCONFIG_MAJOR

  --- stderr
  thread 'main' panicked at /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-build-0.9.85/src/rb_config.rs:206:32:
  Key not found: MAJOR

If I add these values (and use something like export TRUFFLERUBYOPT=-r$PWD/config) I get past those issues.

RbConfig::CONFIG["MAJOR"] = "3"
RbConfig::CONFIG["MINOR"] = "2"
RbConfig::CONFIG["TEENY"] = "1"
RbConfig::CONFIG["PATCHLEVEL"] = "1"
RbConfig::CONFIG["archincludedir"] = "/tmp/archincludedir"

Then I get other errors:

error[E0432]: unresolved import `crate::rb_gc_adjust_memory_usage`
 --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/tracking_allocator.rs:3:13
  |
3 | use crate::{rb_gc_adjust_memory_usage, utils::is_ruby_vm_started};
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^ no `rb_gc_adjust_memory_usage` in the root

error[E0412]: cannot find type `RData` in this scope
-->
/home/spin/.gem/truffleruby/3.2.2/gems/commonmarker-1.0.4/ext/commonmarker/target/release/build/rb-sys-f73ca150f0763414/out/bindings-0.9.85-x86_64-linux-3.2.1.rs:3478:51
     |
3478 |         pub fn rb_tr_rdata(object: VALUE) -> *mut RData;
     |                                                   ^^^^^ not found in this scope

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:19:35
   |
19 |         let flags = rstring.basic.flags;
   |                                   ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:34:35
   |
34 |         let flags = rstring.basic.flags;
   |                                   ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:52:34
   |
52 |         let flags = rarray.basic.flags;
   |                                  ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:56:38
   |
56 |             let mut f = rarray.basic.flags;
   |                                      ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:70:34
   |
70 |         let flags = rarray.basic.flags;
   |                                  ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:94:35
   |
94 |         let ret: u32 = ((*rbasic).flags & crate::ruby_value_type::RUBY_T_MASK as VALUE) as _;
   |                                   ^^^^^ unknown field

Some errors have detailed explanations: E0412, E0432, E0609.
For more information about an error, try `rustc --explain E0412`.
error: could not compile `rb-sys` (lib) due to 8 previous errors
warning: build failed, waiting for other jobs to finish...
make: *** [Makefile:377: target/release/libcommonmarker.so] Error 101

make failed, exit code 2

The text was updated successfully, but these errors were encountered:

jac241 · 2024-09-02T05:58:40Z

Gems that use the rb-sys for rust don't appear to work. I'm running into an identical issue with truffleruby 24.0.2 trying to install the polars-df gem (https://github.com/ankane/ruby-polars) which also uses the rb-sys crate.

eregon · 2024-09-12T15:58:46Z

I have heard @nirvdrum has been looking at this recently.
In general TruffleRuby should be able to support Rust extensions easier now that extensions are run natively.
However there seems to be some issues like the missing RbConfig entries above and rb-sys accessing too many internals that need to be addressed first.

eregon · 2024-09-17T16:16:39Z

With #3664 fixed, I tried gem i -V commonmarker and got:

error: failed to run custom build command for `rb-sys v0.9.99`
...
  thread 'main' panicked at /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.69.4/lib.rs:622:31:
  Unable to find libclang: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"

OK I guess I need to install libclang on Linux: https://github.com/oxidize-rb/rb-sys?tab=readme-ov-file#what-dependencies-do-i-need-to-build-a-ruby-extension-in-rust
After sudo dnf install clang-libs:

  /usr/include/stdio.h:33:10: fatal error: 'stddef.h' file not found
  thread 'main' panicked at /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/build/main.rs:55:6:
  generate bindings: ClangDiagnostic("/usr/include/stdio.h:33:10: fatal error: 'stddef.h' file not found\n")

OK, so after sudo dnf install clang:

  error: 'rustfmt' is not installed for the toolchain 'stable-x86_64-unknown-linux-gnu'.
  To install, run `rustup component add rustfmt`
  Failed to run rustfmt: Internal rustfmt error (non-fatal, continuing)
  error: 'rustfmt' is not installed for the toolchain 'stable-x86_64-unknown-linux-gnu'.
  To install, run `rustup component add rustfmt`
  thread 'main' panicked at /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-build-0.9.99/src/rb_config.rs:209:32:
  Key not found: archincludedir
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

So we are still missing RbConfig::CONFIG['archincludedir'] it seems, we should define that like MRI does:

> RbConfig::MAKEFILE_CONFIG['archincludedir']
=> "$(includedir)/$(arch)"

^ @andrykonchin Could you make a PR adding that?

eregon · 2024-09-17T16:30:57Z

Doing a quick workaround for that, similar to what Randy did in the first post:

# config.rb
RbConfig::CONFIG["archincludedir"] = "#{RbConfig::CONFIG["includedir"]}/#{RbConfig::CONFIG["arch"]}"

I get:

error[E0432]: unresolved import `crate::rb_gc_adjust_memory_usage`
 --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/tracking_allocator.rs:3:13
  |
3 | use crate::{rb_gc_adjust_memory_usage, utils::is_ruby_vm_started};
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^ no `rb_gc_adjust_memory_usage` in the root

error[E0412]: cannot find type `RData` in this scope
 --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/target/release/build/rb-sys-b6e62330b9516279/out/bindings-0.9.99-x86_64-linux-3.2.4.rs:1:299289
  |
1 | ....2.4"] pub fn rb_tr_rdata (object : VALUE) -> * mut RData ; # [doc = "This is the primitive way to wrap an existing C struct into ::RD...
  |                                                        ^^^^^ not found in this scope

   Compiling emojis v0.6.3
error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:19:35
   |
19 |         let flags = rstring.basic.flags;
   |                                   ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:34:35
   |
34 |         let flags = rstring.basic.flags;
   |                                   ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:52:34
   |
52 |         let flags = rarray.basic.flags;
   |                                  ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:56:38
   |
56 |             let mut f = rarray.basic.flags;
   |                                      ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:70:34
   |
70 |         let flags = rarray.basic.flags;
   |                                  ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:94:35
   |
94 |         let ret: u32 = ((*rbasic).flags & crate::ruby_value_type::RUBY_T_MASK as VALUE) as _;
   |                                   ^^^^^ unknown field

Some errors have detailed explanations: E0412, E0432, E0609.
For more information about an error, try `rustc --explain E0412`.
error: could not compile `rb-sys` (lib) due to 8 previous errors

Re rb_gc_adjust_memory_usage, TruffleRuby does define that as static inline:

truffleruby/lib/cext/include/ruby/internal/intern/gc.h

Lines 388 to 395 in e834bb5

    
           #ifdef TRUFFLERUBY 
        
           static inline void rb_gc_adjust_memory_usage(ssize_t diff) { 
        
               // No-op for now 
        
               (void) diff; // To silence -Wunused-parameter 
        
           } 
        
           #else 
        
           void rb_gc_adjust_memory_usage(ssize_t diff); 
        
           #endif

I guess Rust doesn't find it because it doesn't end up in the libruby then, we could fix that by declaring it in src/main/c/cext/gc.c instead.

Regarding RData, no clue.

Regarding RBasic.flags that's something that TruffleRuby on purpose does not and cannot expose, so that's something that will need to be fixed in rb-sys so it doesn't assume struct RBasic to have those fields.

eregon · 2024-09-23T11:03:49Z

@andrykonchin added archincludedir and moved rb_gc_adjust_memory_usage in #3671.
So now the remaining issues are in the rb-sys crate.

eregon · 2024-09-23T11:10:56Z

Looking at the first flags error, it comes from https://github.com/oxidize-rb/rb-sys/blob/991df6de21d244631e1b889070a8e4136f4220fe/crates/rb-sys/src/stable_api/ruby_3_2.rs#L19
rb-sys is hardcoding CRuby internals, but it should use the RSTRING_LEN macro instead.

eregon · 2024-09-23T11:15:05Z

I found some more info in oxidize-rb/rb-sys#229.
So probably TruffleRuby should be treated like ruby-head then, because ruby_3_2.rs hardcoded macros are incompatible and CRuby-specific.

eregon · 2024-09-23T11:25:46Z

I submitted an issue to rb-sys: oxidize-rb/rb-sys#424

eregon · 2024-11-19T15:50:54Z

Would be good to try again now that oxidize-rb/rb-sys#424 got fixed

eregon added the cexts label Jan 22, 2024

rwstauner mentioned this issue Sep 13, 2024

Add version-related keys to RbConfig::CONFIG #3664

Merged

graalvmbot mentioned this issue Sep 18, 2024

[GR-18163] Fix some issues to support rb_sys gem #3671

Closed

eregon mentioned this issue Sep 23, 2024

Use stable_abi/compiled.{c,rs} on non-CRuby oxidize-rb/rb-sys#424

Closed

eregon mentioned this issue Nov 21, 2024

Add MAJOR, MINOR, TEENY to RbConfig #3725

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gem commonmarker 1.x fails to compile native extensions with rust #3396

Gem commonmarker 1.x fails to compile native extensions with rust #3396

rwstauner commented Jan 18, 2024 •

edited

Loading

jac241 commented Sep 2, 2024

eregon commented Sep 12, 2024

eregon commented Sep 17, 2024 •

edited

Loading

eregon commented Sep 17, 2024 •

edited

Loading

eregon commented Sep 23, 2024

eregon commented Sep 23, 2024 •

edited

Loading

eregon commented Sep 23, 2024

eregon commented Sep 23, 2024

eregon commented Nov 19, 2024

Gem commonmarker 1.x fails to compile native extensions with rust #3396

Gem commonmarker 1.x fails to compile native extensions with rust #3396

Comments

rwstauner commented Jan 18, 2024 • edited Loading

jac241 commented Sep 2, 2024

eregon commented Sep 12, 2024

eregon commented Sep 17, 2024 • edited Loading

eregon commented Sep 17, 2024 • edited Loading

eregon commented Sep 23, 2024

eregon commented Sep 23, 2024 • edited Loading

eregon commented Sep 23, 2024

eregon commented Sep 23, 2024

eregon commented Nov 19, 2024

rwstauner commented Jan 18, 2024 •

edited

Loading

eregon commented Sep 17, 2024 •

edited

Loading

eregon commented Sep 17, 2024 •

edited

Loading

eregon commented Sep 23, 2024 •

edited

Loading