Vendor protoc

[dunmatt]

prost-build has the "vendored" feature so that I need not have protoc installed. How hard would it be to add a similar feature to pbjson-build?

[tustvold]

I think you should be able to just enable the vendored feature flag for prost-build and things will work? Ultimately pbjson-build just requires a descriptor file, it is agnostic to how it is produced and doesn't directly interact with protoc at all.

[dunmatt]

Hmmm, my first attempt to do it that way didn't work, I'll do some more digging, it may well be user error.

[dunmatt]

The problem seems to be in pbjson-types:

error: failed to run custom build command for `pbjson-types v0.4.0`

Caused by:
  process didn't exit successfully: `/home/matt/code/comb3/seen_schema/seen-rs/target/debug/build/pbjson-types-53d2f1897182c455/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=/home/matt/.cargo/registry/src/github.com-1ecc6299db9ec823/pbjson-types-0.4.0/protos/google/protobuf/types.proto

  --- stderr
  thread 'main' panicked at '
  Could not find `protoc` installation and this build crate cannot proceed without
  this knowledge. If `protoc` is installed and this crate had trouble finding
  it, you can set the `PROTOC` environment variable with the specific path to your
  installed `protoc` binary.

  For more information: https://docs.rs/prost-build/#sourcing-protoc
  ', /home/matt/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.11.1/src/lib.rs:1227:10
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

[aierui]

I also met

    Checking mockito v0.31.0
   Compiling pbjson-types v0.4.0
error: failed to run custom build command for `pbjson-types v0.4.0`

Caused by:
  process didn't exit successfully: `/Users/develop/aierui/influxdb_iox/target/debug/build/pbjson-types-ee8401d0b4f56f87/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-changed=/Users/yixiao/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/pbjson-types-0.4.0/protos/google/protobuf/types.proto
  Running: "/Users/yixiao/.rpc-tools/protoc" "--include_imports" "--include_source_info" "-o" "/Users/develop/aierui/influxdb_iox/target/debug/build/pbjson-types-7a84daf192de4c6f/out/proto_descriptor.bin" "-I" "/Users/yixiao/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/pbjson-types-0.4.0/protos" "/Users/yixiao/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/pbjson-types-0.4.0/protos/google/protobuf/types.proto"

  --- stderr
  Error: Custom { kind: Other, error: "protoc failed: google/protobuf/any.proto: File not found.\ngoogle/protobuf/api.proto: File not found.\ngoogle/protobuf/descriptor.proto: File not found.\ngoogle/protobuf/duration.proto: File not found.\ngoogle/protobuf/empty.proto: File not found.\ngoogle/protobuf/field_mask.proto: File not found.\ngoogle/protobuf/source_context.proto: File not found.\ngoogle/protobuf/struct.proto: File not found.\ngoogle/protobuf/timestamp.proto: File not found.\ngoogle/protobuf/type.proto: File not found.\ngoogle/protobuf/wrappers.proto: File not found.\ngoogle/protobuf/types.proto:5:1: Import \"google/protobuf/any.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:6:1: Import \"google/protobuf/api.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:7:1: Import \"google/protobuf/descriptor.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:8:1: Import \"google/protobuf/duration.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:9:1: Import \"google/protobuf/empty.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:10:1: Import \"google/protobuf/field_mask.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:11:1: Import \"google/protobuf/source_context.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:12:1: Import \"google/protobuf/struct.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:13:1: Import \"google/protobuf/timestamp.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:14:1: Import \"google/protobuf/type.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:15:1: Import \"google/protobuf/wrappers.proto\" was not found or had errors.\n" }
warning: build failed, waiting for other jobs to finish...

➜ influxdb_iox git:(main) protoc --version
libprotoc 3.13.0

[tustvold]

Aah, yeah I can see that would be a problem. I would be very happy to review a PR adding this functionality, and get it released, otherwise I can probably find time in the latter part of next week to look into this. I'm afraid I am a little bit underwater at the moment 😅

[aierui]

For my problem, can set the following env to solve it.

export PROTOC=/usr/local/bin/protoc
export PROTOC_INCLUDE=/usr/local/include/google/protobuf

[rekby]

I have similar problem - after update prost-build and pbjson - I can't compile crate without protoc.

It is not problem for me personal, but I want transparent compile my crate on end customer computer without installed protoc.

env vars can't solve the case

[rekby]

I can research about solve the protoc (in background, not fast). But I have trouble with build the crates on my laptop:
https://gist.github.com/rekby/34f5e8b072f6eddc5fc18dfd0f124821

Is any prepare need to start work with the repo?

[tustvold]

Unfortunately this is really a prost issue, not a pbjson issue. I recommend checking out tokio-rs/prost#657 which has some background on the various changes that led to this predicament

As far as solutions go, if you wish for users to not need protoc, you will need to not use a build.rs and instead vend the generated code. This can easily be done by making build.rs into a binary, and then running this. Alternatively you could check out https://github.com/neoeinstein/protoc-gen-prost

Is any prepare need to start work with the repo?

This does not appear to be related to pbjson or even prost, it looks like something is messed up with your build toolchain. Perhaps try re-installing Rust, or perhaps raise an issue on anyhow, as that appears to be where the failure is "error: could not compile anyhow due to 56 previous errors".

[notmgsk]

Sort of related: would you be open to not requiring protoc in the pbjson_types crate? Seems like it would be simple enough just to version the generated code in the crate rather than have to build it?