Candid: Spec should not allow for inline types

[hansl]

The current languages that support Candid generation (Motoko and JavaScript) allow for inline types (defining inlined structures and arrays), but new languages that we want to add support to use Candid (C++, Rust, Java) do not. We also introduce hand written Candid files right now, so users might be tempted to use inlined Candid if we don't explicitly tell them not to.

Multiple solutions;

1. we create temporary names for those structures that users are forced to use (e.g. struct _FnnameArg0 for the first argument of the Fnname function). This is very inconvenient for users and not stable, so users would have to guess which names work.
2. we change the spec to disallow inline non-atomic types. Meaning that all hand

Inline types also cannot be named in certain contexts (as arguments to functions and values in arrays cannot be named) so it hides the semantics of these types AND it hides type reusability. We don't know if a function arguments matches a function returns semantically (we know syntactically but it's irrelevant for a developer).

To greatly minimize the pain long term as we add more languages, I suggest doing the second option. This does not break our current tooling as Candid generation never use inline types right now. It also is backward compatible as no JS/Motoko generation use inline types (as we don't generate it). So there is minimal impact (ie. no impact) for current code.

There is a concern for third party libraries, but IMO it's very limited and can be worked on case by case basis. The 2 examples I know of are Kotlin and Dart, both of which are done by 55 Foundry and can be worked with easily (and they're also private so no impact on developers outside of those).

TL;DR: Inline types hide semantics, break the easy-to-use nature of Candid and make the experience harder for developers of language like Rust and C++. They could make things even harder in the longer term. They bring little to no value. I suggest simply disallowing them at the language level.

[nomeata]

This does not break our current tooling as Candid generation never use inline types right now.

Maybe I don’t understand what you mean with inline types; if it is what I think it is then that statement is not true:

/tmp $ cat test.mo 
actor {
    public func foo({a : Int}) : async {a:[Nat]} {{a=[]}}
}
/tmp $ ~/dfinity/motoko/src/moc --idl test.mo 
/tmp $ cat test.did 
service : {
  "foo": (record {"a": int;}) -> (record {"a": vec nat;});
}

I think disallowing that goes very much against the spirit of Candid as embodying structural typing. Names for types should not matter! Every design decision in Candid will be more suitable for some host languages and less for others. (Eg. that tuples aren’t really tuples; the set of number types; the lack of a dictionary type.)

We don't know if a function arguments matches a function returns semantically (we know syntactically but it's irrelevant for a developer).

A design decision behind Candid was to not embed semantic information, only representation. That’s why there are no range types, no dictionary or map types.

Or am I barking up the wrong tree and maybe you mean something else (since anonymous types are produced by the Candid export)?

[hansl]

if it is what I think it is then that statement is not true

Thanks, I learnt something new today. Yan didn't point out this example so I guess it's easy to overlook.

---

that tuples aren’t really tuples; the set of number types; the lack of a dictionary type

These are implementation details, not actual burden to the developer. A record in Rust and C is clearly a struct, which cannot be anonymous. This brings me to the other point;

Every design decision in Candid will be more suitable for some host languages and less for others.

Unfortunately, this is not "more suitable", it makes it a guessing game where you end up fighting the library instead of using it. If we have to generate names, it adds more knowledge surface that the developer needs to understand and know. And any change to the name generation scheme will now be a breaking change.

This doesn't make it "a little annoyance" in terms of performance or memory mapping, this makes it a problem for every developers that want to interact with the IC when using inline types.

[chenyan-dfinity]

Several comments:

• Argument types can be named, e.g. f : (ids: vec principal) -> (result: variant { ok: nat; error: text }), and we can use these names for inlined types if they are provided.
• Option 2 doesn't necessarily give you a better user experience. The names the compiler generates are not pretty either. For example, here is a signature we get from LinkedUp:

"getConnections": (UserId_2) -> (vec Profile);

UserId_2 is not much better than struct _getConnectionsArg0 in my opinion. The number suffix is unavoidable for polymorphic Motoko types.

• Back to the Rust case, there are some crates that allows migration of struct type of the same shape, e.g. https://crates.io/crates/frunk#generic, which can be helpful.

[hansl]

I'm still not sure why we generate type UserId_2 = UserId, to be entirely honest with you. That seems like something we should fix, regardless of whether we implement any option in this proposal.

[nomeata]

I expect that most languages will support importing Candid types (maybe dynamically, maybe statically; maybe with an external tool, or a macro, or type magic, or meta programming), and exporting to Candid types. It’s true for Motoko, JS, Haskell. Not sure about Rust, but probably eventually, too.

I expect that in almost all cases, automatically importing the Candid type will yield something ugly in many cases:

• In Motoko, importing a vec nat8 might give you a [Nat8] when maybe you wanted a Blob.
• In Haskell, importing a record will give you a generic row-type, when maybe you want a plain (nominal) Haskell record.
• In C++ you might get odd manual names.

Therefore I expect that developers will often go the other way: Look at the Candid file, write nice types in their host langauge, export to Candid and then check (using Candid tool and the Candid subtype relationship) that they have described something compatible.

This doesnm’t fulfil all the high hopes people might have for Candid. But at least there is a way out.

[chenyan-dfinity]

I'm still not sure why we generate type UserId_2 = UserId

Yeah, this is not a great example, and can be fixed. But you cannot completely eliminate the suffix for polymorphic types.

a.mo
type Result<Ok,Err> = {
  #ok:Ok;
  #err:Err;
};
actor {
 public func f(x: Result<Nat, Text>):async Result<Int,Text> {
   #ok 42
 };
}

a.did
type Result_2 =
 variant {
   "err": text;
   "ok": int;
 };
type Result =
 variant {
   "err": text;
   "ok": nat;
 };
service : {
  "f": (Result) -> (Result_2);
}

As @nomeata mentioned, the best way to do this is to define a polymorphic Result type in Rust, and use the result type to call the actor.

[hansl]

polymorphic Result type in Rust

But you cannot generate that from just the Candid file, right? Unless you are planning to add polymorphism to candid types.

[chenyan-dfinity]

Yes, that's why we have to monomorphize the polymorphic types in Candid. Transferring polymorphic types over the network is pretty challenging.

[hansl]

You already have a type table ;)

[matthewhammer]

Transferring polymorphic types over the network is pretty challenging.

Off Topic: I think we eventually want to support libraries that mix generic programming via polymorphic type arguments and dynamic canister creation.

BigMap is a canonical example, where I'd eventually like to offer something that more closely resembles the HashMap API (with polymorphic type arguments); today, we assume everything is a blob, partially because of this inability to instantiate type arguments within the canister creation process (which would affect the Candid API, and make it parametric, not monomorphic, generally).

Made an issue to track this, in the Candid project: dfinity/candid#53

[rossberg]

If you forbid inline types, then you are just introducing the dual problem for any type that actually is structural in a language you generate the IDL from. For example, if in Motoko I'd write

func f() : {x : Int; y : Int}

then the Motoko compiler would now have to invent the same ugly names the other way.

But in any case, the real issue here isn't even syntax, but whether types are structural or nominal, i.e., whether you need to canonicalise equivalent types. For example, if you naively map Candid records to C or Rust structs, then even with outlined types you won't get what you want:

type T1 = record {x : nat};
type T2 = record {x : nat};
service : {
  f : () -> T1;
  g : T2 -> ();
};

This allows g(f()), but any naive mapping to C or Rust will fail to allow that. This obviously is a stupid example, but situations like this can arise in practice when you import and compose Candid definitions from multiple sources.

So what you are suggesting does not actually avoid the extra work a Rust binding will have to do to canonicalise types. And once it does, handling inline types probably is not a big problem.

It's just a matter of fact that some forms of types are structural in some language and nominal in others, and most languages are not terribly principled about these choices. So there is no way Candid can directly fit all of them. We went with what's most flexible in terms of evolution and forward/backwards compatibility. And that's the only thing that has a coherent semantics in a distributed setting.

FWIW, when you look at other solutions like protobufs, they don't even try to map IDL defs to language type definitions. It's a luxury if you can, but not necessarily expected to be possible. Types that don't match your language may require mapping to a library abstraction, e.g., some CandidRecord type in the binding.

[chenyan-dfinity]

Equivalence checking is expensive, and even if we do that, we still don't know what name to give for each unique type.

I see two options here:

1. import did file generates only the encoder/decoder function for each actor method. No type defs are introduced. This is the JS library approach.
2. Let the developer to write the IDL defs manually in the host language and use a tool to check whether the exported candid type is compatible with the remote did file.

[hansl]

Motoko compiler would now have to invent the same ugly names the other way

The advantage of this approach is that the candid file itself becomes self documented. It's not dependent on the code generator's implementation, since the DID file contains all the information needed to use it.

---

some CandidRecord type in the binding

Maybe we don't want to codegen native structs in Rust then (since that's the language that triggered this), and have each language use their own set of syntax to build those in ways that approximate native user experience (like a builder pattern in Rust, a set of define macros in C, etc).

A call to (e.g.) install_code would then look like this;

canister("ic:00")
  .call("install_code")
  .arg(
    candid::Record::builder()
      .add("mode", candid::Variant::new("install", candid::Null))
      .add("wasm_module", candid::Vec::<u8>::from(vec![...]))
      // ...
      .build()
    )
  .execute()
  .await

(this is all back of the envelope code of course).

This is a con for usability, but it does make the API surface clear and simpler. We could iterate on it with generators later on.

@chenyan-dfinity WDYT?

[chenyan-dfinity]

The advantage of this approach is that the candid file itself becomes self documented.

The candid file describes the actor interface, it is self documented no matter what.

A call to (e.g.) install_code would then look like this

I would say this is more ergonomic (option 2 in my comment above):

#[derive(CandidType)]
enum Mode = { Install; Upgrade; Reinstall; };
#[derive(CandidType)]
struct InstallConfig {
  mode: Mode,
  wasm_module: Blob,
};
install_code(InstallConfig { mode: Mode::Install, wasm_module:... }).await;

[hansl]

But the code generator will not be able to determine the name InstallConfig, so people using this candid file would have to resolve to install_code(InstallCodeArg0 { mode InstallCodeArg0Enum0::Install, wasm_module: ... )

They have to guess InstallCodeArg0Enum0. There is no way around that.

[chenyan-dfinity]

I mean we let the developer write all the type defs by hand. All the code above is hand-written, no codegen involved. At the build time, we can use a tool to check the the exported candid type from the above code is compatible with the did file we have. This is also how serde works: user defines whatever struct they want, as long as the types are compatible.

[hansl]

How do they write type defs for the functions themselves?

[chenyan-dfinity]

How do they write type defs for the functions themselves?

Right, function needs a builder to take args, just like the candid crate. Having a builder for record seems too much.

[nomeata]

I don’t think this is actionable. If so, please reopen agains the Candid repo.