Raw Calls - What is necessary?

[skilesare]

I've recently been frustrated by Motoko's inability to do raw calls as seen in the cycle wallet(https://github.com/dfinity/cycles-wallet/blob/5725370020593bd744ddbc589c117d76d6ca8ae4/wallet/src/lib.rs#L625).

To be able to do this I would imagine motoko would need to:

• expose the raw call
• enable some kind of type reflection
• formalize some kind of serialization/deserialization

I've seen some mention of these in some of the open/closed meta-issue issues here and I thought I'd stand on top of the building yelling to hopefully get these some attention. :)

What I don't want to do is to have to create a function on every motoko pass through canister called __forwardCall(principal: Principal; method: text, args: [Nat8]/WorkSpace/[DataZones]) and have to have every canister handle a __handleForwardCall that deserializs all the parameters manually into known types and organize the community to universally add these to their canisters. perhaps there is a compiler way to automate this kind of thing? Seems overkill when the underlying functions can just be exposed some how.

[crusso]

If all you want is the ability to dynamically invoke a method by name with an already serialized blob and get the raw (undeserialized) blob back asynchronously, then I think we can actually provide that fairly easily.

call_raw : (Principal, Text, Blob) -> async Blob

which should be enough to implement the call function you mention in the Rust code.

I'm not sure what the cycles argument is used for in Rust, but I imagine it is specifying the number of cycles to deduct from the wallet and could probably be handled separately by adding cycles to the call.

I think the issues you mentioned might have been trying to solve a much more general and harder problem: how to support candid functions and shared functions with polymorphic types which is indeed much trickier.

I guess it depends on what you are trying to achieve in your applications. If it's just implementing this particular wallet function, I think that's achievable. But more ambitious stuff might be tricky indeed.

@rossberg?

[skilesare]

Yes,

call_raw : (Principal, Text, Blob) -> async Blob

gets me 80% of the way to where I want to be and should 100% enable building a motoko wallet canister(or similar pattern) actor.

There is a huge need for some kind of general-purpose candid interpreter/serializer/deserializer, but I think that is a much bigger issue to tackle.

I've been thinking about an axiom along the lines of "any canister that could be built with rust should be able to be built with motoko" as a kind of guiding principle that I'd love to get the team's feedback on. I would imagine there exists a finite set of features that keep this from being true at the moment and it would be interesting to catalog them. "call_raw" is on that list. Most of the list likely revolves around low-level system stuff that should be ignored/abstracted by most devs, but should be accessible as the need arises. I'm doing work on certificates and I'm thinking that some of that stuff may be on the list as well. Looks like they are cbor encoded? Another topic perhaps, but I keep running into these things and I'd like to be proactive in cataloging them so the community and team can coordinate on priorities.

I've really been enjoying the language over the last couple of weeks and feel like I can get a lot done with it now. Thanks for engaging!

[skilesare]

I guess I'll also add query_raw to the list as a request as well. I can see the need for a general-purpose caching canister that can be configured to query a bunch of other canisters to have data 'at the ready' in a way that the caching canister shouldn't have to know about the structure of the data. The canister that uses the caching canister can take the responsibility of deserializing into known types.

[BrantBrown]

It is necessary to implement in motoko.

call(Principal,Text,args)
call_raw(Principal, Text,Blob) 

[zire]

@chenyan-dfinity Hey Yan, could you take a look at this issue? Some developers are watching this one closely and waiting for a solution. Thanks!

[chenyan-dfinity]

I think what's needed here is a way of transferring generic data over the wire. See related discussion dfinity/candid#245.

Meanwhile, as a stop gap measure, it's always possible to use both Rust and Motoko in your project, if certain feature is not available in one of the languages.

[skilesare]

I think what's needed here is a way of transferring generic data over the wire. See related discussion dfinity/candid#245.

This would be wonderful, but what we need immediately is just a call_raw(principal, method, blob): Blob that gives the user the ability to handle whatever comes their way or to just pass the call through to another canister. This maximises optionality.

Meanwhile, as a stop gap measure, it's always possible to use both Rust and Motoko in your project, if certain feature is not available in one of the languages.

Is it? I've never seen an example that imports a rust library into motoko! I understood that is was a coming soon feature. We could have a motoko canister talked to a rust canister, but that adds a x-canister call which is slow and can't do a query.

[chenyan-dfinity]

We could have a motoko canister talked to a rust canister, but that adds a x-canister call which is slow and can't do a query.

Yes, they can communicate at the canister level, and it will be an inter-canister call. Not ideal.

Motoko playground is using Rust canister to parse and transform Wasm module: https://github.com/dfinity/motoko-playground/blob/main/service/pool/Main.mo#L109, and it doesn't add much latency in practice.

[BrantBrown]

@chenyan-dfinity
Motoko x-canister calls are only called in a predefined way , we need a general way to get more flexibility
For example, DFT : transferAndCall,we need to support calling any method of any canister with any parameters.We support it through ic-cdk's api::call, motoko does not have an api like api::call.

It is necessary to implement in motoko.

call(Principal,Text,args)
call_raw(Principal, Text,Blob) 

[skilesare]

I'm popping this back up another two months later. @matthewhammer mentioned today that you all @rossberg, @crusso, etc discussed it in a meeting and the it wasn't a priority at the moment. I was kind of bummed so I went back to the drawing board. I spent a few minutes so messing around with the following:

** Edited **

import Debug "mo:base/Debug";

actor Echo1 {

  var message : Text = "";

  // Say the given phase.
  public func say(phrase : Text) : async Text {
    Debug.print(debug_show(phrase));
    message := phrase;
    return phrase;
  };

  public func get() : async Text{
    return message;
  }
};

import Debug "mo:base/Debug";

actor Echo2 {

  var message : Text = "";

  public func say(phrase : Text) : async Text {
    Debug.print(debug_show(phrase));
    message := phrase;
    return phrase;
  };

  public func get() : async Text{
    return message;
  }
};

I thought maybe I'd been a idiot for a few months and it might be as simple as telling the candid type definition in Echo2 that a function took a blob and then the receiving function in Echo 1 would say "hey...this is binary data, I bet it is in my candid syntax...I'll parse it as such"...but no luck. It seems that the something is putting the IDL into the message and then sending it across and then when it gets to Echo 1 if fails with The Replica returned an error: code 4, message: "IC0503: Canister vo5te-2aaaa-aaaaa-aaazq-cai trapped explicitly: IDL error: unexpected IDL type when parsing Text". Sigh.

After that spent about 2 hours in the motoko repo trying to ferret out if there was some lines here or there I could change that would get the call_raw exposed as a Prim functor or something. It is like right there...I can see ICCallPrim places and I can see the Prim.mo file, but OCaml, Haskell, and what ever else is going on here is so foreign to me that I might as well be trying to bang out Shakespeare with a room full of monkeys.

But I need this functionality so I'm going to keep banging my head against a wall. I could go into a fairly long diatribe about why we need it, but I'm starting to think that maybe I'm just missing something. In my mind this single feature is holding Motoko back from actually being a lingua franca for the IC.

What am I missing?

I really don't want to shift all my attention to another sdk, but we need something simple and straight forward that ALL programs can be written in. I've got a lot invested in Motoko right now, but the reality is I can't a build a wallet for the IC right now. And I really...really... need to build a wallet and some other passthrough proxy like services. I need to support my users interacting with canisters and standards that don't exist yet through an identity canister...or wallet...or proxy...call it what you will. I want to hold assets, data, and permissions and potential send those to services that I don't know about at the time the code is installed. This does not seem like an optional thing it is essential and already caused me to go chasing less interesting problems because the really valuable ones can't be solved with motoko right now.

The other option here is to convince the community that every public function needs a my function_variant(params : VariantWrapper) : async VariantWrapper{} that wraps all types in a corresponding variant. That just seems to undo most of what motoko was trying to do. I still want to use proper motoko for everything...there are just a few instances where I really need break through to serve the needs of my users.

So as an exercise...because maybe I have some resources to do so....if I wanted to expose this functionality to motoko...where would I make changes? If I was going to fork the code and make it so that an array of bytes could be sent to a motoko canister without strict IDL definitions and the other canister would assume it was in the right format and take in the message...I'm fin if it throws if the format isn't right....how would I do that? What language would I need to know and what files would I need to look in? Is it even possible to fork and implement, or is there something in the way the IC works that prevents it...Rust can do it so I don't think it is at the IC level.

[chenyan-dfinity]

I'm lost. The code for Echo1 and Echo2 looks exactly the same.

[rossberg]

Same here, some c&p error with your example?

[skilesare]

Sorry...yes...a copy paste error...up too late.

import Debug "mo:base/Debug";

actor Echo1 {

  var message : Text = "";

  // Say the given phase.
  public func say(phrase : Text) : async Text {
    Debug.print(debug_show(phrase));
    message := phrase;
    return phrase;
  };

  public func get() : async Text{
    return message;
  }
};

import Debug "mo:base/Debug";
import Blob "mo:base/Blob";
actor Echo2 {


  type first = actor {
    say : (Blob) -> async Blob;
    get : () -> async Text;
  };

  public func testSay() : async (){
    let thisFirst : first = actor("vo5te-2aaaa-aaaaa-aaazq-cai");
    Debug.print(debug_show(await thisFirst.say(Blob.fromArray([1:Nat8,2:Nat8, 3:Nat8, 4:Nat8]))));
    Debug.print(debug_show(await thisFirst.get()));
  };



};

[crusso]

(summarising from another slack conversation)

This is what we want to implement in Motoko, right?

Candid:

https://github.com/dfinity/cycles-wallet/blob/9ef38bb7cd0fe17cda749bf8e9bbec5723da0e95/wallet/src/lib.did#L150

Rust Code:

https://github.com/dfinity/cycles-wallet/blob/9ef38bb7cd0fe17cda749bf8e9bbec5723da0e95/wallet/src/lib.rs#L666

As I said above, I think it just requires a dynamic call instruction that take a principal, method name and blob and returns the raw binary response as an async blob. Needs compiler support but no candid extension AFAICT.

If you want Motoko to do more than just pass on a given blob and return a given blob, then we would indeed need do quite a lot more work. But for the scenario of the particular method above, I see no real difficulty and no need for a Candid extension and additional serialization prims.

(The reason this would need some compiler support rather than just a new primitive is that the return type is asynchronous, so I would probably need to extend the async/await pipeline to translate the function appropriately.)

[skilesare]

Yes...that is what we need. Just like the cycle wallet.

That would be enough for now.

I was thinking last night that maybe if there were a type called "raw" that would ignore the IDL on ingress and attempt to apply the expected data type it would be interesting as well...but if there is an easy way to tap into the underlying system call like in the cycle wallet it would help us immediately.

[rossberg]

Before we short-circuit to a completely untyped low-level reflection hack (and accept having it leak into Candid APIs), let's see if we can provide the necessary functionality in a suitably safe and high-level manner.

As far as I can see, the problem to solve here is that you want to be able to have some actor make a call on another actor's behalf (*). Now, inside a programming language, you could solve this easily by creating a thunk, i.e., passing a small closure that encapsulates the call.

We could support something similar in a typeful manner if we extended Candid's notion of function type to include not just bare receivers, but closures. Those would be limited to (perhaps partially) applied calls. The wire representation would be a pair of the bare target function and its (partial) list of arguments annotated with their Candid types.

On the Motoko side, we could receive such closures as if they were regular shared functions and call them as if they were ordinary functions with the remaining (possibly zero) arguments. That would execute the call, with the current actor being the caller. Inversely, we also need to provide a way to construct such closures. For that, we could either introduce a library function for partial application or dedicated syntax.

Similarly, Candid bindings for other languages would define a means to produce and consume such closures.

For example, with that, the forwarding function from the wallet would simply have this type:

wallet_call: (() -> ()) -> (WalletResultCall)

The Motoko implementation of this method would be

func wallet_call(f : shared () -> async ()) : async WalletResultCall {
  try #ok (await f()) catch e { #err e.text /* or something */ }
}

To call an actor method a.f : (Nat, Nat, Nat) -> () through the wallet in Motoko, you'd be doing something like this:

wallet.wallet_call(IC.bind(a.f, (1, 2, 3));

assuming IC.bind<A, R> : (fun : shared A -> R, arg : A) -> (shared () -> R) is provided by the system library.

AFAICS, that provides the mechanism needed in a typeful and convenient manner and avoids any messing with exposing low-level argument representations, constructing calls via reflection, and hoping that this is used correctly. It would also allow encapsulating potential future additions to the call mechanics, e.g., cycle transfer (which I think the call_raw type as proposed above would not handle without a further extension).

WDYT?

(*) \Rant: I suspect that the need to do this at all is a consequence of the IC's unfortunate choice of exposing a caller identity and recommending forms of authentication through it. The use case at hand essentially is a form of delegating authentication. If we had opted for a modern capability-based authentication model then raw call forwarding wouldn't be necessary because a message caller would not be observable and the ad-hoc dynamic scoping that goes with it wouldn't exist in the first place.

[rossberg]

PS: The call result is () in my example above, while the original API returns a Blob. To handle a generic result properly, we'd want Candid-level generics, or at least the dyn type we talked about.

[crusso]

That seems like a considerable amount of extra design and implementation work to solve a more general problem than the one at hand. I'm fine with this in principle, but in practice, I fear our users will have migrated to Rust before we deliver.

Another solution would be to provide raw_rand in the interim, deprecating its use in future once a better solution is available.

(I don't actually see any issue with cycle transfer using call_raw.)

[matthewhammer]

I'm fine with this in principle, but in practice, I fear our users will have migrated to Rust before we deliver.

I see your fear, but this solution would benefit canisters of all languages, not just Motoko. It would (hopefully) obviate the need for using call_raw there too, assuming a suitable variant of the IC.bind primitive can be coded up there. It's a higher-order language, so I would hope so.

[matthewhammer]

@rossberg wrote

... WDYT?

👍 -- My take on this idea is that Candid (and the IC) should adopt the pains and benefits of higher order programming (passing closures that stay abstract!) at the Candid level, rather than drop down into some kind of first-order language when we do IC-level calls. I agree, for all of the reasons that higher order programming is eventually what we all want, once we want real abstraction in our APIs. This is a great example.

[crusso]

FTR the raw version has the (ok, dubious) advantage of working with non-candid binary formats, e.g protobuf. Not that we should encourage more of that thing...

[skilesare]

This looks like it will work. In this case the client would need to know and provide the candid format right?

The protobuff comment is interesting because a big blocker for a lot of projects right now is that they can't access the protobuff functions on the ledger and the code advises not to use send_dfx. A motoko protobuff library might help this if the type of those functions is binary, but that is probably another issue.

[skilesare]

Organizationally, I don't know what the current pace of motoko development inside the organization is, but this arrived in my inbox this morning: https://github.com/lastmjs/azle/blob/update-calls/examples/basic/canisters/basic/app.ts

I'd say the window for motoko to be a dominant language for development on the IC is closing quickly. A migration to rust was mentioned, but if this JS stuff gets mature AND as more raw rust like access it will be tough to tell people to start with motoko, even if the JS engine is 10x more expensive.

I've been contemplating some ICDevs bounties to try to help speed some of these features along, but I'd need to better understand how the foundation would receive that and where to find oCamel developers.

[rossberg]

Yes, this would be an extension to Candid that both sides would have to use accordingly.

I would argue that the extension is relatively small, targeted, and well-behaved. It's more minor for Motoko in the sense that its implications for the language as a whole are much less drastic than exposing a primitive that throws intra-language type safety out the Window and pierces all abstractions. Such a move should be the last resort only.

My even bigger concern about the the shown use of call_raw is that these questionable properties even leak into public canister interfaces. Passing on arbitrary blobs authenticated as "yours" with no way of validating the well-formedness of their contents strikes me as a recipe for attacks. Furthermore, the blobs that the forwarding method exposes aren't even individual Candid-encoded values, but function argument lists – those ought to be an implementation detail of Candid calls.

As for non-Candid args, that would be a further step in the wrong direction IMHO. I think we all agree that the ledger needs fixing, not the rest of the ecosystem.

[skilesare]

Passing on arbitrary blobs authenticated as "yours" with no way of validating the well-formedness of their contents strikes me as a recipe for attacks

Great point!

[matthewhammer]

this arrived in my inbox this morning: https://github.com/lastmjs/azle/blob/update-calls/examples/basic/canisters/basic/app.ts

That URL 404's for me, but I can visit the project page and see that it's pretty immature.

Even if it were very, very mature, you'd still have the issue that TypeScript's type system isn't one. Everything built with it suffers from more than the performance of JS, but also, the lack of real guarantees about whether a canister's main business logic is free of any type errors. At some point, there is no substitute for the precision and economy of a real type system.

I'd say the window for motoko to be a dominant language for development on the IC is closing quickly.

I hear what you are saying. I also think it's strategically important for Motoko to become more popular among IC developers, and more broadly too, perhaps. We agree about that.

But if it becomes popular by accepting the poor design choices of other languages, and undermining the very principles that set it apart (a real type system, very much unlike that of TypeScript), then there is no reason to even build this language at all, much less use it.

I always think about complexity budgets when I think about languages. Not having a real type system that we can trust means that what we have instead is additional complexity (a test suite, and CI, etc.) to stand in its absence. If "unnecessary complexity" is the enemy that we are trying to conquer together (long term) with the Internet Computer and Motoko efforts that we do collectively, then it seems like we should stay true to these principles, even if it means paying more investment in R&D time to get to where we want to be.

[rossberg]

PR for extending the Candid spec along the lines of my suggestion: dfinity/candid#291

[crusso]

Update: given the lack of progress on a full solution that addresses the general call forwarding scenario, with n-ary arguments and n-ary returns, for arbitrary n, I will try to implement the interim workaround of

call_raw : (Principal, Text, Blob) -> async Blob

(restricted to async contexts).

@skilesare please let me know if that is still a useful stepping stone for you.

I'm not convinced this opens up any more of a pandora's box than being able to bounce of a Rust canister that does the call for you.

(hopefully next week).

[skilesare]

This would be a huge help and at least a step in the right direction.

[crusso]

@skilesare unfortunately, the chances of me getting round to this this week are now approximately nil. Sorry about that.

[skilesare]

There is always next week! Thanks for the update.

…

On Wed, Jan 12, 2022 at 11:06 AM Claudio Russo ***@***.***> wrote: @skilsare unfortunately, the chances of me getting round to this this week are now approximately nil. Sorry about that. — Reply to this email directly, view it on GitHub <#2703 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE5CO6V4BD6HAWZPKV6QF3UVWYH5ANCNFSM5BNENAIQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Austin Fatheree twitter: @afat http://twitter.com/afat

[skilesare]

Any updates this week?

[crusso]

Not yet. I've been dealing with a GC bug and other things but can actually take a look tomorrow.

[crusso]

@skilesare

Finally started work on the poor man's solution:

#3086

Examples are here:

https://github.com/dfinity/motoko/blob/f179358a311681d0a10ed07757d15b49158a7cfa/test/run-drun/call-raw.mo

It's basically working, apart from some CI issues (and some refactoring I'd like to do to the implementation).