Error: "Aggregator failed to respond to task" when running with local DevNet

[wawrzek]

I use a local devnet (https://github.com/ivy-net/iv1) to run EigenLayer and this AVS locally. Smart contracts are deployed properly, both aggregator and operator starts after only adjusting the Makefile (or passing 2 variables). However, aggregator fails with the following error message (more context from logs below).

 Error submitting SubmitTaskResponse tx while calling respondToTask {"err":"execution reverted: BLSSignatureChecker.checkSignatures: invalid reference block"}

I tried to track where the reference block set was, and how it became invalid. I cannot find anything obvious in the code, especially the anvil deployment works fine. On Discord, it was suggested that there is a problem with the AVS code, and it would be fixed.

I see that the anvil deployment is pushed by 100 block forward. Is that related?

[2024-05-14 13:41:28.763 BST] INFO (aggregator/aggregator.go:199) Aggregator sending new task {"numberToSquare":"9"}
[2024-05-14 13:41:30.774 BST] INFO (txmgr/txmgr.go:143) Transaction not yet mined {"txID":"0x86a0ecfab3410f48821ad6e8360cff4c02691865d855d706d48b755e7c459d9f"}
[2024-05-14 13:41:31.603 BST] INFO (aggregator/rpc_server.go:51) Received signed task response: &aggregator.SignedTaskResponse{TaskResponse:contractIncredibleSquaringTaskManager.IIncredibleSquaringTaskManagerTaskResponse{ReferenceTaskIndex:0x9, NumberSquared:81}, BlsSignature:bls.Signature{G1Point:(*bls.G1Point)(0x140000d41d8)}, OperatorId:types.Bytes32{0xc4, 0xc2, 0x10, 0x30, 0xe, 0x28, 0xab, 0x4b, 0xa7, 0xb, 0x7f, 0xbb, 0xe, 0xfa, 0x55, 0x7d, 0x2a, 0x2a, 0x5f, 0x1f, 0xbf, 0xa6, 0xf8, 0x56, 0xe4, 0xcf, 0x3e, 0x9d, 0x76, 0x6a, 0x21, 0xdc}}
[2024-05-14 13:41:33.605 BST] INFO (aggregator/rpc_server.go:51) Received signed task response: &aggregator.SignedTaskResponse{TaskResponse:contractIncredibleSquaringTaskManager.IIncredibleSquaringTaskManagerTaskResponse{ReferenceTaskIndex:0x9, NumberSquared:81}, BlsSignature:bls.Signature{G1Point:(*bls.G1Point)(0x140000fa068)}, OperatorId:types.Bytes32{0xc4, 0xc2, 0x10, 0x30, 0xe, 0x28, 0xab, 0x4b, 0xa7, 0xb, 0x7f, 0xbb, 0xe, 0xfa, 0x55, 0x7d, 0x2a, 0x2a, 0x5f, 0x1f, 0xbf, 0xa6, 0xf8, 0x56, 0xe4, 0xcf, 0x3e, 0x9d, 0x76, 0x6a, 0x21, 0xdc}}
[2024-05-14 13:41:33.610 BST] INFO (aggregator/aggregator.go:142) Received response from blsAggregationService {"blsAggServiceResp":{"Err":null,"NonSignerQuorumBitmapIndices":[],"NonSignerStakeIndices":[[]],"NonSignersPubkeysG1":[],"QuorumApkIndices":[1],"QuorumApksG1":[{"X":"643552363890320897587044283125191574906281609959531590546948318138132520777","Y":"7028377728703212953187883551402495866059211864756496641401904395458852281995"}],"SignersAggSigG1":{"g1_point":{"X":"8443873543902734941423897488132867553403723007728710173648445437781070293444","Y":"21607580942168206894016089909611195006504832862680716687827944912459106791693"}},"SignersApkG2":{"X":{"A0":"15669747281918965782125375489377843702338327900115142954223823046525120542933","A1":"10049360286681290772545787829932277430329130488480401390150843123809685996135"},"Y":{"A0":"14982008408420160629923179444218881558075572058100484023255790835506797851583","A1":"4979648979879607838890666154119282514313691814432950078096789133613246212107"}},"TaskIndex":9,"TaskResponseDigest":[158,82,41,43,207,191,248,182,122,229,107,175,3,37,244,192,174,97,210,14,175,116,199,54,192,157,239,85,1,22,173,103],"TotalStakeIndices":[1]}}
[2024-05-14 13:41:33.610 BST] INFO (aggregator/aggregator.go:181) Threshold reached. Sending aggregated response onchain. {"taskIndex":9}
[2024-05-14 13:41:33.614 BST] ERROR (chainio/avs_writer.go:115) Error submitting SubmitTaskResponse tx while calling respondToTask {"err":"execution reverted: BLSSignatureChecker.checkSignatures: invalid reference block"}
Stacktrace
    github.com/Layr-Labs/incredible-squaring-avs/core/chainio.(*AvsWriter).SendAggregatedResponse
    	/Users/wawrzek/Ethereum/EigenLayer/incredible-squaring-avs/core/chainio/avs_writer.go:115
    github.com/Layr-Labs/incredible-squaring-avs/aggregator.(*Aggregator).sendAggregatedResponseToContract
    	/Users/wawrzek/Ethereum/EigenLayer/incredible-squaring-avs/aggregator/aggregator.go:190
    github.com/Layr-Labs/incredible-squaring-avs/aggregator.(*Aggregator).Start
    	/Users/wawrzek/Ethereum/EigenLayer/incredible-squaring-avs/aggregator/aggregator.go:143
    main.aggregatorMain
    	/Users/wawrzek/Ethereum/EigenLayer/incredible-squaring-avs/aggregator/cmd/main.go:57
    github.com/urfave/cli.HandleAction
    	/Users/wawrzek/go/pkg/mod/github.com/urfave/[email protected]/app.go:524
    github.com/urfave/cli.(*App).Run
    	/Users/wawrzek/go/pkg/mod/github.com/urfave/[email protected]/app.go:286
    main.main
    	/Users/wawrzek/Ethereum/EigenLayer/incredible-squaring-avs/aggregator/cmd/main.go:33
    runtime.main
    	/opt/homebrew/Cellar/go/1.22.2/libexec/src/runtime/proc.go:271
[2024-05-14 13:41:33.614 BST] ERROR (aggregator/aggregator.go:192) Aggregator failed to respond to task {"err":"execution reverted: BLSSignatureChecker.checkSignatures: invalid reference block"}
Stacktrace
    github.com/Layr-Labs/incredible-squaring-avs/aggregator.(*Aggregator).sendAggregatedResponseToContract
    	/Users/wawrzek/Ethereum/EigenLayer/incredible-squaring-avs/aggregator/aggregator.go:192
    github.com/Layr-Labs/incredible-squaring-avs/aggregator.(*Aggregator).Start
    	/Users/wawrzek/Ethereum/EigenLayer/incredible-squaring-avs/aggregator/aggregator.go:143
    main.aggregatorMain
    	/Users/wawrzek/Ethereum/EigenLayer/incredible-squaring-avs/aggregator/cmd/main.go:57
    github.com/urfave/cli.HandleAction
    	/Users/wawrzek/go/pkg/mod/github.com/urfave/[email protected]/app.go:524
    github.com/urfave/cli.(*App).Run
    	/Users/wawrzek/go/pkg/mod/github.com/urfave/[email protected]/app.go:286
    main.main
    	/Users/wawrzek/Ethereum/EigenLayer/incredible-squaring-avs/aggregator/cmd/main.go:33
    runtime.main
    	/opt/homebrew/Cellar/go/1.22.2/libexec/src/runtime/proc.go:271

Makefile diff

diff --git a/Makefile b/Makefile
index a2f5164..435d00e 100644
--- a/Makefile
+++ b/Makefile
@@ -6,8 +6,8 @@ help:

 AGGREGATOR_ECDSA_PRIV_KEY=0x2a871d0798f97d79848a013d4936a73bf4cc922c825d33c1cf7073dff6d409c6
 CHALLENGER_ECDSA_PRIV_KEY=0x5de4111afa1a4b94908f83103eb1f1706367c2e68ca870fc3fb9a804cdab365a
-DEPLOYER_PRIVATE_KEY=0x2a871d0798f97d79848a013d4936a73bf4cc922c825d33c1cf7073dff6d409c6
-CHAINID=31337
+DEPLOYER_PRIVATE_KEY=0x2e0834786285daccd064ca17f1654f67b4aef298acbb82cef9ec422fb4975622
+CHAINID=32382
 # Make sure to update this if the strategy address changes
 # check in contracts/script/output/${CHAINID}/credible_squaring_avs_deployment_output.json
 STRATEGY_ADDRESS=0x7a2088a1bFc9d81c55368AE168C2C02570cB814F

[samlaf]

That's very strange. I don't think this is related to advancing the chain by 100 blocks upon startup, that's for separate reasons.

You're hitting this require: https://github.com/Layr-Labs/eigenlayer-middleware/blob/a23de118e7d16081d350c7f83c24261d1421b0ba/src/BLSSignatureChecker.sol#L115

We recently updated the check to be an inequality < instead of <=, so now you can't use a RBN that is the current block.
But when submitting a tx the block is advanced (new block is mined when tx is submitted), which seems to indicate that a RBN of currentBlockNumber + 1 would be set..? That seems very strange.

Let me know if this helps... hopefully this will be enough for you to debug the issue.

[wawrzek]

@samlaf looks like you are right. I applied the following change and error went away:

diff --git a/src/BLSSignatureChecker.sol b/src/BLSSignatureChecker.sol
index c79a225..84a13c7 100644
--- a/src/BLSSignatureChecker.sol
+++ b/src/BLSSignatureChecker.sol
@@ -112,7 +112,7 @@ contract BLSSignatureChecker is IBLSSignatureChecker {
             "BLSSignatureChecker.checkSignatures: input nonsigner length mismatch"
         );

-        require(referenceBlockNumber < uint32(block.number), "BLSSignatureChecker.checkSignatures: invalid reference block");
+        require(referenceBlockNumber <= uint32(block.number), "BLSSignatureChecker.checkSignatures: invalid reference block");

         // This method needs to calculate the aggregate pubkey for all signing operators across
         // all signing quorums. To do that, we can query the aggregate pubkey for each quorum

Of course, it's not something I cannot keep long term, or suggest to adjust. I just wanted to have a quick test.

We recently updated the check to be an inequality < instead of <=, so now you can't use a RBN that is the current block. But when submitting a tx the block is advanced (new block is mined when tx is submitted)

Is it true for a POS chain? My understanding is that for a POS Ethereum network, a block is validated every 12 seconds (on average). It would mean that for that this AVS is too quick for local network.

[wawrzek]

I'm not sure if's pure timing.
a) I adjusted the block time creation in Anvil (using the --block-time 10). AVS works fine.
b) I looked at the POS network, and block ticking much faster than 12 seconds. My knowledge is limited, so I don't know how it's controlled, but maybe that's the problem somewhere in the settings.

diff --git a/tests/anvil/utils.sh b/tests/anvil/utils.sh
index 84184e3..f6f9ded 100644
--- a/tests/anvil/utils.sh
+++ b/tests/anvil/utils.sh
@@ -35,6 +35,7 @@ start_anvil_docker() {
     docker run --rm -d --name anvil -p 8545:8545 $LOAD_STATE_VOLUME_DOCKER_ARG $DUMP_STATE_VOLUME_DOCKER_ARG \
         --entrypoint anvil \
         $FOUNDRY_IMAGE \
-        $LOAD_STATE_ANVIL_ARG $DUMP_STATE_ANVIL_ARG --host 0.0.0.0
-    sleep 2
+        $LOAD_STATE_ANVIL_ARG $DUMP_STATE_ANVIL_ARG --host 0.0.0.0 \
+        --block-time 10
+    sleep 12
 }

[wawrzek]

After further investigation and playing with 2 parameters in the prysm (included below), I noticed that the problem become intermediate. Sometimes no errors at all, sometimes all the time, and I had runs when the error appeared from time to time.

# Time parameters
SECONDS_PER_SLOT: 2
SLOTS_PER_EPOCH: 2