Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Auctioneer geth #58

Draft
wants to merge 86 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
c3cfc16
fix: use mutex lock for tx removal (#51)
joroshiba Oct 9, 2024
459cc3d
Merge pull request #13 from astriaorg/bharath/cherry-pick-fix-mempool
bharath-123 Oct 16, 2024
3ce87f8
Fix issue with using a pointer to an iteration variable of a range lo…
mycodecrafting Sep 25, 2024
9f109b2
build: merge release/0.14.3 into main (#49)
mycodecrafting Oct 1, 2024
00e8930
chore: upgrade to v1 server api (#53)
joroshiba Oct 17, 2024
dcd937f
fix the fork diff (#54)
joroshiba Oct 25, 2024
7c318f4
Merge pull request #19 from astriaorg/bharath/rebase-with-astria-geth
bharath-123 Oct 29, 2024
7601571
implement optimistic block execution with tests
bharath-123 Oct 29, 2024
c22be20
use commitmentUpdateLock when accessing the safe block
bharath-123 Oct 29, 2024
d03a695
fix test
bharath-123 Oct 29, 2024
cfa9025
add a flag to check if the payload is being built optimistically
bharath-123 Sep 30, 2024
9095195
add tx validaton to optimistic block execution
bharath-123 Oct 24, 2024
539b8e1
take the block execution lock when accessing nextFeeRecipient
bharath-123 Oct 24, 2024
c925895
use v1 protos
bharath-123 Oct 29, 2024
cb8bf6f
acquire fee recipient lock after getting the softblock
bharath-123 Nov 12, 2024
c85c783
avoid taking block commitment lock when fetching softblock pointer
bharath-123 Nov 20, 2024
6aeee0e
add event for optimistic head
bharath-123 Sep 27, 2024
84c2ab3
add optimistic head subscription method to mempool Blockchain interface
bharath-123 Sep 24, 2024
556dc4a
test fixes
bharath-123 Sep 27, 2024
289b10b
use optimistic head event in txpool's maintanance loop
bharath-123 Sep 24, 2024
674bb32
fix tests
bharath-123 Oct 24, 2024
59175c7
add logic to clear mempool
bharath-123 Sep 26, 2024
ad6237d
remove invalids
bharath-123 Sep 26, 2024
57a351f
add comments
bharath-123 Sep 26, 2024
23eae86
remove mempool conditionals
bharath-123 Oct 16, 2024
89cee7d
split out reset logic and head only reset logc
bharath-123 Oct 24, 2024
af14e44
ensure that unreserved addresses are removed
bharath-123 Nov 10, 2024
963dc60
add event for mempool clearance
bharath-123 Nov 10, 2024
fe1d461
update subscription interfaces
bharath-123 Sep 27, 2024
38c6e60
fix potential panic while subscribing to mempool clearance
bharath-123 Nov 4, 2024
fd3dd2c
dont send mempool clearing event while holding mempool lock
bharath-123 Nov 10, 2024
e8f2032
implement stream execute optimistic block
bharath-123 Oct 29, 2024
d5c0aef
unit tests
bharath-123 Nov 4, 2024
1fb4f09
use generics to implement mock bi directional stream
bharath-123 Oct 1, 2024
06264bc
wip
bharath-123 Oct 16, 2024
1271668
use an atomic pointer for sequencer block hash
bharath-123 Oct 29, 2024
1c01f14
reduce mempool clearing timeout
bharath-123 Oct 24, 2024
33698a4
fix imports
bharath-123 Oct 29, 2024
5e71eb5
update grpc method names
bharath-123 Nov 5, 2024
8be3de9
only allow 1 client to be connected to the execute optimistic block s…
bharath-123 Nov 12, 2024
3013a91
rename executeBlockStreamConnected to executeOptimisticBlockStreamCon…
bharath-123 Nov 12, 2024
ae9498d
remove restrictions to allow just one client to connect to the optimi…
bharath-123 Dec 2, 2024
cad2e6c
implement bundle streaming
bharath-123 Nov 5, 2024
6d873a2
update unit tests
bharath-123 Nov 5, 2024
26ced3f
only send the effective tip as part of the fee
bharath-123 Oct 1, 2024
a4112a2
minor test updates
bharath-123 Oct 16, 2024
b5af27d
rename grpc methods
bharath-123 Nov 5, 2024
e9fcfc7
close the bundle stream when client closes the connection
bharath-123 Nov 11, 2024
690803a
allow only 1 client to connect to the bundle stream
bharath-123 Dec 2, 2024
779ff8d
fix minor error
bharath-123 Dec 2, 2024
e07a264
fetch the next fee recipient under the block execution lock
bharath-123 Oct 11, 2024
06f88fd
validate txs before optimistically executing them
bharath-123 Oct 16, 2024
d4cd2ee
support uds endpoints for auctioneer
bharath-123 Oct 16, 2024
02d6af9
remove duplicate code
bharath-123 Oct 24, 2024
76c1393
add uds flag to options
bharath-123 Nov 4, 2024
9757320
separate out execution api services and optimistic execution api serv…
bharath-123 Dec 2, 2024
8584611
minor updates
bharath-123 Oct 17, 2024
91f416d
minor nits
bharath-123 Oct 17, 2024
d81d766
make tests more robust
bharath-123 Oct 17, 2024
13feec9
move BigIntoToProtoU128 to shared test utils
bharath-123 Oct 24, 2024
929c864
minor updates
bharath-123 Oct 29, 2024
17dc6f4
update grpc methods
bharath-123 Nov 5, 2024
55d77e9
close the bundle stream when client closes the connection
bharath-123 Nov 11, 2024
4047e06
remove UDS references
bharath-123 Nov 12, 2024
7f8ab67
rename a wrongly renamed word
bharath-123 Nov 12, 2024
9b8a386
re add single client connection checks
bharath-123 Nov 12, 2024
55e1a02
maintain only 1 server instance
bharath-123 Nov 12, 2024
6268181
renaming
bharath-123 Nov 20, 2024
2d4ad7b
feature flag auctioneer
bharath-123 Nov 12, 2024
c64f277
add flags
bharath-123 Oct 29, 2024
018aff7
save
bharath-123 Nov 12, 2024
96734cd
maintain a copy of legacy pool tests to test the cases when auctionee…
bharath-123 Nov 12, 2024
99fdc20
unmarshall auction result
bharath-123 Nov 13, 2024
4f1b2d7
add signature verification
bharath-123 Nov 13, 2024
b83016a
set the trusted builder public key in genesis
bharath-123 Nov 17, 2024
06292e9
add some tests for the auction results
bharath-123 Nov 17, 2024
c12fcd3
dilineate trusted builder public keys by block number
bharath-123 Nov 19, 2024
b19fd2c
renaming
bharath-123 Nov 20, 2024
53e34f3
update protos
bharath-123 Nov 20, 2024
fe193af
change trusted builder instances to auctioneer
bharath-123 Nov 21, 2024
ac98b35
fix typo
bharath-123 Nov 21, 2024
3a24801
add an api to query the optimistic block
bharath-123 Nov 20, 2024
bb92573
support querying optimistic block using the optimistic string
bharath-123 Nov 21, 2024
a95cd26
close the stream when we get the done signal
bharath-123 Nov 27, 2024
92242a4
add debug logs for when the stream rpcs are called
bharath-123 Nov 27, 2024
ffddf4b
add some logs
bharath-123 Nov 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions cmd/devp2p/internal/ethtest/suite_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ func TestEthSuite(t *testing.T) {
if err != nil {
t.Fatalf("could not create new test suite: %v", err)
}

for _, test := range suite.EthTests() {
t.Run(test.Name, func(t *testing.T) {
if test.Slow && testing.Short() {
Expand Down Expand Up @@ -149,5 +150,6 @@ func setupGeth(stack *node.Node, dir string) error {
return fmt.Errorf("failed to register catalyst service: %v", err)
}
_, err = backend.BlockChain().InsertChain(chain.blocks[1:])
backend.BlockChain().SetOptimistic(chain.blocks[len(chain.blocks)-1])
return err
}
13 changes: 10 additions & 3 deletions cmd/geth/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ import (
"errors"
"fmt"
"github.com/ethereum/go-ethereum/eth/catalyst"
"github.com/ethereum/go-ethereum/grpc/optimistic"
"github.com/ethereum/go-ethereum/grpc/shared"
"os"
"reflect"
"runtime"
Expand Down Expand Up @@ -206,11 +208,16 @@ func makeFullNode(ctx *cli.Context) *node.Node {

// Configure gRPC if requested.
if ctx.IsSet(utils.GRPCEnabledFlag.Name) {
serviceV1a2, err := execution.NewExecutionServiceServerV1Alpha2(eth)
sharedService, err := shared.NewSharedServiceContainer(eth)
if err != nil {
utils.Fatalf("failed to create execution service: %v", err)
utils.Fatalf("failed to create shared service container: %v", err)
}
utils.RegisterGRPCExecutionService(stack, serviceV1a2, &cfg.Node)

serviceV1a2 := execution.NewExecutionServiceServerV1(sharedService)

optimisticServiceV1a1 := optimistic.NewOptimisticServiceV1Alpha(sharedService)

utils.RegisterGRPCServices(stack, serviceV1a2, optimisticServiceV1a1, optimisticServiceV1a1, &cfg.Node)
}

// Add the Ethereum Stats daemon if requested.
Expand Down
1 change: 1 addition & 0 deletions cmd/geth/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ var (
utils.MinerRecommitIntervalFlag,
utils.MinerPendingFeeRecipientFlag,
utils.MinerNewPayloadTimeoutFlag, // deprecated
utils.AuctioneerEnabledFlag,
utils.NATFlag,
utils.NoDiscoverFlag,
utils.DiscoveryV4Flag,
Expand Down
22 changes: 18 additions & 4 deletions cmd/utils/flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package utils

import (
optimisticGrpc "buf.build/gen/go/astria/execution-apis/grpc/go/astria/bundle/v1alpha1/bundlev1alpha1grpc"
"context"
"crypto/ecdsa"
"encoding/hex"
Expand All @@ -35,7 +36,7 @@ import (
"strings"
"time"

astriaGrpc "buf.build/gen/go/astria/execution-apis/grpc/go/astria/execution/v1alpha2/executionv1alpha2grpc"
astriaGrpc "buf.build/gen/go/astria/execution-apis/grpc/go/astria/execution/v1/executionv1grpc"
"github.com/ethereum/go-ethereum/accounts"
"github.com/ethereum/go-ethereum/accounts/keystore"
bparams "github.com/ethereum/go-ethereum/beacon/params"
Expand Down Expand Up @@ -769,6 +770,13 @@ var (
Category: flags.APICategory,
}

// auctioneer
AuctioneerEnabledFlag = &cli.BoolFlag{
Name: "auctioneer",
Usage: "Enable the auctioneer server",
Category: flags.MinerCategory,
}

// Network Settings
MaxPeersFlag = &cli.IntFlag{
Name: "maxpeers",
Expand Down Expand Up @@ -1438,6 +1446,12 @@ func SetNodeConfig(ctx *cli.Context, cfg *node.Config) {
SetDataDir(ctx, cfg)
setSmartCard(ctx, cfg)

if ctx.Bool(AuctioneerEnabledFlag.Name) {
cfg.EnableAuctioneer = true
} else {
cfg.EnableAuctioneer = false
}

if ctx.IsSet(JWTSecretFlag.Name) {
cfg.JWTSecret = ctx.String(JWTSecretFlag.Name)
}
Expand Down Expand Up @@ -1987,10 +2001,10 @@ func RegisterGraphQLService(stack *node.Node, backend ethapi.Backend, filterSyst
}
}

// RegisterGRPCExecutionService adds the gRPC API to the node.
// RegisterGRPCServices adds the gRPC API to the node.
// It was done this way so that our grpc execution server can access the ethapi.Backend
func RegisterGRPCExecutionService(stack *node.Node, execServ astriaGrpc.ExecutionServiceServer, cfg *node.Config) {
if err := node.NewGRPCServerHandler(stack, execServ, cfg); err != nil {
func RegisterGRPCServices(stack *node.Node, execServ astriaGrpc.ExecutionServiceServer, optimisticExecutionServ optimisticGrpc.OptimisticExecutionServiceServer, bundleStreamingServ optimisticGrpc.BundleServiceServer, cfg *node.Config) {
if err := node.NewGRPCServerHandler(stack, execServ, optimisticExecutionServ, bundleStreamingServ, cfg); err != nil {
Fatalf("Failed to register the gRPC service: %v", err)
}
}
Expand Down
54 changes: 36 additions & 18 deletions core/blockchain.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,12 @@ import (
)

var (
headBlockGauge = metrics.NewRegisteredGauge("chain/head/block", nil)
headHeaderGauge = metrics.NewRegisteredGauge("chain/head/header", nil)
headFastBlockGauge = metrics.NewRegisteredGauge("chain/head/receipt", nil)
headFinalizedBlockGauge = metrics.NewRegisteredGauge("chain/head/finalized", nil)
headSafeBlockGauge = metrics.NewRegisteredGauge("chain/head/safe", nil)
headBlockGauge = metrics.NewRegisteredGauge("chain/head/block", nil)
headHeaderGauge = metrics.NewRegisteredGauge("chain/head/header", nil)
headFastBlockGauge = metrics.NewRegisteredGauge("chain/head/receipt", nil)
headFinalizedBlockGauge = metrics.NewRegisteredGauge("chain/head/finalized", nil)
headSafeBlockGauge = metrics.NewRegisteredGauge("chain/head/safe", nil)
headOptimisticBlockGauge = metrics.NewRegisteredGauge("chain/head/optimistic", nil)

chainInfoGauge = metrics.NewRegisteredGaugeInfo("chain/info", nil)

Expand Down Expand Up @@ -219,24 +220,26 @@ type BlockChain struct {
stateCache state.Database // State database to reuse between imports (contains state cache)
txIndexer *txIndexer // Transaction indexer, might be nil if not enabled

hc *HeaderChain
rmLogsFeed event.Feed
chainFeed event.Feed
chainSideFeed event.Feed
chainHeadFeed event.Feed
logsFeed event.Feed
blockProcFeed event.Feed
scope event.SubscriptionScope
genesisBlock *types.Block
hc *HeaderChain
rmLogsFeed event.Feed
chainFeed event.Feed
chainSideFeed event.Feed
chainHeadFeed event.Feed
chainOptimisticHeadFeed event.Feed
logsFeed event.Feed
blockProcFeed event.Feed
scope event.SubscriptionScope
genesisBlock *types.Block

// This mutex synchronizes chain write operations.
// Readers don't need to take it, they can just read the database.
chainmu *syncx.ClosableMutex

currentBlock atomic.Pointer[types.Header] // Current head of the chain
currentSnapBlock atomic.Pointer[types.Header] // Current head of snap-sync
currentFinalBlock atomic.Pointer[types.Header] // Latest (consensus) finalized block
currentSafeBlock atomic.Pointer[types.Header] // Latest (consensus) safe block
currentBlock atomic.Pointer[types.Header] // Current head of the chain
currentSnapBlock atomic.Pointer[types.Header] // Current head of snap-sync
currentFinalBlock atomic.Pointer[types.Header] // Latest (consensus) finalized block
currentSafeBlock atomic.Pointer[types.Header] // Latest (consensus) safe block
currentOptimisticBlock atomic.Pointer[types.Header] // Latest optimistic block

currentBaseCelestiaHeight atomic.Uint64 // Latest finalized block height on Celestia

Expand Down Expand Up @@ -325,6 +328,7 @@ func NewBlockChain(db ethdb.Database, cacheConfig *CacheConfig, genesis *Genesis
bc.currentBlock.Store(bc.genesisBlock.Header())
bc.currentFinalBlock.Store(bc.genesisBlock.Header())
bc.currentSafeBlock.Store(bc.genesisBlock.Header())
bc.currentOptimisticBlock.Store(bc.genesisBlock.Header())
bc.currentBaseCelestiaHeight.Store(bc.Config().AstriaCelestiaInitialHeight)

// Update chain info data metrics
Expand Down Expand Up @@ -532,6 +536,7 @@ func (bc *BlockChain) loadLastState() error {
bc.currentFinalBlock.Store(block.Header())
headFinalizedBlockGauge.Update(int64(block.NumberU64()))
bc.currentSafeBlock.Store(block.Header())
bc.currentOptimisticBlock.Store(block.Header())
headSafeBlockGauge.Update(int64(block.NumberU64()))
}
}
Expand Down Expand Up @@ -639,6 +644,19 @@ func (bc *BlockChain) SetSafe(header *types.Header) {
}
}

// SetOptimistic sets the optimistic block.
func (bc *BlockChain) SetOptimistic(block *types.Block) {
header := block.Header()
bc.currentOptimisticBlock.Store(header)
if header != nil {
headOptimisticBlockGauge.Update(int64(header.Number.Uint64()))
} else {
headOptimisticBlockGauge.Update(0)
}

bc.chainOptimisticHeadFeed.Send(ChainOptimisticHeadEvent{Block: block})
}

// rewindHashHead implements the logic of rewindHead in the context of hash scheme.
func (bc *BlockChain) rewindHashHead(head *types.Header, root common.Hash) (*types.Header, uint64) {
var (
Expand Down
11 changes: 11 additions & 0 deletions core/blockchain_reader.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,12 @@ func (bc *BlockChain) CurrentSafeBlock() *types.Header {
return bc.currentSafeBlock.Load()
}

// CurrentOptimisticBlock retrieves the current optimistic block of the canonical
// chain. The block is retrieved from the blockchain's internal cache.
func (bc *BlockChain) CurrentOptimisticBlock() *types.Header {
return bc.currentOptimisticBlock.Load()
}

// CurrentBaseCelestiaHeight retrieves the current base celestia height of the
// canonical chain. The height is retrieved from the blockchain's internal cache.
func (bc *BlockChain) CurrentBaseCelestiaHeight() uint64 {
Expand Down Expand Up @@ -439,6 +445,11 @@ func (bc *BlockChain) SubscribeChainHeadEvent(ch chan<- ChainHeadEvent) event.Su
return bc.scope.Track(bc.chainHeadFeed.Subscribe(ch))
}

// SubscribeChainOptimisticHeadEvent registers a subscription of ChainOptimisticHeadEvent.
func (bc *BlockChain) SubscribeChainOptimisticHeadEvent(ch chan<- ChainOptimisticHeadEvent) event.Subscription {
return bc.scope.Track(bc.chainOptimisticHeadFeed.Subscribe(ch))
}

// SubscribeChainSideEvent registers a subscription of ChainSideEvent.
func (bc *BlockChain) SubscribeChainSideEvent(ch chan<- ChainSideEvent) event.Subscription {
return bc.scope.Track(bc.chainSideFeed.Subscribe(ch))
Expand Down
10 changes: 10 additions & 0 deletions core/events.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ import (
// NewTxsEvent is posted when a batch of transactions enter the transaction pool.
type NewTxsEvent struct{ Txs []*types.Transaction }

// NewMempoolClearedEvent is posted when the mempool is cleared after a head reset for trusted auctioneer
type NewMempoolCleared struct {
// the new head to which the mempool state was reset to before clearing the mempool
NewHead *types.Header
}

// NewMinedBlockEvent is posted when a block has been imported.
type NewMinedBlockEvent struct{ Block *types.Block }

Expand All @@ -41,3 +47,7 @@ type ChainSideEvent struct {
}

type ChainHeadEvent struct{ Block *types.Block }

type ChainOptimisticHeadEvent struct {
Block *types.Block
}
5 changes: 5 additions & 0 deletions core/txpool/blobpool/blobpool.go
Original file line number Diff line number Diff line change
Expand Up @@ -339,6 +339,7 @@ func (p *BlobPool) ClearAstriaOrdered() {}
func (p *BlobPool) AddToAstriaExcludedFromBlock(*types.Transaction) {}
func (p *BlobPool) AstriaExcludedFromBlock() *types.Transactions { return &types.Transactions{} }
func (p *BlobPool) AstriaOrdered() *types.Transactions { return &types.Transactions{} }
func (p *BlobPool) ValidateTx(tx *types.Transaction) error { return nil }

// Filter returns whether the given transaction can be consumed by the blob pool.
func (p *BlobPool) Filter(tx *types.Transaction) bool {
Expand Down Expand Up @@ -1601,6 +1602,10 @@ func (p *BlobPool) SubscribeTransactions(ch chan<- core.NewTxsEvent, reorgs bool
}
}

func (p *BlobPool) SubscribeMempoolClearance(ch chan<- core.NewMempoolCleared) event.Subscription {
return nil
}

// Nonce returns the next nonce of an account, with all transactions executable
// by the pool already applied on top.
func (p *BlobPool) Nonce(addr common.Address) uint64 {
Expand Down
Loading