Eng 824/auctioneer #1839

SuperFluffy · 2024-11-29T11:09:21Z

Summary

Brief summary of the changes made, ie "what?"

Background

Brief background on why these changes were made, ie "why?"

Changes

List changes which were made.

Testing

How are these changes tested?

Changelogs

Ensure all relevant changelog files are updated as necessary. See
keepachangelog for change
categories. Replace this text with e.g. "Changelogs updated." or "No updates
required." to acknowledge changelogs have been considered.

Metrics

List out metrics added by PR, delete section if none.

Breaking Changelist

Bulleted list of breaking changes, any notes on migration. Delete section if none.

Related Issues

Link any issues that are related, prefer full GitHub links.

closes

## Summary This updates the protos used for the auctioneer to be based on the new v1 protos. ## Background We bumped our proto definition versions from `v1alpha_` to `v1`. ## Changes - Moved the `sequencerblock/optimistic_block` file into its own subpackage in order to have `sequencerblock.optimisticblock.v1alpha1` - Updated to use `FilteredSequencerBlock` from v1 ## Breaking Changelist - This shouldn't break anything as none of these protos are used by anything in main currently. closes #1767

## Summary Ignore two new RustSec warnings. ## Background We get two non-critical warnings when running `cargo audit`: [RUSTSEC-2024-0384](https://rustsec.org/advisories/RUSTSEC-2024-0384) and [RUSTSEC-2024-0388](https://rustsec.org/advisories/RUSTSEC-2024-0388). <details><summary>RUSTSEC-2024-0384</summary><p> ``` Crate: instant Version: 0.1.13 Warning: unmaintained Title: `instant` is unmaintained Date: 2024-09-01 ID: RUSTSEC-2024-0384 URL: https://rustsec.org/advisories/RUSTSEC-2024-0384 Dependency tree: instant 0.1.13 ├── fastrand 1.9.0 │ └── futures-lite 1.13.0 │ ├── isahc 1.7.2 │ │ └── astria-sequencer-relayer 1.0.0 │ └── http-types 2.12.0 │ └── wiremock 0.5.22 │ ├── astria-sequencer-relayer 1.0.0 │ ├── astria-sequencer-client 0.1.0 │ │ ├── astria-sequencer-relayer 1.0.0 │ │ ├── astria-conductor 1.0.0 │ │ ├── astria-composer 1.0.0 │ │ ├── astria-cli 0.5.1 │ │ └── astria-bridge-withdrawer 1.0.1 │ ├── astria-conductor 1.0.0 │ ├── astria-composer 1.0.0 │ └── astria-bridge-withdrawer 1.0.1 ├── ethers-providers 2.0.14 │ ├── ethers-middleware 2.0.14 │ │ └── ethers 2.0.14 │ │ ├── astria-test-utils 0.1.0 │ │ │ └── astria-composer 1.0.0 │ │ ├── astria-composer 1.0.0 │ │ ├── astria-cli 0.5.1 │ │ ├── astria-bridge-withdrawer 1.0.1 │ │ └── astria-bridge-contracts 0.1.0 │ │ ├── astria-cli 0.5.1 │ │ └── astria-bridge-withdrawer 1.0.1 │ ├── ethers-contract 2.0.14 │ │ ├── ethers-middleware 2.0.14 │ │ └── ethers 2.0.14 │ └── ethers 2.0.14 └── ethers-middleware 2.0.14 ``` </p></details> <details><summary>RUSTSEC-2024-0388</summary><p> ``` Crate: derivative Version: 2.2.0 Warning: unmaintained Title: `derivative` is unmaintained; consider using an alternative Date: 2024-06-26 ID: RUSTSEC-2024-0388 URL: https://rustsec.org/advisories/RUSTSEC-2024-0388 Dependency tree: derivative 2.2.0 ├── penumbra-tct 0.80.7 │ ├── penumbra-txhash 0.80.7 │ │ └── penumbra-ibc 0.80.7 │ │ ├── astria-sequencer 1.0.0 │ │ └── astria-core 0.1.0 │ │ ├── astria-sequencer-utils 0.1.0 │ │ ├── astria-sequencer-relayer 1.0.0 │ │ ├── astria-sequencer-client 0.1.0 │ │ │ ├── astria-sequencer-relayer 1.0.0 │ │ │ ├── astria-conductor 1.0.0 │ │ │ ├── astria-composer 1.0.0 │ │ │ ├── astria-cli 0.5.1 │ │ │ └── astria-bridge-withdrawer 1.0.1 │ │ ├── astria-sequencer 1.0.0 │ │ ├── astria-core 0.1.0 │ │ ├── astria-conductor 1.0.0 │ │ ├── astria-composer 1.0.0 │ │ ├── astria-cli 0.5.1 │ │ ├── astria-bridge-withdrawer 1.0.1 │ │ └── astria-bridge-contracts 0.1.0 │ │ ├── astria-cli 0.5.1 │ │ └── astria-bridge-withdrawer 1.0.1 │ ├── penumbra-sct 0.80.7 │ │ └── penumbra-ibc 0.80.7 │ └── penumbra-keys 0.80.7 │ └── penumbra-sct 0.80.7 ├── penumbra-num 0.80.7 │ ├── penumbra-ibc 0.80.7 │ └── penumbra-asset 0.80.7 │ ├── penumbra-keys 0.80.7 │ └── penumbra-ibc 0.80.7 ├── penumbra-keys 0.80.7 ├── penumbra-asset 0.80.7 ├── ark-r1cs-std 0.4.0 │ ├── poseidon377 1.2.0 │ │ ├── penumbra-tct 0.80.7 │ │ ├── penumbra-sct 0.80.7 │ │ ├── penumbra-keys 0.80.7 │ │ └── penumbra-asset 0.80.7 │ ├── poseidon-permutation 1.1.0 │ │ └── poseidon377 1.2.0 │ ├── penumbra-tct 0.80.7 │ ├── penumbra-sct 0.80.7 │ ├── penumbra-num 0.80.7 │ ├── penumbra-keys 0.80.7 │ ├── penumbra-asset 0.80.7 │ └── decaf377 0.10.1 │ ├── poseidon377 1.2.0 │ ├── poseidon-permutation 1.1.0 │ ├── poseidon-parameters 1.1.0 │ │ ├── poseidon377 1.2.0 │ │ └── poseidon-permutation 1.1.0 │ ├── penumbra-tct 0.80.7 │ ├── penumbra-sct 0.80.7 │ ├── penumbra-num 0.80.7 │ ├── penumbra-keys 0.80.7 │ ├── penumbra-asset 0.80.7 │ ├── decaf377-rdsa 0.11.0 │ │ ├── penumbra-sct 0.80.7 │ │ ├── penumbra-proto 0.80.7 │ │ │ ├── penumbra-txhash 0.80.7 │ │ │ ├── penumbra-tct 0.80.7 │ │ │ ├── penumbra-sct 0.80.7 │ │ │ ├── penumbra-num 0.80.7 │ │ │ ├── penumbra-keys 0.80.7 │ │ │ ├── penumbra-ibc 0.80.7 │ │ │ ├── penumbra-asset 0.80.7 │ │ │ ├── astria-sequencer 1.0.0 │ │ │ └── astria-core 0.1.0 │ │ ├── penumbra-num 0.80.7 │ │ ├── penumbra-keys 0.80.7 │ │ └── penumbra-asset 0.80.7 │ ├── decaf377-ka 0.80.7 │ │ └── penumbra-keys 0.80.7 │ └── decaf377-fmd 0.80.7 │ ├── penumbra-proto 0.80.7 │ ├── penumbra-num 0.80.7 │ ├── penumbra-keys 0.80.7 │ └── penumbra-asset 0.80.7 ├── ark-poly 0.4.2 │ ├── ark-groth16 0.4.0 │ │ ├── poseidon377 1.2.0 │ │ ├── penumbra-num 0.80.7 │ │ └── decaf377 0.10.1 │ └── ark-ec 0.4.2 │ ├── poseidon377 1.2.0 │ ├── decaf377 0.10.1 │ ├── ark-r1cs-std 0.4.0 │ ├── ark-groth16 0.4.0 │ ├── ark-ed-on-bls12-377 0.4.0 │ │ ├── penumbra-tct 0.80.7 │ │ └── decaf377 0.10.1 │ ├── ark-crypto-primitives 0.4.0 │ │ └── ark-groth16 0.4.0 │ └── ark-bls12-377 0.4.0 │ ├── decaf377 0.10.1 │ └── ark-ed-on-bls12-377 0.4.0 ├── ark-ff 0.4.2 │ ├── ruint 1.12.3 │ │ └── celestia-types 0.1.1 │ │ ├── celestia-rpc 0.1.1 │ │ │ └── astria-conductor 1.0.0 │ │ ├── astria-sequencer-relayer 1.0.0 │ │ ├── astria-core 0.1.0 │ │ └── astria-conductor 1.0.0 │ ├── poseidon377 1.2.0 │ ├── poseidon-permutation 1.1.0 │ ├── penumbra-tct 0.80.7 │ ├── penumbra-sct 0.80.7 │ ├── penumbra-num 0.80.7 │ ├── penumbra-keys 0.80.7 │ ├── penumbra-ibc 0.80.7 │ ├── penumbra-asset 0.80.7 │ ├── decaf377-rdsa 0.11.0 │ ├── decaf377-ka 0.80.7 │ ├── decaf377-fmd 0.80.7 │ ├── decaf377 0.10.1 │ ├── ark-snark 0.4.0 │ │ ├── poseidon377 1.2.0 │ │ ├── penumbra-num 0.80.7 │ │ ├── decaf377 0.10.1 │ │ └── ark-crypto-primitives 0.4.0 │ ├── ark-relations 0.4.0 │ │ ├── poseidon377 1.2.0 │ │ ├── poseidon-permutation 1.1.0 │ │ ├── penumbra-tct 0.80.7 │ │ ├── penumbra-sct 0.80.7 │ │ ├── penumbra-num 0.80.7 │ │ ├── penumbra-keys 0.80.7 │ │ ├── penumbra-asset 0.80.7 │ │ ├── decaf377 0.10.1 │ │ ├── ark-snark 0.4.0 │ │ ├── ark-r1cs-std 0.4.0 │ │ ├── ark-groth16 0.4.0 │ │ └── ark-crypto-primitives 0.4.0 │ ├── ark-r1cs-std 0.4.0 │ ├── ark-poly 0.4.2 │ ├── ark-groth16 0.4.0 │ ├── ark-ed-on-bls12-377 0.4.0 │ ├── ark-ec 0.4.2 │ ├── ark-crypto-primitives 0.4.0 │ └── ark-bls12-377 0.4.0 ├── ark-ff 0.3.0 │ └── ruint 1.12.3 ├── ark-ec 0.4.2 └── ark-crypto-primitives 0.4.0 ``` </p></details> Given that the RustSec report doesn't suggest any concrete problems with either crate, and how difficult it will be to move away from these dependencies, I have just ignored these warnings in CI. ## Changes - Ignore RustSec warnings in `.cargo/audit.toml`. ## Testing Ran `cargo audit` locally. ## Changelogs No updates required - nothing really fixed or updated. ## Related Issues Closes #1796. Closes #1797.

…1689) ## Summary minor updates to the data verification and inclusion proof specs, mostly renaming/updating the types and some clarifications. ## Changes - update [specs/data-flow-and-verification.md](https://github.com/astriaorg/astria/compare/noot/specs?expand=1#diff-ec3cc86eef70a0285e3cd7fbc902eeb2ef082783cda5916b3d9b81f80e0b4917) - update [specs/sequencer-inclusion-proofs.md](https://github.com/astriaorg/astria/compare/noot/specs?expand=1#diff-0b52370ce97078002b7f4682377d6847fad36814fea97e7eab930e1edc9270ec)

## Summary removes immediate celestia balance check after performing ibc-transfer **from** celestia ## Background Current ibc-tests checks celestia wallet balance immediately after permorming an ibc transfer, this is redundant as we check the bablance change on the sequencer side. In addition to currently being broken duo to outdated expected value, the test currently passes randomly, most likely duo to latency. ## Changes - remove celestia balance checks from ibc transfer from celestia - moves ibc-transfer check in `run-without-native` above fee-asset change, avoiding latency issues. ## Testing - locally tested bad behavior by adding a 1s sleep prior to the first balance check ## Changelogs No updates required.

## Summary Adds `astria-cli sequencer fee-assets get` query command, getting all allowed fee-assets ## Background The astria CLI should have a query command for all abci query paths, the PR is part of an effort to accomplish this. It also enables more robust smoke-tests. ## Changes - Adds allowed fee assets query subcommand. ## Testing Manually runs the command against local, Dawn and mainnet. ## Changelogs Changelogs updated. ## Related Issues Link any issues that are related, prefer full GitHub links. closes #1815

…e doc comments

…tion logic

the intended way to communicate with a single server is by multiplexing over one connection. this also modifies the executed stream to not also forward something to another stream (happening outside of it, in the optimistic executor broker)

all streams were Unpin anyways

also makes pushing into the duplex stream a method on on the stream (rather than on handle to it). moves conversion of optimistic to base block to outside the channel so that it doesn't get swallowed (the error event would be happening in a long-running span, if under any span at all).

The "optimistic executioner" task was an extra indirection that provided no clear benefit over just running all stream directly in the auctioneer task.

we only to bias toward the cancellation token. all other events should be dealt with using the same (pseudorandom) selector.

…on manager

…nnection timeouts so the spans actually end

…tioneer

it was only emitted inside the future constructed by the closure used to start the reconnection. Returning it means a downstream user can explicitly log it. The next invocation of RestartingStream::next will then return Ready(None).

…ice mechanism

Auctions are very short lived tasks: it does not matter if their channels are bounded or not. After a short period they will end and their memory freed.

…uction steps the oneshot channels communicate cleanly what's happening inside the auction and it's easier to track what was already sent to the auction worker (to reject even triggering other steps on the worker from in its handle).

the trigger to start the auction timer comes from the sequencer block commitment. the trigger to start bids comes from the rollup executing the optimistic block in time. this means that the timer must start running independently of whether or not the rollup returned with the the executed block in time.

…r: just log and return

… executed

itamarreif and others added 30 commits November 19, 2024 06:26

fix(ci, core): delete unused generated code, generate overlooked code

66704f2

set msrv to 1.81, remove bin entry

837ae4e

remove unused dependencies

f00c9fb

clean up logging in main, ensure root spans exist

15fe94a

remove unused associated const

b9ea05e

rename auctioneer::inner::Auctioneer -> auctioneer::Inner and fix som…

b4c0577

…e doc comments

have bundle stream match other stream impls

9a33b04

import traits as _

5e63066

remove explicit retries for opening streams; rely on tonic's reconnec…

93a2d1e

…tion logic

core: impl Protobuf for FilteredSequencerBlock

92b2223

run sequencer streams off of a single channel

3d47389

the intended way to communicate with a single server is by multiplexing over one connection. this also modifies the executed stream to not also forward something to another stream (happening outside of it, in the optimistic executor broker)

remove needless pin projection

195f25f

all streams were Unpin anyways

flatten optimistic executioner into auctioneer

1dd405f

The "optimistic executioner" task was an extra indirection that provided no clear benefit over just running all stream directly in the auctioneer task.

have auctioneer shutdown events outside it's long running task

9ac1596

don't short circuit in the auctioneer select loop

2ef11e0

shutdown all still active auctions

5a69451

remember to shutdown all constituent tasks during shutdown

dcc40dd

move running and starting runstates into their own modules

bd7ceef

remove unnecessary exclusive ref

3a0355a

run all startup initialization futures concurrently

172bf53

auctioneer is unbiased toward all events

bf83e2b

we only to bias toward the cancellation token. all other events should be dealt with using the same (pseudorandom) selector.

treat handling of bundle events like all other events

2501b8f

use the channel singleton to pass a sequencer service client to aucti…

453aeda

…on manager

use the same rollup ID everywhere

c857188

github-actions bot added conductor pertaining to the astria-conductor crate sequencer pertaining to the astria-sequencer crate labels Dec 2, 2024

itamarreif added the auctioneer label Dec 2, 2024

SuperFluffy added 18 commits December 2, 2024 15:55

remove connect timeouts

e16952b

instrument grpc at the request/response level

37568f8

emit events for failed stream connection requests; also bring back co…

60e9da1

…nnection timeouts so the spans actually end

REMOVE FROM PR: emit span events in tracing subscriber

946fdbd

add astria-auctioneer to shortnames

a940b2e

Merge branch 'superfluffy/delete-old-generated-code' into ENG-824/auc…

e5e85bb

…tioneer

update all imports to use optimistic, not optimisticblock

739746f

unset running auction after completion

78fa646

return the connection error in restarting stream

b074a8b

it was only emitted inside the future constructed by the closure used to start the reconnection. Returning it means a downstream user can explicitly log it. The next invocation of RestartingStream::next will then return Ready(None).

fix allocation rule never taking bids with zero-bids

9c6f926

use Arcs when passing bundles to running auction; instrument first pr…

070a9e7

…ice mechanism

make the auction bundle channel unbounded

6bff3e5

Auctions are very short lived tasks: it does not matter if their channels are bounded or not. After a short period they will end and their memory freed.

small refactor: start_processing_bids -> start_bids

2ac6504

add some todos to the auction

0a8a595

not receiving winning bids does not constitute an error for auctionee…

c4ea203

…r: just log and return

skip_all not only skip(self)

82e401a

github-actions bot added documentation Improvements or additions to documentation proto pertaining to the Astria Protobuf spec sequencer-relayer pertaining to the astria-sequencer-relayer crate composer pertaining to composer cd labels Dec 5, 2024

cancel auctions instead of aborting them

779b36f

SuperFluffy force-pushed the ENG-824/auctioneer branch from c26dd21 to 779b36f Compare December 11, 2024 19:59

SuperFluffy added 2 commits December 11, 2024 21:05

some clippy changes to auctioneer

e17fe69

compare bundle parent hash against executed block hash, not parent of…

2edf112

… executed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eng 824/auctioneer #1839

Eng 824/auctioneer #1839

SuperFluffy commented Nov 29, 2024

Eng 824/auctioneer #1839

Are you sure you want to change the base?

Eng 824/auctioneer #1839

Conversation

SuperFluffy commented Nov 29, 2024

Summary

Background

Changes

Testing

Changelogs

Metrics

Breaking Changelist

Related Issues