feat: add pagination to GetNetworkAccountIds #1452

SantiagoPittella · 2025-12-12T19:33:24Z

I'm working on #1299 .

I noticed that we do not store the block in which an account is created, so the typical pagination that we are using in the other endpoint does not work here. We would need some kind of cursor to keep track the position of the overall query for the pagination. The problem is that we cannot count on any of the fields in the accounts schema to be ordered. A solution could be to add a created_at field storing the block number in which the account was deployed.

cc @Mirko-von-Leipzig @drahnr

bobbinth · 2025-12-12T20:03:03Z

I noticed that we do not store the block in which an account is created, so the typical pagination that we are using in the other endpoint does not work here. We would need some kind of cursor to keep track the position of the overall query for the pagination. The problem is that we cannot count on any of the fields in the accounts schema to be ordered. A solution could be to add a created_at field storing the block number in which the account was deployed.

This is for the accounts table, right? And the proposal is to add created_at field to the accounts table so that we have a stable way to sort accounts (i.e., to avoid accounts getting re-sorted if a new account is added or an existing account is updated) - correct?

Mirko-von-Leipzig · 2025-12-13T04:51:40Z

I think we can technically rely on rowid here if we wanted since we never delete an account again (for now). That said, I think having the created_at height will be useful at some stage in any case. It can also be useful to short-circuit other queries perhaps e.g. if someone is querying the account state at some height this can be a quicker check than checking all account deltas.

Its a cheap column to add so imo we can do it.

…tion

drahnr · 2025-12-19T19:17:00Z

crates/ntx-builder/src/store.rs

        Ok(all_notes)
    }

    // TODO: add pagination.


I think this can be removed?

drahnr · 2025-12-19T19:17:46Z

crates/ntx-builder/src/store.rs

-        let response = self.inner.clone().get_network_account_ids(()).await?.into_inner();
+    pub async fn get_network_account_ids(
+        &self,
+        block_range: Option<RangeInclusive<BlockNumber>>,


Can we avoid the Option? Nobody should request this without anyways, no?

I actually think we need to retain the original behavior within this method. The ntx builder currently expects this to return all network account IDs.

To retain this behavior for now I would

revert function signature change

internally loop over and aggregate the returned pages

As currently used this will only return the first page in the main builder https://github.com/0xMiden/miden-node/pull/1452/changes#diff-630c8daca8771b806879dbee6a3059b92341ac7418c345180f537c911dd272a2R15

@SantiagoPittella ^^

I implemented this changes

drahnr · 2025-12-19T20:09:54Z

crates/store/src/db/models/queries/accounts.rs

-        schema::accounts::account_id,
+    block_range: RangeInclusive<BlockNumber>,
+) -> Result<(Vec<AccountId>, BlockNumber), DatabaseError> {
+    const ROW_OVERHEAD_BYTES: usize = AccountId::SERIALIZED_SIZE;


future work/probably too much: we might want to use build.rs that generates this const value

Should I create an issue for this one?

I'd like @Mirko-von-Leipzig 's input on it first :)

This is an internal only endpoint, so I think we can get away with simplifying this as much as possible such that we don't need a derived MAX_ROWS at all, but can simply keep it at some arbitrary value.

With a small bit of refactoring in the ntx builder (briefly touched upon in #1478 (comment)), we can simplify this function signature to

pub(crate) fn select_all_network_account_ids( conn: &mut SqliteConnection, token: ContinuationToken, ) -> Result<(Vec<AccountId>, ContinuationToken), DatabaseError>

This is because the ntx builder won't need to care about block-delimited pages, and instead only needs a mechanism to get all currently committed network account IDs.

So all we need is an iteration key that retains the order, which could be

ORDER BY rowid -- if we don't delete, or ORDER BY created_at, account_id -- for an agnostic one

and then

struct ContinuationToken(rowid); // or struct ContinuationToken(BlockNumber, AccountId);

If we do decide to go the stream route, then ContinuationToken would remain internal only, otherwise it forms part of the protobuf.

crates/store/src/db/models/queries/accounts.rs

Mirko-von-Leipzig

Overall looks good, but definitely requires the fix on the ntx client side.

I would also be interested in an experiment where we change the proto to return a stream of AccountId instead. Internally, the store would manage the stream by paginating internally (which no longer needs to be block bound, but can use rowid or any other internally consistent row token). e.g. request a page from db, stream the page, request next page, stream it etc.

Client can recover from a broken stream by initializing stream with Option<AccountI>, the last account ID received. Though we could also just restart from scratch since its an internal only API.

Mirko-von-Leipzig · 2025-12-22T11:31:45Z

crates/ntx-builder/src/store.rs

-        let response = self.inner.clone().get_network_account_ids(()).await?.into_inner();
+    pub async fn get_network_account_ids(
+        &self,
+        block_range: Option<RangeInclusive<BlockNumber>>,


I actually think we need to retain the original behavior within this method. The ntx builder currently expects this to return all network account IDs.

To retain this behavior for now I would

revert function signature change

internally loop over and aggregate the returned pages

As currently used this will only return the first page in the main builder https://github.com/0xMiden/miden-node/pull/1452/changes#diff-630c8daca8771b806879dbee6a3059b92341ac7418c345180f537c911dd272a2R15

@SantiagoPittella ^^

crates/store/src/db/models/queries/accounts.rs

…tion

SantiagoPittella · 2025-12-23T20:05:16Z

@Mirko-von-Leipzig I created #1478 to test the stream approach

drahnr · 2025-12-26T00:35:48Z

crates/store/src/db/models/queries/accounts.rs

+
+    const _: () = assert!(
+        MAX_ROWS > miden_objects::MAX_ACCOUNTS_PER_BLOCK,
+        "Block pagination limit must exceed maximum block capacity"


nit:

Suggested change

"Block pagination limit must exceed maximum block capacity"

"Block pagination limit must exceed maximum block capacity to uphold assumed logic invariant"

…tion

bobbinth

Looks good! Thank you! I left a few comments inline - mostly related to making sure the comments/docs are up to date.

bobbinth · 2025-12-29T09:57:05Z

crates/store/src/db/models/queries/accounts.rs

 /// Returns all network account IDs.
 ///
 /// # Returns
 ///
 /// A vector with network account IDs, or an error.
 pub(crate) fn select_all_network_account_ids(


I don't think doc comments here are correct - i.e., this does not return all network account IDs. Let's update it to indicate that this tries to return all IDs within the specified block range (based on account creation block), but could return less if there are too many account IDs (we should specify what's the maximum number of account IDs that can be returned).

bobbinth · 2025-12-29T10:03:28Z

crates/store/src/db/models/queries/accounts.rs

+    if let Some(&(_, last_created_at_block)) = account_ids_raw.last()
+        && account_ids_raw.len() > MAX_ROWS
+    {


nit: to improve readability, I think we should be able to write this as something like this:

if account_ids_raw.len() > MAX_ROWS { let Some(&(_, last_created_at_block)) = account_ids_raw.last().expect("..."); ... }

bobbinth · 2025-12-29T10:10:27Z

crates/store/src/db/models/queries/accounts.rs

 pub(crate) fn upsert_accounts(
    conn: &mut SqliteConnection,
    accounts: &[BlockAccountUpdate],
    block_num: BlockNumber,
 ) -> Result<usize, DatabaseError> {


Not for this PR, but this method has gotten pretty complicated with multiple requests to the database being made. Let's create an issue to refactor it so that we issue only one read and one write request during this method (I think this should be enough).

I created #1483

bobbinth · 2025-12-29T10:10:54Z

crates/store/src/db/mod.rs

    /// Loads all network account IDs from the DB.
    #[instrument(level = "debug", target = COMPONENT, skip_all, ret(level = "debug"), err)]
-    pub async fn select_all_network_account_ids(&self) -> Result<Vec<AccountId>> {
-        self.transact("Get all network account IDs", queries::select_all_network_account_ids)
-            .await
+    pub async fn select_all_network_account_ids(
+        &self,
+        block_range: RangeInclusive<BlockNumber>,
+    ) -> Result<(Vec<AccountId>, BlockNumber)> {


Doc comments here need to be updated.

bobbinth · 2025-12-29T10:11:23Z

crates/store/src/server/ntx_builder.rs

    // TODO: add pagination.
    #[instrument(
        parent = None,
        target = COMPONENT,
        name = "store.ntx_builder_server.get_network_account_ids",
        skip_all,
        ret(level = "debug"),
        err
    )]
    async fn get_network_account_ids(


Doc comments here need to be updated.

bobbinth · 2025-12-29T10:12:04Z

crates/store/src/state.rs

    /// Returns account IDs for all public (on-chain) network accounts.
-    pub async fn get_all_network_accounts(&self) -> Result<Vec<AccountId>, DatabaseError> {
-        self.db.select_all_network_account_ids().await
+    pub async fn get_all_network_accounts(
+        &self,
+        block_range: RangeInclusive<BlockNumber>,
+    ) -> Result<(Vec<AccountId>, BlockNumber), DatabaseError> {


Doc comments here need to be updated.

bobbinth · 2025-12-29T10:14:48Z

crates/ntx-builder/src/store.rs

-            .collect();
-
-        Ok(accounts?)
+        const MAX_ITERATIONS: u32 = 1000;


Would be great to add a comment here indicating why 1000 iterations is enough or at least state what kind of assumption we are making here about the maximum number of network accounts.

draft: add pagination to GetNetworkAccountIds

a8f3c0b

SantiagoPittella added 2 commits December 16, 2025 16:43

chore: add created_at_block field to accounts table

d356cb4

docs: add changelog entry

c1ca1e9

SantiagoPittella marked this pull request as ready for review December 16, 2025 19:51

SantiagoPittella changed the title ~~draft: add pagination to GetNetworkAccountIds~~ feat: add pagination to GetNetworkAccountIds Dec 16, 2025

SantiagoPittella added 2 commits December 17, 2025 12:43

Merge branch 'next' into santiagopittella-get-network-accounts-pagina…

28ce096

…tion

Merge branch 'next' into santiagopittella-get-network-accounts-pagina…

c4b33cb

…tion

SantiagoPittella requested review from Mirko-von-Leipzig and drahnr December 17, 2025 15:46

drahnr reviewed Dec 19, 2025

View reviewed changes

crates/store/src/db/models/queries/accounts.rs Show resolved Hide resolved

drahnr reviewed Dec 19, 2025

View reviewed changes

crates/store/src/db/models/queries/accounts.rs Show resolved Hide resolved

Mirko-von-Leipzig requested changes Dec 22, 2025

View reviewed changes

SantiagoPittella added 4 commits December 23, 2025 16:52

review: add loop to fetch all account, add docs

e3bf418

review: add static check for accounts limit

c1561c6

Merge branch 'next' into santiagopittella-get-network-accounts-pagina…

cf67a55

…tion

fix: lint

88b139e

This was referenced Dec 23, 2025

Revisit and add missing conversions to conv.rs #1477

Open

Stream GetNetworkAccountIds response #1478

Open

Mirko-von-Leipzig approved these changes Dec 24, 2025

View reviewed changes

drahnr reviewed Dec 26, 2025

View reviewed changes

bobbinth and others added 2 commits December 28, 2025 02:19

Merge branch 'next' into santiagopittella-get-network-accounts-pagina…

d1b7060

…tion

chore: fix merge error

a4a7fea

bobbinth reviewed Dec 29, 2025

View reviewed changes

review: improve assert comment

83b0bb4

review: add pagination to docs

cd46c29

SantiagoPittella mentioned this pull request Jan 5, 2026

Refactor upsert_accounts to reduce database round-trips #1483

Open

	"Block pagination limit must exceed maximum block capacity"
	"Block pagination limit must exceed maximum block capacity to uphold assumed logic invariant"

feat: add pagination to GetNetworkAccountIds #1452

Are you sure you want to change the base?

feat: add pagination to GetNetworkAccountIds #1452

Uh oh!

Conversation

SantiagoPittella commented Dec 12, 2025

Uh oh!

bobbinth commented Dec 12, 2025

Uh oh!

Mirko-von-Leipzig commented Dec 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mirko-von-Leipzig Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Mirko-von-Leipzig left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SantiagoPittella commented Dec 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bobbinth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Mirko-von-Leipzig Dec 26, 2025 •

edited

Loading