feat: [2/4] integrate smtforest, avoid ser/de of full account/vault data in database

[drahnr]

Part 2/4 of #1185 and lays the ground for #1354

⚠️ The PR is rather large, a lot of the changes affect large pieces of code across.

Intent

• Provide the baseline for cheap querying partial storage maps, but not implement it
• Prepare for potential deprecation of AccountInfo

core changes

1. Remove the vault / storage_map BLOB entries from the accounts table.
2. Add SmtForest and integrate into apply_block and State::load

significant changes in the following files:

• crates/store/src/db/schema.rs introduces account_storage_headers and removes storage (the full serialized account storage) from accounts table
• crates/store/src/state.rs / fn State::apply_block now updates the database, but also the lookup table of roots for the SmtForest and the entries in the forest itself, indirect lookup tables

out of scope

• usage of the LargeSmtForest for partial storage maps - [devops] Deploy without nuking db #670
• individual of partial storage map queries - Implement support for retrieving subsets of storage maps in GetAccountProof endpoint #1354
• cleanup for both DB and (Large|)SmtForest - Add cleanup routine for outdated historical storage entries #1355

how to review

Take the existing TODOs into consideration, if they make sense. This will be the follow up PR.

[Mirko-von-Leipzig]

Mostly questions; no real issues found as yet

[bobbinth]

Thank you! I left some comments inline and pushed a few commits to try to organize the code a bit better. Overall, we are still quite far from being able to merge this PR as there are some bugs and logical inconsistencies.

Also, given the amount of remaining issues, I'm a bit concerned about our ability to wrap this PR up in a timely manner. Given the complexity of this PR, deep reviews take quite a bit of time. Maybe there is a way to split this up into smaller incremental PRs that we can review and merge much more quickly.

[bobbinth]

This is pretty inefficient because we just need to get a list of accounts with public state, but instead we are loading all account commitments. This is OK for now because we'll replace the initialization code as we migrate to RocksDB-based backends.

[drahnr]

We could filter as part of the query using lt/gt given a certain prefix range if the layout is advantageous, LIKE statements are not supported by diesel after taking a brief look.
EDIT: looking at the layout of the exact bits in AccountId and the prefix, it seems the pub/priv/network types are encoding it at the lower octet of the prefix, range filtering won't work.

[bobbinth]

I'm not seeing a good reason to separate InnerForest from InnerState - so, let's combine them. This could look simply as:

struct InnerState<S = MemoryStorage>
where
    S: SmtStorage,
{
    blockchain: Blockchain,
    account_tree: AccountTreeWithHistory<S>,
    nullifier_tree: NullifierTree<LargeSmt<S>>,
    forest: InnerForest,
}

In the future (when moving to a RocksDB backend), we should refactor this to move InnerState into a separate file and clean up the module hierarchy.

[drahnr]

It takes care of encapsulating the indices required for lookup and hiding the internals, which will come in handy once LargeSmtForest lands, which has a largely different API.

That's not withstanding a separate module cleanup/InnerState move.