docs: data storage layout

[Aliemeka]

What

Fills in docs/data_storage.md (previously a stub already linked from the mdBook TOC) with a full reference for the storage layer: the Store/StorageBackend split and what lives in each table.

Based on this issue: #164

Contents

The page covers the whole data lifecycle, with ASCII diagrams in the style of the existing consensus docs:

• Store / StorageBackend split — the byte-oriented StorageBackend trait (read views, atomic write batches) vs the Store that owns all semantics; the RocksDB and in-memory backends; how cloned Stores share one backend and the in-memory pools across actors. Includes a layered-architecture diagram and a persisted-vs-in-memory breakdown.
• The seven tables — one subsection per Table variant with key encoding, value type, write/read/prune behavior. Documents the details the enum doc comments gloss over: the big-endian slot ‖ root key layout for early-stop pruning scans, empty-body elision via EMPTY_BODY_ROOT, and that BlockSignatures actually stores the merged aggregate proof keyed by slot ‖ root (the name is historical).
• State storage — the snapshot + parent-linked diff scheme: 1024-slot anchor snapshots, what a StateDiff stores vs recovers, and the LRU cache → snapshot → reconstruction read path, with a reconstruction-walk diagram.
• Write paths — the sequence of batches a block import commits, with a diagram of which tables each step touches, plus the honest caveat that import is not one transaction (and why that's safe).
• Pruning — immediate (live chain + in-memory buffers on finalization) vs deferred (BlockSignatures below the ~1-day window), and what is never pruned.
• In-memory only — the attestation payload/gossip-signature buffers and state cache, their capacities, and what is lost on restart.
• Startup and restore — the three constructors (genesis, checkpoint sync, DB resume) and the MAX_RESUMABLE_DB_STATE_AGE fallback to checkpoint sync.

Why

The DB layout has grown complex (block data split across three tables, diff-layer state storage, two pruning paths), but the only documentation was the terse Table enum doc comments — one of which (BlockSignatures) has drifted from the implementation. A page in docs/ next to the other architecture docs gives contributors a single accurate reference for what is on disk, what is in memory, and what survives a restart.

Verification

Every claim was fact-checked against the source in crates/storage and crates/blockchain/src/store.rs: table key encodings (encode_slot_root_key), the constants (SNAPSHOT_ANCHOR_INTERVAL = 1024, STATE_CACHE_CAPACITY = 32, SIGNATURE_PRUNING_RANGE = 21_600, MAX_RESUMABLE_DB_STATE_AGE = 450, buffer caps 64/512/2048), the on_block commit sequence, the pruning conditions, and the constructor/restore paths. mdbook build passes, including the linkcheck2 backend that validates the internal links.

Related issue

Closes #164

[greptile-apps]

Greptile Summary

This PR adds a full reference for the storage layer. The main changes are:

• Documents the Store and StorageBackend split.
• Describes all seven tables and their key layouts.
• Explains state reconstruction, block writes, and pruning.
• Covers in-memory buffers and startup restoration.
• Adds the page to the mdBook navigation.

Confidence Score: 4/5

The crash-recovery description should be corrected before merging.

• Checkpoint metadata can commit before the corresponding block and state.
• A crash in that window can leave fork choice pointing to unavailable persisted data.
• The remaining table, pruning, reconstruction, and navigation details match the implementation.

docs/data_storage.md

Important Files Changed

Filename	Overview
docs/data_storage.md	Adds the storage reference; the crash-recovery section overstates safety across the checkpoint-before-block commit window.
docs/SUMMARY.md	Adds a valid navigation entry for the new data storage page.

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
docs/data_storage.md:300-305
**Checkpoint Commit Breaks Crash Recovery**

`update_checkpoints()` commits justified metadata before the block and state batches. If the process crashes between those commits, restart can load a justified root with no block, `LiveChain` entry, or state; fork choice may then select that unavailable root and panic when `update_head()` expects its state to exist. Re-import can repair the store, but the on-disk state is not self-reconciling as this paragraph claims.

_{Reviews (1): Last reviewed commit: "update: clarify description of prune_liv..." | Re-trigger Greptile}

[MegaRedHand]

Looks good! The diagrams have some confusing parts. Once that's fixed, we can merge this

[MegaRedHand]

Thanks for raising this. Opened #527

[MegaRedHand]

This is a problem, thanks for raising it. Opened #528 to fix it

[MegaRedHand]

Thanks for your contribution!