feat: add signature aggregation using python bindings #238

anshalshukla · 2025-12-18T09:39:19Z

🗒️ Description

Supports aggregation as per our discussion for devnet2.

The current implementation of aggregation bindings don't actually aggregate the signatures and instead return a single 0 byte, same for the verification API. This will be updated when we have aggregation support for current signature encoding scheme on TEST_CONFIG.

✅ Checklist

Ran tox checks to avoid unnecessary CI fails:
```
uvx tox
```
Considered adding appropriate tests for the changes.
Considered updating the online docs in the ./docs/ directory.

src/lean_spec/types/byte_arrays.py

src/lean_spec/subspecs/xmss/aggregation.py

src/lean_spec/subspecs/forkchoice/store.py

…ations

src/lean_spec/subspecs/containers/block/block.py

src/lean_spec/subspecs/containers/state/state.py

anshalshukla · 2025-12-22T11:43:07Z

In my bindings of leanMultisig I've added a is_test flag which when set return a single 0 byte, similarly in verification API with the is_test flag set it returns true if the payload is single 0 byte.

I've commented out testcases related to invalid signatures as current the aggregation & verification is in test mode.

Once we have working devnet-2 branch on leanMultisig for TEST_CONFIG signatures, it will be a basic upgrade to remove the is_test flag from the aggregation and verification APIs in aggregation.py along with other updates on the bindings repo. I think given the minimal changes we will require here once we have the aggregation stuff starts working we can merge this PR. Client teams can start working on this and they should be able to aggregate as sigs as well as devnet-2 branch works fine for PROD_CONFIG.

src/lean_spec/subspecs/forkchoice/store.py

packages/testing/src/consensus_testing/test_fixtures/fork_choice.py

src/lean_spec/subspecs/containers/block/block.py

src/lean_spec/subspecs/containers/state/state.py

unnawut · 2025-12-26T06:53:33Z

src/lean_spec/subspecs/containers/block/block.py

-            )
-
-            attestation_root = aggregated_attestation.data.data_root_bytes()
+            attestation_data_root = aggregated_attestation.data.data_root_bytes()


Quite minor and I should've raised in the previous PR. But just want to flag this for quick opinion @tcoratger spec-writing wise...

Since data_root_bytes() is doing just bytes(hash_tree_root(self)), should we do this instead and remove data_root_bytes()?

Suggested change

attestation_data_root = aggregated_attestation.data.data_root_bytes()

attestation_data_root = hash_tree_root(aggregated_attestation.data)

also the bytes(...) can be handled inside verify_aggregated_payload() so we don't lose type information early

I'm open to both; I don't have a preference. It seems to me that this pattern is used a lot in the code, so perhaps having the utility is simpler and less verbose.

Of course, if there are only one or two uses across the entire codebase, that is useless from a spec point of view.

unnawut · 2025-12-26T09:20:38Z

I wonder for the process of build_block(), do we need to manipulate aggregated attestations at all?

If I understand these parts correctly (and I have low confidence I do 😅 ), I think these are all to do with re-using/merging/splitting/deduplicating aggregated attestations that are the complex ones:

But can we say that:

Aggregated attestations that are already included in a block is final. Once it's processed, it becomes part of state.justifications and so it never needs to be revisited/reused. It can be dropped after processing into the state. The only need for storing it is for slashing detection (which we don't consider soon).
New individual attestations can be aggregated without concerning about existing aggregations. Because 1) leanMultisig can merge duplicate attestations, 2) Duplicate attestations don't affect the state.
Each block can already have multiple aggregated attestations, so late aggregation and on-time aggregation don't compete for a block. So multiple aggregations can keep flowing in anyway without the merging/splitting.
Eventually we'll have a separate aggregated_attestation gossip topic. We can do the merging by listening & responding to gossips than being part of block building? This would allow for a separate aggregator role in the future too.

If this all makes sense then maybe we can drop all the aggregation merging and splitting from the specs and only need 2 features around aggregation?

Build block by aggregating individual attestations per unique attestation data. There may be chances a block has multiple aggregations due to different attestation data
Ability to process multiple aggregated attestations from a block.body.attestations

tcoratger · 2025-12-26T11:07:55Z

packages/testing/src/consensus_testing/test_fixtures/fork_choice.py

+        else:
+            signature = Signature(
+                path=HashTreeOpening(siblings=HashDigestList(data=[])),
+                rho=Randomness(data=[Fp(0) for _ in range(PROD_CONFIG.RAND_LEN_FE)]),


I don't think that we should have any PROD_CONFIG in the codebase, we should use the env variable right?

tcoratger · 2025-12-26T11:16:01Z

packages/testing/src/consensus_testing/keys.py

+        For each aggregated attestation, collect the participating validators' public keys and
+        signatures, then produce a single leanVM aggregated signature proof blob using
+        `xmss_aggregate_signatures` (via `aggregate_signatures`).


These implementation details should change in the future (especially when we will not rely on the bindings anylire) so no need to add them here in the doc.

Suggested change

For each aggregated attestation, collect the participating validators' public keys and

signatures, then produce a single leanVM aggregated signature proof blob using

`xmss_aggregate_signatures` (via `aggregate_signatures`).

For each aggregated attestation, collect the participating validators' public keys and

signatures, then produce a single leanVM aggregated signature proof.

tcoratger · 2025-12-26T11:20:00Z

src/lean_spec/subspecs/containers/block/block.py

-            )
-
-            attestation_root = aggregated_attestation.data.data_root_bytes()
+            attestation_data_root = aggregated_attestation.data.data_root_bytes()


I'm open to both; I don't have a preference. It seems to me that this pattern is used a lot in the code, so perhaps having the utility is simpler and less verbose.

Of course, if there are only one or two uses across the entire codebase, that is useless from a spec point of view.

tcoratger · 2025-12-26T11:29:11Z

src/lean_spec/subspecs/containers/block/types.py

-                return True
-            seen.add(attestation.data)
-        return False
+    def each_duplicate_attestation_has_unique_participant(self) -> bool:


Here maybe we could have a less verbose name such as validate_unique_contributions ? Or maybe another idea?

tcoratger · 2025-12-26T11:33:43Z

src/lean_spec/subspecs/containers/block/types.py

+        groups: dict[bytes, list[AggregationBits]] = defaultdict(list)
+
+        for att in self:
+            groups[att.data.data_root_bytes()].append(att.aggregation_bits)
+
+        for bits_list in groups.values():
+            if len(bits_list) <= 1:
+                continue
+
+            counts: Counter[int] = Counter()
+
+            # Pass 1: count participants across the group
+            for bits in bits_list:
+                for i, bit in enumerate(bits.data):
+                    if bit:
+                        counts[i] += 1
+
+            # Pass 2: each attestation must have a participant that appears exactly once
+            for bits in bits_list:
+                unique = False
+                for i, bit in enumerate(bits.data):
+                    if bit and counts[i] == 1:
+                        unique = True
+                        break
+                if not unique:
+                    return False
+
+        return True


Something like this should be more pythonic and compact no?

groups = defaultdict(list) for att in self: groups[att.data.data_root_bytes()].append(att.aggregation_bits) for bits_list in groups.values(): if len(bits_list) <= 1: continue # 1. Convert bitfields to sets of active indices sets = [{i for i, bit in enumerate(bits.data) if bit} for bits in bits_list] # 2. Count index occurrences across the entire group counts = Counter(idx for s in sets for idx in s) # 3. Verify EVERY attestation has ANY index that appears EXACTLY once if not all(any(counts[i] == 1 for i in s) for s in sets): return False return True

yes looks simple to read

tcoratger · 2025-12-26T12:50:20Z

tests/lean_spec/subspecs/containers/test_state_aggregation.py

+def make_attestation(validator_index: int, data: AttestationData) -> Attestation:
+    """Create an attestation for the provided validator."""
+    return Attestation(validator_id=Uint64(validator_index), data=data)


Same I think that is useless

tcoratger · 2025-12-26T12:52:34Z

tests/lean_spec/subspecs/containers/test_state_aggregation.py

+# ============================================================================
+# Additional edge case tests for _aggregate_signatures_from_gossip
+# ============================================================================


No need for this we can directly put the other tests without this comment.

Suggested change

# ============================================================================

# Additional edge case tests for _aggregate_signatures_from_gossip

# ============================================================================

tcoratger · 2025-12-26T12:52:52Z

tests/lean_spec/subspecs/containers/test_state_aggregation.py

+# ============================================================================
+# Additional edge case tests for _aggregate_signatures_from_block_payload
+# ============================================================================


Same

Suggested change

# ============================================================================

# Additional edge case tests for _aggregate_signatures_from_block_payload

# ============================================================================

tcoratger · 2025-12-26T12:53:22Z

tests/lean_spec/subspecs/containers/test_state_aggregation.py

+# ============================================================================
+# Additional edge case tests for split_aggregated_attestations
+# ============================================================================


Suggested change

# ============================================================================

# Additional edge case tests for split_aggregated_attestations

# ============================================================================

tcoratger · 2025-12-26T12:53:39Z

tests/lean_spec/subspecs/containers/test_state_aggregation.py

+# ============================================================================
+# Additional edge case tests for compute_aggregated_signatures
+# ============================================================================


Suggested change

# ============================================================================

# Additional edge case tests for compute_aggregated_signatures

# ============================================================================

g11tech · 2025-12-27T13:24:06Z

src/lean_spec/subspecs/containers/state/state.py

+                    aggregated_payloads and aggregated_payloads.get(attestation_key)
+                )
+
+                if has_gossip_sig or has_block_payload:


this condition should always be true, we just need to track if we have gossip sig for it or not

g11tech · 2025-12-27T14:01:15Z

src/lean_spec/subspecs/containers/state/state.py

+        aggregated_payloads: dict[AttestationSignatureKey, AggregatedSignaturePayloads]
+        | None = None,
+    ) -> tuple[list[AggregatedAttestation], list[LeanAggregatedSignature]]:
+        """


i think this function is better written as to give an aggregate signature of whatever validator indices it can find it gossip for that data root and also returning pending validator ids which can be used in the next segments.

since the validators are voting every slot, most of the signatures will be covered in this, and only small remaining validator ids might need to be covered from already aggregated signatures.

this is the aggregation flow we need :

completely_aggregated_attestations = AggregatedAttestation.aggregate_by_data(attestations)

for completely_aggregated_attestation in completely_aggregated_attestations:
i) gossip_aggregated_signature, gossip_aggregated_signature_bit_list, remaining_validator_ids = self._aggregate_signatures_from_gossip
most of the times remaining_validator_ids should be almost empty or null and clients can choose to not further find aggregated attestations for the remaining_validator_ids since in the next round of voting they will get the gossip signatures if those validators are on the same fork of the chain.
ii) If client choose to go further, then from the aggregated signatures, one by one pick where aggregated signature with most remaining validators ids, will no remaining validator ids set is zero
i.e.
aggregates_signatures_list = [ ] aggregated_signature_bitlist_list = [] # add previously aggregated signature from gossip into this list we are maintaining aggregates_signatures_list.append(gossip_aggregated_signature) aggregated_signature_bitlist_list.append(gossip_aggregated_signature_bit_list) # pick and collect other aggregated signatures to cover all the validator ids while(remaining_validator_ids != []): # we should be able to find a previously aggregated signature because a vote is only imported if we have recieved from gossip or from a previously aggregated signature aggregated_signature, aggregated_signature_bitlist,remaining_validators_ids = pick_from_aggregated_signatures(remaining_validator_ids) aggregates_signatures_list.append(aggregated_signature) aggregated_signature_bitlist_list.append(aggregated_signature_bitlist) # now we can aggregate the aggregated signature list recurively into just one aggregated signature # but since for now we can't so we will just create multiple aggregated attestations with the same data, but using the # signature and bit list from here

```

feat: add signature aggregation using python bindings

b27fd5a

anshalshukla marked this pull request as draft December 18, 2025 09:39