feat: soc dispersed replica v2 #5279

gacevicljubisa · 2025-11-06T10:45:19Z

Checklist

I have read the coding guide.
My change requires a documentation update, and I have done it.
I have added tests to cover my changes.
I have filled out the description and linked the related issues.

Description

This PR is a continuation of the original PR #5057.

Original description

Add support for dispersed replicas on Single Owner Chunks (SOCs) to improve data availability and retrieval reliability in the Swarm network.

SOC replicas are generated by allowing additional addresses to represent the same SOC. It is achieved by lighten the validation of SOCs which ignores the first byte of the address. This makes possible to saturate dispersed replicas evenly across the whole network since nodes arrange into neighborhoods based on address prefix. The addresses created in a way to iterate over all variations in the given depth of the redundancy level + 1 (e.g. level is 2, and the original address starts with 101, then it uploads SOCs with addresses same after the 3 first bits where the first 3 bit variations are 001, 011, (101 is not because the original address has it), 111 and 100 (flipping the last bit).

New changes

Fixes SOC dispersed replica functionality by setting explicit default redundancy levels and fixing implementation issues.

Set default redundancy level to PARANOID for handlers
Use wg.Go() instead of manual goroutine management
Change redundancy level header to pointer type for proper optional handling
Refactor feeds factory to use functional options pattern
Add comprehensive tests for SOC replicas
Introduce the iterator for iterating over replica addreses one by one for better concurrency management and as the original implementation had bugs

Open API Spec Version Changes (if applicable)

feed and soc replica PUT endpoints swarm-redundancy-level header: create and push dispersed replicas according to the passed level: MEDIUM 2, STRONG 4, INSANE 8, PARANOID 16.

feed and soc replica GET endpoints swarm-redundancy-level header: for calibrating how deeply dispersed replicas should be checked. By default it is zero.

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

…utter

gacevicljubisa · 2025-11-13T08:10:46Z

pkg/api/feed.go

 	headers := struct {
-		OnlyRootChunk bool `map:"Swarm-Only-Root-Chunk"`
+		OnlyRootChunk   bool             `map:"Swarm-Only-Root-Chunk"`
+		RedundancyLevel redundancy.Level `map:"Swarm-Redundancy-Level"`


Should we use here Paranoid as default as on other endpoints?

What about all other places where Swarm-Redundancy-Level is used as header?

This is a good question. In all other handlers getRedundancyLevel is used to get the redundancy from header and if it is not set, PARANOID is used. Here in feeds, replicas getter is not used if the header is not set. I do not know it that is the expected behaviour.

Maybe Viktor should know when he reviews this PR.

I am not sure if we need headers.RedundancyLevel > redundancy.NONE at all, as replicas getter should not make any replicas requests for NONE redundancy.

Regarding redundancy.NONE, yes, that make sense, maybe we can directly use:

getter = replicas.NewSocGetter(s.storer.Download(true), rLevel)

janos · 2025-11-14T11:11:43Z

Hi @zelig. I would kindly ask you to review this PR for correctness and in general. It is a combined effort from multiple developers and we have tried to validate the behaviour through unit tests, but we would like to have your validation because of potential risks. Would you check also what should be default redundancy levels in api handlers if Swarm-Redundancy-Level is not set in the request?

janos · 2025-11-14T13:25:39Z

pkg/replicas/getter_soc.go

+					mu.Unlock()
+					return
+				}
+


@zelig Would it make sense to check if the chunk data content validates with the requested address addr? Like this:

cac.Valid(swarm.NewChunk(addr, ch.Data()))

As the replica's data hash should be equal with the chunk original address.

yes absolutely

Thanks, it is added.

zelig

try to keep PRs focussed and clean please. More comments on the concurrency and cancellation needed. How is this used in feeds?

zelig · 2025-11-21T06:42:14Z

.github/workflows/beekeeper.yml

  SETUP_CONTRACT_IMAGE_TAG: "0.9.4"
  BEELOCAL_BRANCH: "main"
-  BEEKEEPER_BRANCH: "master"
+  BEEKEEPER_BRANCH: "feat/soc-dispersed"


This is for CI to use the updated beekeeper checks. Before this PR is merged, beekeeper main will be up to date. If beekeeper main has the changes for this PR, other PRs CI checks would fail.

This is the beekeeper branch (PR) where are modifications that are able to test soc disparsed replica on CI. This needs to be reverted to master before merge. Also, beekeeper PR should be merged as well.

zelig · 2025-11-21T10:25:11Z

pkg/api/pin.go


 	getter := s.storer.Download(true)
-	traverser := traversal.New(getter, s.storer.Cache(), redundancy.DefaultLevel)
+	traverser := traversal.New(getter, s.storer.Cache(), redundancy.PARANOID)


why is this change?

zelig · 2025-11-21T12:21:48Z

pkg/hive/hive_test.go

 		} else {
 			n := (i % 3) + 1
-			for j := 0; j < n; j++ {
+			for j := range n {


what does this change have anyhting to do with the PR?

zelig · 2025-11-21T12:31:19Z

pkg/replicas/getter.go

 	defer cancel()

+	var wg sync.WaitGroup
+	defer wg.Wait()


why would you want to wait for all routines to terminate? unless you cancel them each when you get one response

This is actually needed as a test hack (to use replicas.Wait only avalilable in tests) to avoid data races is removed by nugaon. The current code is race free in production, as well.

Canceling is done when when the chunk is found by calling cancel() on line 92.

If we do not care about other goroutines, we can revert changes in getter.go and getter_test.go and export_test.go so that tests do not report data race. Should we do that?

zelig · 2025-11-21T12:35:21Z

pkg/replicas/getter_soc.go

+					mu.Unlock()
+					return
+				}
+


yes absolutely

zelig · 2025-11-21T12:43:50Z

pkg/topology/kademlia/kademlia_test.go

 	underlays := make([]ma.Multiaddr, n)

-	for i := 0; i < n; i++ {
+	for i := range n {


why here, why now?

janos

try to keep PRs focussed and clean please.

Will do.

More comments on the concurrency and cancellation needed.

Added.

How is this used in feeds?

SocGetter is used in API handlers feedGetHandler, socGetHandler and serveReference. This is left as it is from commits from the nugaon's PR. What is else required to be added?

janos · 2025-11-21T14:43:24Z

.github/workflows/beekeeper.yml

  SETUP_CONTRACT_IMAGE_TAG: "0.9.4"
  BEELOCAL_BRANCH: "main"
-  BEEKEEPER_BRANCH: "master"
+  BEEKEEPER_BRANCH: "feat/soc-dispersed"


This is for CI to use the updated beekeeper checks. Before this PR is merged, beekeeper main will be up to date. If beekeeper main has the changes for this PR, other PRs CI checks would fail.

janos · 2025-11-21T14:44:25Z

pkg/api/pin.go


 	getter := s.storer.Download(true)
-	traverser := traversal.New(getter, s.storer.Cache(), redundancy.DefaultLevel)
+	traverser := traversal.New(getter, s.storer.Cache(), redundancy.PARANOID)


janos · 2025-11-21T14:45:01Z

pkg/hive/hive_test.go

 		} else {
 			n := (i % 3) + 1
-			for j := 0; j < n; j++ {
+			for j := range n {


janos · 2025-11-21T14:48:45Z

pkg/replicas/getter.go

 	defer cancel()

+	var wg sync.WaitGroup
+	defer wg.Wait()


This is actually needed as a test hack (to use replicas.Wait only avalilable in tests) to avoid data races is removed by nugaon. The current code is race free in production, as well.

Canceling is done when when the chunk is found by calling cancel() on line 92.

If we do not care about other goroutines, we can revert changes in getter.go and getter_test.go and export_test.go so that tests do not report data race. Should we do that?

janos · 2025-11-21T14:49:53Z

pkg/replicas/getter_soc.go

+					mu.Unlock()
+					return
+				}
+


Thanks, it is added.

janos · 2025-11-21T14:53:58Z

pkg/topology/kademlia/kademlia_test.go

 	underlays := make([]ma.Multiaddr, n)

-	for i := 0; i < n; i++ {
+	for i := range n {


nugaon and others added 30 commits June 5, 2025 10:10

feat: soc dispersed replica

a684873

feat: new soc putter session

c252d50

refactor: lint issue

2f42145

test: outline race condition

19971d1

fix: randomize ids at soc generation

2ff200c

fix: data race in putter soc

40c5c17

test: soc api

b982b20

fix: same replica address logic as in validity

61d8a8d

fix: lint issue

7ec15cb

chore: comments

ec191e7

Merge remote-tracking branch 'origin/master' into feat/soc-dispersed

f1039df

fix: gsoc parallel upload

eafef39

chore: print out error message for integration test

2fd1e12

fix: parallel gsoc upload

8acb990

fix: nil reference on redundancy header

8ff9e53

fix: goroutines access loop variable

1f5011c

feat: feedfactory integration on bzz and feed

20671a4

test: feed api for red

e41cb55

test: bzz api

7199f6a

feat: soc validation check bytes except first

db800a8

test: add soc valid test

1706ccc

feat: flaky attempt to replicate address

031d6f5

feat: saturating by mirrorbits

415c66f

fix: flip bit after mirroredbits if necessary

d3f46a3

test: change test according to the new validation rules

342df84

test: getter

1aca908

Merge branch 'master' into feat/soc-dispersed

e6c6c95

fix: race issue

a97908f

docs: openapi changes

7d7b587

Merge branch 'master' into feat/soc-dispersed

4bc2c00

janos added 8 commits November 12, 2025 00:45

feat(combinator): add maxDepth argument to IterateAddressCombinations

0006ec1

feat(replicas): use IterateAddressCombinations for socGetter and socP…

1e3889c

…utter

feat: paralel attempts for socReplica.Get

1e54f3f

feat: add +1 replica in combinator

b48a44f

chore: update IterateAddressCombinations function comment

83e9ac5

fix: pass the replica chunk in socGetter.Get

1232754

feat(soc): more strict disperse replica validation

2fed6d6

chore: return ErrNotFound from socGetter.Get

fec7400

gacevicljubisa changed the base branch from feat/soc-dispersed to master November 12, 2025 13:54

gacevicljubisa changed the title ~~fix: soc dispersed replica~~ feat: soc dispersed replica v2 Nov 12, 2025

janos and others added 8 commits November 12, 2025 17:04

fix: adjust combinator not to return original address

1508169

fix: safe addresses from combinator

7aa70e4

fix(soc): correclty check the 5th bit flipping

5db0800

chore: adjust comments in tests

3b6452b

feat: new replicas soc getter tests

77fa1e3

chore(ci): set beekeeper branch to feat/soc-dispersed

b3c9781

chore: polish some code comments in replicas package

ee3f535

chore(replicas): remove Questions.md

78bca3c

gacevicljubisa commented Nov 13, 2025

View reviewed changes

janos added 4 commits November 13, 2025 11:45

chore: correclty set the copyright year

5ef6f8d

chore: add more tests to replicas and combinator

b83417c

chore(combinator): use term replication level instead depth for clarity

1afb16f

chore(api): remove debug error in json http response

89cd8af

janos requested a review from zelig November 14, 2025 11:11

janos reviewed Nov 14, 2025

View reviewed changes

zelig requested changes Nov 21, 2025

View reviewed changes

fix: address pr review comments

50acac1

janos reviewed Nov 21, 2025

View reviewed changes

janos requested a review from zelig November 21, 2025 16:07

feat: soc dispersed replica v2 #5279

Are you sure you want to change the base?

feat: soc dispersed replica v2 #5279

Conversation

gacevicljubisa commented Nov 6, 2025 • edited by janos Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Description

Original description

New changes

Open API Spec Version Changes (if applicable)

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janos commented Nov 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zelig left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janos left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

gacevicljubisa commented Nov 6, 2025 •

edited by janos

Loading