Published asset deletion protection design doc #2586

mvandenburgh · 2025-10-07T17:03:02Z

This PR proposes a design for #2501, and provides "instance protection"-like safety guards for published assets as requested in #2367 (comment).

yarikoptic

Initial comments/thoughts

doc/design/s3-object-lock-legal-holds.md

yarikoptic

minor questions/comments

doc/design/s3-object-lock-legal-holds.md

yarikoptic · 2025-11-04T19:46:09Z

doc/design/s3-object-lock-legal-holds.md

+
+1. Enable S3 Object Lock on the bucket
+2. Deploy code changes to apply legal holds during publish celery task
+3. Run a one-time script to apply legal holds to all existing published asset blobs


note: yet another opportunity for some kind of fsck style command to ensure consistency of the bucket with our assumptions.

and I thought about this again, and wondered if

Suggested change

3. Run a one-time script to apply legal holds to all existing published asset blobs

3. Run a one-time script to apply legal holds to all existing published asset blobs

4. Implement and deploy for weekly run an admin-level maintenance script to verify consistency of legal holds on S3 to the information about published assets in DB

since that would allow to actually check that all is good in staging, otherwise a benefit from deploying in staging is kinda limited (e.g. how would you know that it does work correctly in staging?)

doc/design/s3-object-lock-legal-holds.md

Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>

yarikoptic

Almost there -- some minor comments on various aspects. Some could potentially be ignored.

yarikoptic · 2025-11-25T16:32:32Z

doc/design/s3-object-lock-legal-holds.md

+
+The DANDI Archive's core value proposition is hosting reliable, citable scientific data. Once a dandiset is published and assigned a DOI, the data it references MUST remain immutable to preserve the integrity of scientific citations and reproducibility. Users who cite a published dandiset expect that the data will remain exactly as it was at the time of publication.
+
+Currently, while our application logic prevents modification of published assets, there is no infrastructure-level enforcement.  Adding S3-level immutability provides stronger data safety.


I feel it is jumping over the different levels (S3: blobs, zarr prefixes; and SQL DB: assets associated with those) without due description. May be smth like

Suggested change

Currently, while our application logic prevents modification of published assets, there is no infrastructure-level enforcement. Adding S3-level immutability provides stronger data safety.

Dandiset is a collection of assets which are associated many-to-one to content on S3 of two data types: blobs and zarrs.

A blob is a representation of a single key ("file") under `blobs/` prefix, while zarr is a prefix ("folder") under `zarr/` prefix).

`blobs/` store is already immutable, as existing keys MUST NOT mutate, while zarrs are allowed to be updated under `zarr/` but are associated with different S3 key `versionId`'s.

Only a single blob or a single zarr is associated with a single asset.

Whenever zarr assets remain mutable and thus cannot be published ATM, published blob assets MUST remain immutable and our application logic already prevents modification of published assets.

But there is no infrastructure-level enforcement to guarantee immutability of published blobs on S3. Adding S3-level immutability provides stronger data safety.

yarikoptic · 2025-11-25T16:37:22Z

doc/design/s3-object-lock-legal-holds.md

+
+## Proposed Solution
+
+AWS S3 Object Lock with legal holds provides exactly the protection we need. A legal hold is a flag that can be applied to an S3 object version to prevent deletion or modification, regardless of any retention policies or IAM permissions. Unlike retention modes, legal holds have no expiration date and remain in effect until explicitly removed via the `s3:PutObjectLegalHold` S3 API. See the [documentation for S3 legal holds](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html#object-lock-legal-holds) for more details.


Although those are indeed two neighboring subsections, new reader would not know of such proximities reference would be handy IMHO, especially since "retention mode" keeps coming up.

Suggested change

AWS S3 Object Lock with legal holds provides exactly the protection we need. A legal hold is a flag that can be applied to an S3 object version to prevent deletion or modification, regardless of any retention policies or IAM permissions. Unlike retention modes, legal holds have no expiration date and remain in effect until explicitly removed via the `s3:PutObjectLegalHold` S3 API. See the [documentation for S3 legal holds](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html#object-lock-legal-holds) for more details.

AWS S3 Object Lock with legal holds provides exactly the protection we need. A legal hold is a flag that can be applied to an S3 object version to prevent deletion or modification, regardless of any retention policies or IAM permissions. Unlike [retention modes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html#object-lock-retention-modes), legal holds have no expiration date and remain in effect until explicitly removed via the `s3:PutObjectLegalHold` S3 API. See the [documentation for S3 legal holds](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html#object-lock-legal-holds) for more details.

yarikoptic · 2025-11-25T16:39:28Z

doc/design/s3-object-lock-legal-holds.md

+- Only published assets need protection from deletion, other objects in the bucket MUST remain deletable
+- Published assets MUST remain immutable permanently (no expiration)
+- There MUST be the ability to remove the hold if absolutely necessary (per the "backdoor" requirement)
+- Legal holds are independent of retention policies and work alongside bucket versioning


Suggested change

- Legal holds are independent of retention policies and work alongside bucket versioning

- Legal holds are independent of retention policies and work alongside bucket versioning

- Legal holds apply to an object version thus MAY later be adopted for protection of published zarr assets

yarikoptic · 2025-11-25T16:40:00Z

doc/design/s3-object-lock-legal-holds.md

+2. **Compliance mode**: No user can delete the object version until the retention period expires, including the root account
+3. **Legal hold**: A separate flag independent of retention mode that prevents deletion until removed
+
+Goverence and compliance modes operate at the bucket level, while legal holds operate at the object level.


Suggested change

Goverence and compliance modes operate at the bucket level, while legal holds operate at the object level.

Goverence and compliance modes operate at the bucket level, while legal holds operate at the object version level.

IMHO an important feature (if I got it right) since will allow later to protect Zarr versions. (added below as an item)

yarikoptic · 2025-11-25T16:42:07Z

doc/design/s3-object-lock-legal-holds.md

+
+## Interaction with Garbage Collection
+
+Legal holds MUST NOT interfere with garbage collection of unpublished assets, as only published asset blobs SHALL have legal holds applied. The garbage collection processes for orphaned uploads, unreferenced AssetBlobs, and unpublished assets MUST continue to work as designed.


Suggested change

Legal holds MUST NOT interfere with garbage collection of unpublished assets, as only published asset blobs SHALL have legal holds applied. The garbage collection processes for orphaned uploads, unreferenced AssetBlobs, and unpublished assets MUST continue to work as designed.

Legal holds MUST NOT interfere with garbage collection of unpublished assets, as only published asset blobs SHALL have legal holds applied. The garbage collection processes (e.g., of orphaned uploads, unreferenced AssetBlobs, and unpublished assets) MUST continue to work as designed.

since this list might be complete currently but could change later (zarrs GC etc)

yarikoptic · 2025-11-25T16:44:23Z

doc/design/s3-object-lock-legal-holds.md

+
+### Potential Interference Scenarios (and Why They Don't Occur)
+
+The following scenarios could theoretically interfere with garbage collection, but do NOT occur with this design:


Suggested change

The following scenarios could theoretically interfere with garbage collection, but do NOT occur with this design:

The following scenarios MAY theoretically interfere with garbage collection, but SHOULD NOT occur with this design:

or should even be made

Suggested change

The following scenarios could theoretically interfere with garbage collection, but do NOT occur with this design:

The following scenarios MAY theoretically interfere with garbage collection, but MUST NOT occur with this design:

thus warranting ERROR reporting?

yarikoptic · 2025-11-25T16:44:47Z

doc/design/s3-object-lock-legal-holds.md

+
+The following scenarios could theoretically interfere with garbage collection, but do NOT occur with this design:
+
+1. **Overly broad legal holds**: If legal holds were accidentally applied to unpublished blobs, those blobs could not be deleted by garbage collection processes until the holds were manually removed.


Suggested change

1. **Overly broad legal holds**: If legal holds were accidentally applied to unpublished blobs, those blobs could not be deleted by garbage collection processes until the holds were manually removed.

1. **Overly broad legal holds**: If legal holds were accidentally applied to unpublished blobs, those blobs MUST NOT be deleted by garbage collection processes until the holds were manually removed.

but I am thinking about deleting a published dandiset overall (e.g. per author's or regulatory organ instruction). We would need to implement a process to review if each blob has any other "publication" or even just a "draft" version and potentially be able to decide on blob-by-blob case on what to do.

I think it is worth adding some statement here that we are not aiming to design/implement that logic here but that it MAY be designed/implemented later

Suggested change

1. **Overly broad legal holds**: If legal holds were accidentally applied to unpublished blobs, those blobs could not be deleted by garbage collection processes until the holds were manually removed.

1. **Overly broad legal holds**: If legal holds were accidentally applied to unpublished blobs, those blobs could not be deleted by garbage collection processes until the holds were manually removed.

- Under force-major circumstances requiring deletion of a published dandiset, and thus its assets (all or some), legal holds might remain attached to assets associated with published or not blobs. Analysis and design of handling of such cases MAY be implemented later, but overall the mechanism of legal holds seems to be appropriate and will not become an obstacle.

yarikoptic · 2025-11-25T16:54:45Z

doc/design/s3-object-lock-legal-holds.md

+
+**Why legal holds avoid these issues:**
+- Unpublished assets never receive legal holds, so they remain fully deletable while the deletion permissions remain intact for object versions without legal holds
+- Legal holds are applied at the **object level**, only to specific published asset blobs. i.e., they are **object-level retention policies** instead of **bucket-wide retention policies**.


Suggested change

- Legal holds are applied at the **object level**, only to specific published asset blobs. i.e., they are **object-level retention policies** instead of **bucket-wide retention policies**.

- Legal holds are applied at the **object version level**, only to specific published asset blobs. i.e., they are **object-level retention policies** instead of **bucket-wide retention policies**.

yarikoptic · 2025-11-25T16:58:15Z

doc/design/s3-object-lock-legal-holds.md

+
+1. Enable S3 Object Lock on the bucket
+2. Deploy code changes to apply legal holds during publish celery task
+3. Run a one-time script to apply legal holds to all existing published asset blobs


and I thought about this again, and wondered if

Suggested change

3. Run a one-time script to apply legal holds to all existing published asset blobs

3. Run a one-time script to apply legal holds to all existing published asset blobs

4. Implement and deploy for weekly run an admin-level maintenance script to verify consistency of legal holds on S3 to the information about published assets in DB

since that would allow to actually check that all is good in staging, otherwise a benefit from deploying in staging is kinda limited (e.g. how would you know that it does work correctly in staging?)

Design doc for s3 legal hold for published objects

ed7defc

mvandenburgh mentioned this pull request Oct 7, 2025

Add Asset garbage collection design doc #2367

Open

mvandenburgh marked this pull request as ready for review October 7, 2025 17:06

mvandenburgh requested review from jjnesbitt, waxlamp and yarikoptic October 7, 2025 17:22

mvandenburgh added the documentation Changes only affect the documentation label Oct 7, 2025

mvandenburgh self-assigned this Oct 7, 2025

yarikoptic reviewed Oct 7, 2025

View reviewed changes

mvandenburgh requested a review from satra October 7, 2025 17:38

mvandenburgh added 3 commits October 14, 2025 10:55

Mention S3 API for removing legal holds

9cc5808

Add link to legal hold docs

77f3850

Add sentence about doing sandbox rollout first

7264248

mvandenburgh force-pushed the published-object-lock branch from 2e49a28 to 7264248 Compare October 14, 2025 14:58

mvandenburgh added 3 commits October 16, 2025 17:47

Adjust language to follow RFC 2119

a90dcc6

Add 'potential interference scenarios' section

40be884

More exact language - "object version" vs "object"

b593dd9

mvandenburgh requested a review from yarikoptic October 16, 2025 19:06

waxlamp added the design-doc Involves creating or discussing a design document label Nov 3, 2025

yarikoptic reviewed Nov 4, 2025

View reviewed changes

waxlamp and others added 2 commits November 25, 2025 10:28

Clarify expectations around attempted deletes of locked objects

7a4ce10

Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>

Soften imperative for non-functional requirement

6d32ef6

Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>

yarikoptic requested changes Nov 25, 2025

View reviewed changes

	3. Run a one-time script to apply legal holds to all existing published asset blobs
	3. Run a one-time script to apply legal holds to all existing published asset blobs
	4. Implement and deploy for weekly run an admin-level maintenance script to verify consistency of legal holds on S3 to the information about published assets in DB


		The DANDI Archive's core value proposition is hosting reliable, citable scientific data. Once a dandiset is published and assigned a DOI, the data it references MUST remain immutable to preserve the integrity of scientific citations and reproducibility. Users who cite a published dandiset expect that the data will remain exactly as it was at the time of publication.

		Currently, while our application logic prevents modification of published assets, there is no infrastructure-level enforcement. Adding S3-level immutability provides stronger data safety.

-Currently, while our application logic prevents modification of published assets, there is no infrastructure-level enforcement.  Adding S3-level immutability provides stronger data safety.
+Dandiset is a collection of assets which are associated many-to-one to content on S3 of two data types: blobs and zarrs.
+A blob is a representation of a single key ("file") under `blobs/` prefix, while zarr is a prefix ("folder") under `zarr/` prefix).
+`blobs/` store is already immutable, as existing keys MUST NOT mutate, while zarrs are allowed to be updated under `zarr/` but are associated with different S3 key `versionId`'s.
+Only a single blob or a single zarr is associated with a single asset.
+Whenever zarr assets remain mutable and thus cannot be published ATM, published blob assets MUST remain immutable and our application logic already prevents modification of published assets.
+But there is no infrastructure-level enforcement to guarantee immutability of published blobs on S3.  Adding S3-level immutability provides stronger data safety.


		## Proposed Solution

		AWS S3 Object Lock with legal holds provides exactly the protection we need. A legal hold is a flag that can be applied to an S3 object version to prevent deletion or modification, regardless of any retention policies or IAM permissions. Unlike retention modes, legal holds have no expiration date and remain in effect until explicitly removed via the `s3:PutObjectLegalHold` S3 API. See the [documentation for S3 legal holds](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html#object-lock-legal-holds) for more details.

	- Legal holds are independent of retention policies and work alongside bucket versioning
	- Legal holds are independent of retention policies and work alongside bucket versioning
	- Legal holds apply to an object version thus MAY later be adopted for protection of published zarr assets

	Goverence and compliance modes operate at the bucket level, while legal holds operate at the object level.
	Goverence and compliance modes operate at the bucket level, while legal holds operate at the object version level.


		## Interaction with Garbage Collection

		Legal holds MUST NOT interfere with garbage collection of unpublished assets, as only published asset blobs SHALL have legal holds applied. The garbage collection processes for orphaned uploads, unreferenced AssetBlobs, and unpublished assets MUST continue to work as designed.


		### Potential Interference Scenarios (and Why They Don't Occur)

		The following scenarios could theoretically interfere with garbage collection, but do NOT occur with this design:

	The following scenarios could theoretically interfere with garbage collection, but do NOT occur with this design:
	The following scenarios MAY theoretically interfere with garbage collection, but SHOULD NOT occur with this design:


		The following scenarios could theoretically interfere with garbage collection, but do NOT occur with this design:

		1. Overly broad legal holds: If legal holds were accidentally applied to unpublished blobs, those blobs could not be deleted by garbage collection processes until the holds were manually removed.

	1. Overly broad legal holds: If legal holds were accidentally applied to unpublished blobs, those blobs could not be deleted by garbage collection processes until the holds were manually removed.
	1. Overly broad legal holds: If legal holds were accidentally applied to unpublished blobs, those blobs could not be deleted by garbage collection processes until the holds were manually removed.
	- Under force-major circumstances requiring deletion of a published dandiset, and thus its assets (all or some), legal holds might remain attached to assets associated with published or not blobs. Analysis and design of handling of such cases MAY be implemented later, but overall the mechanism of legal holds seems to be appropriate and will not become an obstacle.

	- Legal holds are applied at the object level, only to specific published asset blobs. i.e., they are object-level retention policies instead of bucket-wide retention policies.
	- Legal holds are applied at the object version level, only to specific published asset blobs. i.e., they are object-level retention policies instead of bucket-wide retention policies.

Published asset deletion protection design doc #2586

Are you sure you want to change the base?

Published asset deletion protection design doc #2586

Uh oh!

Conversation

mvandenburgh commented Oct 7, 2025

Uh oh!

yarikoptic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yarikoptic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yarikoptic left a comment

Choose a reason for hiding this comment

Uh oh!

yarikoptic Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yarikoptic Nov 25, 2025 •

edited

Loading