Add support for opening structured dtypes as void for `zarr` driver #272

BrianMichell · 2026-01-05T19:02:10Z

Supersedes #264

Removes bool open_as_void = false default in C++ function declarations and definitions.

Uses a derived DataCache implementation similar to strategy used in neuroglancer_precomputed driver for UnshardedDataCache and ShardedDataCache

laramiel · 2026-01-06T02:14:10Z

Ok, I imported this locally and am looking at a few things.

Please make sure that all the tests build and pass. By removing the open_as_void = true, the spec tests fail to build.

tensorstore/driver/zarr/driver.cc

…ussion_r2663351949` and extend test coverage

…ussion_r2663353574` and extend test coverage

BrianMichell · 2026-01-06T15:42:55Z

Ok, I imported this locally and am looking at a few things.

Please make sure that all the tests build and pass. By removing the open_as_void = true, the spec tests fail to build.

My bad, I don't know how I missed the compilation issue for the spec test. I've fixed the issue and resolved your comments.

Thanks for taking a look and getting back to this so fast!

tensorstore/driver/zarr/schema.yml

tensorstore/driver/zarr/driver.cc

…ges/BASE..a42b6f511375dc1bd402ad525519f57210546735#r2666111665` Enforce schema validation for one-of `field` and `open_as_void`

…ussion_r2666195572` Use synthesized open_as_void field

tensorstore/driver/zarr/driver.cc

tensorstore/driver/zarr/spec.cc

tensorstore/driver/zarr/driver.cc

tensorstore/driver/zarr/schema.yml

…ges/BASE..7fb91d751f24a1dd479fd389378f9e6ad33c8112#r2669500148` Make the schema valid

…223557` Compose ZarrMetadata with `open_as_void` field

…205910` Add error check for creating a structured Store but not providing the dtype

…216841` Simplify field index logic for `open_as_void` case

BrianMichell · 2026-01-12T19:54:00Z

Sorry for the delay responding to the latest round of feedback. I believe everything raised so far has been addressed.

laramiel · 2026-01-12T20:06:51Z

tensorstore/driver/zarr/spec.h

 /// \param partial_metadata Constraints in the form of partial zarr metadata.
 /// \param selected_field The field to which `schema` applies.
 /// \param schema Schema constraints for the `selected_field`.
+/// \param open_as_void If true, opens the array as raw bytes.


selected_field and open_as_void are mutually exclusive in GetNewMetadata.
At least document that & add an error return/test case.

Also for GetSpecRankAndFieldInfo

laramiel · 2026-01-12T20:22:44Z

tensorstore/driver/zarr/driver.cc

-    fill_value = (*metadata.fill_value)[field_index];
+    // For void access, synthesize a byte-level fill value
+    if (open_as_void) {
+      const Index nbytes = metadata.dtype->bytes_per_outer_element;


I would expect this to get the fill_value from the metadata. More like:

size_t field_index = 0; // open_as_void has a single field.
if (!open_as_void) {
TENSORSTORE_ASSIGN_OR_RETURN(
field_index,
GetFieldIndex(*metadata.dtype, selected_field));
}
fill_value = (*metadata.fill_value)[field_index];

That might require CreateVoidMetadata to set a proper fill value?

Yes CreateVoidMetadata should set a proper fill value.

laramiel · 2026-01-12T20:28:03Z

tensorstore/driver/zarr/driver.cc

 #include "absl/status/status.h"
 #include "absl/strings/cord.h"
 #include <nlohmann/json_fwd.hpp>
+#include "riegeli/bytes/cord_reader.h"


missing BUILD deps for these.

laramiel · 2026-01-12T20:32:59Z

tensorstore/driver/zarr/dtype.h

 /// \file
 /// Support for encoding/decoding zarr "dtype" specifications.
 /// See: https://zarr.readthedocs.io/en/stable/spec/v2.html



This file is missing some includes (*it already was, but now it's missing more).

#include <optional> #include <string> #include <string_view> #include <vector> #include "absl/status/status.h" #include "tensorstore/index.h"

And the BUILD deps for the new non-std library includes as well.

laramiel · 2026-01-12T20:35:39Z

tensorstore/driver/zarr/dtype.h

  friend void to_json(::nlohmann::json& out,  // NOLINT
                      const ZarrDType& dtype);
+
+  /// Lazily-computed cache for GetVoidField().


Not sure we need this cached? Seems like just returning a Field is reasonable, as the CreateVoidMetadata is already going to copy it into vector, and it's not expensive to recreate when opening a new zarr volume.

See jbms@ below; he's in favor of keeping this cached but guarding it.

jbms · 2026-01-13T00:01:19Z

tensorstore/driver/zarr/driver.cc

+  // since we're treating the data as raw bytes regardless of the actual dtype.
+  // Shape is allowed to differ (handled by base class for resizing).
+  // Other fields like compressor, order, chunks must still match.
+  if (existing_metadata.dtype.bytes_per_outer_element !=


I think we could just rely on the normal validate but applied to the void metadata.

jbms · 2026-01-13T00:04:14Z

tensorstore/driver/zarr/driver.cc

+      // contain only the void field, allowing standard encode/decode to work.
+      // CreateVoidMetadata uses the same chunks and bytes_per_outer_element as
+      // the original validated metadata, so it should never fail.
+      initializer.metadata = CreateVoidMetadata(original_metadata).value();


Probably would be better to have the original metadata cache the pointer to the void metadata and initialize it on first access.

jbms · 2026-01-13T00:06:43Z

tensorstore/driver/zarr/driver.cc

+          field_index,
+          GetFieldIndex(metadata.dtype, spec().selected_field));
+    }
    TENSORSTORE_RETURN_IF_ERROR(


we need to make sure to validate the schema against the void metadata, not the regular metadata. Note: The partial_metadata validation is still against the regular metadata, though.

jbms · 2026-01-13T00:07:59Z

tensorstore/driver/zarr/metadata.cc

+
+  // Set fill_value for the single void field.
+  // Empty/null fill value is handled by GetChunkGridSpecification.
+  metadata->fill_value.resize(1);


Need to have a proper fill value here.

jbms · 2026-01-13T00:09:23Z

tensorstore/driver/zarr/dtype.cc

 }

+const ZarrDType::Field* ZarrDType::GetVoidField() const {
+  if (!void_field_cache_) {


This cache isn't thread safe which is a problem. However, instead of caching void_field instead could just cache the entire void metadata (and do it thread safely).

Actually looking more closely I see that it will be inconvenient to not have this cache, since we need a stable pointer to the void field. So you should keep the cache but just make it thread safe, e.g. using absl::call_once.

BrianMichell added 3 commits January 5, 2026 16:08

Add open_as_void option to zarr v2 driver (#6)

59a6871

Remove default open_as_void from definitions

2aedabf

Use derived DataCache for open_as_void

46d9902

BrianMichell mentioned this pull request Jan 5, 2026

Add support for structured dtypes to zarr3 driver, open zarr 2 and 3 structs as void #264

Closed

laramiel reviewed Jan 6, 2026

View reviewed changes

tensorstore/driver/zarr/driver.cc Outdated Show resolved Hide resolved

laramiel reviewed Jan 6, 2026

View reviewed changes

tensorstore/driver/zarr/driver.cc Show resolved Hide resolved

laramiel reviewed Jan 6, 2026

View reviewed changes

tensorstore/driver/zarr/driver.cc Outdated Show resolved Hide resolved

BrianMichell added 5 commits January 6, 2026 14:36

Fix compile issues for missing argument

ccc4bd7

Correct tests, add argument comment for open as void value

5d4a68f

Add test coverage for GetSpecInfo

c410f5e

Resolve feedback https://github.com/google/tensorstore/pull/272#disc…

d886c2f

…ussion_r2663351949` and extend test coverage

Resolve feedback https://github.com/google/tensorstore/pull/272#disc…

a42b6f5

…ussion_r2663353574` and extend test coverage

laramiel reviewed Jan 6, 2026

View reviewed changes

tensorstore/driver/zarr/schema.yml Show resolved Hide resolved

laramiel reviewed Jan 6, 2026

View reviewed changes

tensorstore/driver/zarr/driver.cc Outdated Show resolved Hide resolved

BrianMichell added 2 commits January 7, 2026 14:57

Resolve feedback https://github.com/google/tensorstore/pull/272/chan…

e9c15da

…ges/BASE..a42b6f511375dc1bd402ad525519f57210546735#r2666111665` Enforce schema validation for one-of `field` and `open_as_void`

Resolve feedback https://github.com/google/tensorstore/pull/272#disc…

7fb91d7

…ussion_r2666195572` Use synthesized open_as_void field

laramiel reviewed Jan 7, 2026

View reviewed changes

tensorstore/driver/zarr/driver.cc Show resolved Hide resolved

laramiel reviewed Jan 7, 2026

View reviewed changes

tensorstore/driver/zarr/spec.cc Show resolved Hide resolved

laramiel reviewed Jan 7, 2026

View reviewed changes

tensorstore/driver/zarr/spec.cc Outdated Show resolved Hide resolved

laramiel reviewed Jan 7, 2026

View reviewed changes

tensorstore/driver/zarr/driver.cc Show resolved Hide resolved

jbms reviewed Jan 7, 2026

View reviewed changes

tensorstore/driver/zarr/schema.yml Outdated Show resolved Hide resolved

BrianMichell added 4 commits January 7, 2026 21:00

Resolve feedback https://github.com/google/tensorstore/pull/272/chan…

389d6a9

…ges/BASE..7fb91d751f24a1dd479fd389378f9e6ad33c8112#r2669500148` Make the schema valid

Resolve https://github.com/google/tensorstore/pull/272/changes#r2669…

101011b

…223557` Compose ZarrMetadata with `open_as_void` field

Resolve https://github.com/google/tensorstore/pull/272/changes#r2669…

62fd8f9

…205910` Add error check for creating a structured Store but not providing the dtype

Resolve https://github.com/google/tensorstore/pull/272/changes#r2669…

9735318

…216841` Simplify field index logic for `open_as_void` case

laramiel reviewed Jan 12, 2026

View reviewed changes

jbms reviewed Jan 13, 2026

View reviewed changes

Add support for opening structured dtypes as void for zarr driver #272

Are you sure you want to change the base?

Add support for opening structured dtypes as void for zarr driver #272

Uh oh!

Conversation

BrianMichell commented Jan 5, 2026

Uh oh!

laramiel commented Jan 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BrianMichell commented Jan 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BrianMichell commented Jan 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laramiel Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laramiel Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add support for opening structured dtypes as void for `zarr` driver #272

Add support for opening structured dtypes as void for `zarr` driver #272

laramiel Jan 12, 2026 •

edited

Loading

laramiel Jan 12, 2026 •

edited

Loading