GH-534: Clarify versioning and V2 #535

emkornfield · 2025-12-02T18:18:07Z

Rationale for this change

Questions still arise on the state of V2, this tries to clarify my current understanding.

All encodings can be used any place
Version in the footer is meaningless and readers should determine readability by features.

What changes are included in this PR?

Spec clarification. Will start a discussion on the mailing list for word-smithing and consensus

Do these changes have PoC implementations?

No

Closes Clarify state of versioning and V2 #534
related mailing list thread https://lists.apache.org/thread/0bdyyb7qobrxx94x8v7t5z7g2ksnpyr2
vote thread on mailing list: https://lists.apache.org/thread/1nf596dq7fpc0zp35bt17zy7srvo5wt8

danielcweeks · 2025-12-02T18:47:56Z

src/main/thrift/parquet.thrift

+ *
+ * N.B. this page header is not necessarily strictly better then DataPageHeader.
+ * Page indexes already require that rows are aligned on page boundaries, and compressing
+ * repetition and definition levels can still be effective in some cases.


Are you saying this is deprecated? Why do we need this comment? It's not clear what you're trying to achieve here. (Nit: prefer not to use abbreviations like N.B.)

Rephrased, no I'm wasn't just saying it is not necessarily a strict improvement.

danielcweeks · 2025-12-02T18:54:12Z

src/main/thrift/parquet.thrift

+  /** Version of this file 
+    * 
+    * Deprecated.  Readers should determine if they support reading based on
+    * specific metadata (e.g. encoding enum) rather then relying on this field
+    * to make this determination.
+    */


I disagree with this. I don't think we should abandon versioning, but rather be more explicit about breaking changes and what is included with version update. Regardless, this needs more discussion with the community and a clear path forward for how we support breaking changes.

I rephrased this to explain rationale. I agree it needs discussion in the community but I think based on previous conversations this is probably the consensus path.

Rather than potentially (implicitly) changing the spec, I suggest we add some langague to help people understand how to interpret this field and the tradeoffs between using different versions.

For example, maybe we could say something like

As of December 2025, there is no agreed upon set of features that constitute version 2, so for maximum compatibility, writers should populate "1" for version and accept "1" and "2" interchangeably.

rephrased along these lines.

danielcweeks · 2025-12-02T19:24:43Z

Encodings.md

+Some Parquet implementations distinguish encodings as "v1" and "v2". From
+a specification perspective this distinction is considered meaningless. Writers may use any 
+encoding with both data page v1 and data page v2. Readers should lazily evaluate if they can 
+read a file (e.g. only error when required to a read a page with an unknown encoding).
+


I feel like we're redefining what version means to be scoped only to encodings and then saying that it's not necessary. It seems like we want to either separate encodings from versioning (e.g. any encoding that is understood by a client should be considered supported regardless of when it was introduced) or be more explicit about associating new encodings with a version (along with other possible breaking structural/representational changes).

I rephrased to remove version entirely as it might cause confusion.

danielcweeks · 2025-12-04T18:10:56Z

After thinking this through a little more, I think we should more clearly define what each "versioned identifier" means and clearly articulate under what conditions it would change. For example:

Magic Number `PAR1`:

What it means: Indicates that footer is still thrift compatible with Parquet V1 format and is expected to be parseable by any existing Parquet V1 client.
When it changes: the footer is changed in an incompatible way with the thrift definition or the footer is substantively changed in a way that older clients should not even attempt to read the file.
What it solves: helps determine whether a file/footer is parquet/corrupt/other or substantively different between versions.
Example: replacing the footer with FlatBuffers representation in an incompatible way.

Footer Version Number

What does it mean: largely redundant with PAR1
When it changes: (same as magic number)
What it solves: if the footer is stored somewhere outside of the file (e.g. a cache or supplied via some other mechanism)
Example: the footer is stored in a high-performance cache and keyed by path for faster pruning. No magic number is available for the reader.

What changes have been made without version updates:

[backward incompatible] New compression codecs: brotli, zstandard, etc.
[backward incompatible] New data types: variant, geo types, etc.
[backward compatible] Addition of Page indexes
[backward compatible] Updates to Column Stats

Given the incompatible changes like addition of codecs without a version change, it's confusing as to why the addition of encodings would require a version change. I do think there are valid reasons to update the version, but we just haven't had an example of it yet (or we've been hesitant to leverage the version because of the impact on readers).

emkornfield

change wording for version.

src/main/thrift/parquet.thrift

alamb

Looks good to me. THanks @emkornfield

I do think we should make sure that @danielcweeks 's concerns about version number are addressed before merging this

alamb · 2025-12-06T00:36:53Z

src/main/thrift/parquet.thrift

+ * in some scenarios). Page indexes require pages start and end at row boundaries regardless of which
+ * page header is used.
+ *
+ * As of December 2025, most known Parquet readers can read pages using this header. 


Perhaps we could refer readers to https://parquet.apache.org/docs/file-format/implementationstatus/ for the most up to date information

Maybe we can also change the content of

New page format allowing reading levels without decompressing the data

To another word to make it clearer that there is no agreed upon expectation that all writers will eventually use this new header. Perhaps:

Alternate page format allowing reading levels without decompressing the data

rehrased along these lines.

alamb · 2025-12-06T00:39:57Z

src/main/thrift/parquet.thrift

+  /** Version of this file 
+    * 
+    * Deprecated.  Readers should determine if they support reading based on
+    * specific metadata (e.g. encoding enum) rather then relying on this field
+    * to make this determination.
+    */


Rather than potentially (implicitly) changing the spec, I suggest we add some langague to help people understand how to interpret this field and the tradeoffs between using different versions.

For example, maybe we could say something like

As of December 2025, there is no agreed upon set of features that constitute version 2, so for maximum compatibility, writers should populate "1" for version and accept "1" and "2" interchangeably.

emkornfield · 2025-12-08T07:43:09Z

I do think we should make sure that @danielcweeks 's concerns about version number are addressed before merging this

I discussed the changes with @danielcweeks offline and I think he is OK with the current state. @danielcweeks please let me know if I misunderstood.

emkornfield · 2025-12-08T07:48:45Z

@alamb thanks for the review, I believe I addressed your concerns.

alkis

Thank you for putting this together @emkornfield!

src/main/thrift/parquet.thrift

alamb

Thanks again @emkornfield and @danielcweeks

src/main/thrift/parquet.thrift

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

Encodings.md

src/main/thrift/parquet.thrift

Co-authored-by: Fokko Driesprong <fokko@apache.org>

pitrou · 2025-12-11T09:51:05Z

src/main/thrift/parquet.thrift

+ * https://parquet.apache.org/docs/file-format/implementationstatus/ tracks the implementation of readers and 
+ * writers that support this page format. 


It does not seem useful to provide this link here as it would apply to most other Parquet features as well.

Removed. @alamb suggested adding it but I agree we generally don't reference it.

pitrou

LGTM for the most part, just one suggestion

Encodings.md

pitrou · 2025-12-18T16:29:32Z

src/main/thrift/parquet.thrift


 /**
- * New page format allowing reading levels without decompressing the data
+ * Alternate page format allowing reading levels without decompressing the data


I would perhaps keep "New" or "More recent".

I think Alternate probably makes the most sense here. It is not exactly "new" any more.

How about "Alternative page format, introduced to the specification after DataPageHeader, ...."

the naming of "V2" somewhat implies that it came after DataPageHeader

What I think is confusing about the word "new" is that (to me) it implies an expectation that this header will eventually replace the existing one, which I don't think there is consensus on

the naming of "V2" somewhat implies that it came after DataPageHeader

Right, IIUC the concern from Antoine is to maybe note that this wasn't part of the original spec?

src/main/thrift/parquet.thrift

pitrou

+1 in general, two small suggestions

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

alamb · 2025-12-19T01:04:47Z

I also added a links to the related mailing list threads to the description on this PR:

related mailing list thread https://lists.apache.org/thread/0bdyyb7qobrxx94x8v7t5z7g2ksnpyr2
vote thread on mailing list: https://lists.apache.org/thread/1nf596dq7fpc0zp35bt17zy7srvo5wt8

emkornfield · 2025-12-19T16:36:46Z

Vote passed. Going to merge. @pitrou @alamb we can maybe take another pass at the V2 data page header in another PR if there are serious concerns here.

Clarify versioning

9e705aa

emkornfield marked this pull request as draft December 2, 2025 18:18

danielcweeks reviewed Dec 2, 2025

View reviewed changes

emkornfield added 3 commits December 4, 2025 09:15

rephrase deprecation of version

cb1e64b

update page documentation.

56f3da1

reword encoding.

98be3c5

emkornfield commented Dec 4, 2025

View reviewed changes

emkornfield added 2 commits December 4, 2025 13:48

rephrase version

2dd4942

some more small comments.

aaa4c88

emkornfield commented Dec 5, 2025

View reviewed changes

src/main/thrift/parquet.thrift Outdated Show resolved Hide resolved

Update src/main/thrift/parquet.thrift

582f0ec

emkornfield commented Dec 5, 2025

View reviewed changes

src/main/thrift/parquet.thrift Outdated Show resolved Hide resolved

Update src/main/thrift/parquet.thrift

0dba068

emkornfield marked this pull request as ready for review December 5, 2025 22:50

alamb approved these changes Dec 6, 2025

View reviewed changes

address feedback.

e91f260

alkis approved these changes Dec 8, 2025

View reviewed changes

jorisvandenbossche reviewed Dec 8, 2025

View reviewed changes

src/main/thrift/parquet.thrift Outdated Show resolved Hide resolved

alamb approved these changes Dec 8, 2025

View reviewed changes

src/main/thrift/parquet.thrift Outdated Show resolved Hide resolved

Apply suggestions from code review

f01c0c6

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

Fokko reviewed Dec 9, 2025

View reviewed changes

Encodings.md Outdated Show resolved Hide resolved

Fokko reviewed Dec 9, 2025

View reviewed changes

src/main/thrift/parquet.thrift Outdated Show resolved Hide resolved

Apply suggestions from code review

f2aca5f

Co-authored-by: Fokko Driesprong <fokko@apache.org>

pitrou reviewed Dec 11, 2025

View reviewed changes

wgtmac reviewed Dec 11, 2025

View reviewed changes

Encodings.md Outdated Show resolved Hide resolved

address comments

58d585e

wgtmac approved these changes Dec 18, 2025

View reviewed changes

pitrou reviewed Dec 18, 2025

View reviewed changes

src/main/thrift/parquet.thrift Outdated Show resolved Hide resolved

pitrou approved these changes Dec 18, 2025

View reviewed changes

Update src/main/thrift/parquet.thrift

e86d86e

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

emkornfield merged commit 4b1c72c into apache:master Dec 19, 2025
4 checks passed

		* https://parquet.apache.org/docs/file-format/implementationstatus/ tracks the implementation of readers and
		* writers that support this page format.

GH-534: Clarify versioning and V2 #535

GH-534: Clarify versioning and V2 #535

Uh oh!

Conversation

emkornfield commented Dec 2, 2025 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danielcweeks commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Magic Number PAR1:

Footer Version Number

What changes have been made without version updates:

Uh oh!

emkornfield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield commented Dec 8, 2025

Uh oh!

emkornfield commented Dec 8, 2025

Uh oh!

alkis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pitrou left a comment

emkornfield commented Dec 2, 2025 •

edited by alamb

Loading

danielcweeks commented Dec 4, 2025 •

edited

Loading

Magic Number `PAR1`:

emkornfield commented Dec 19, 2025 •

edited

Loading