-
Notifications
You must be signed in to change notification settings - Fork 7
Open
Labels
enhancementNew feature or requestNew feature or request
Description
retry to implement Arrow record typed intermediate representation, once more! I think we can gradually switch to that by below steps:
-
prototyping for PoC: (various inputs) -> map's -> arrow -> map's -> json -> parquet
- implement Arrow -> JSON conversion in Go
- integrate it easily
-
remove parquet writing side Go intermediates: (various inputs) -> map's -> arrow -> json -> parquet
-
remove input side Go intermediates: (various inputs) -> arrow -> json -> parquet
- It requires input -> arrow formatter for each input types
-
ideal: (various inputs) -> arrow -> parquet
- It's so complicated because of arrow -> parquet (a part depends on
parquet-go) - It'll require some improvements of Arrow Go implementation.
- It's so complicated because of arrow -> parquet (a part depends on
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request