Skip to content

Conversation

@Vedin
Copy link

@Vedin Vedin commented Nov 13, 2025

The purpose of this PR is actually to show some new approach for spillable hash join. This works on TPCH but still has OOM on large dataset due to probe side isn't spilled correctly.
Actually, I want to split this PR into small next one based on #40.
Next I want to make this modification step-by-step on top of the initial GraceHashJoin:

  1. Add flag for enable_grace_hash_join. Add actual node and changes to join_selection.
  2. Add metrics for spillable mode.
  3. Changes partitioning and repartitioning. (This need a lot of rewrites to be readable)
  4. Add new scheduler.
  5. TODO: actually change how we spill on probe side. So we don't keep entire hashmap in memory.

This way we'd have more structured PR's. Queries should be fully working and testable after each step.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants