Skip to content

Conversation

@pedroigorjs
Copy link
Contributor

Context

This PR fixes an issue where datetime nodes would fail when handling dates that fall into nonexistent or ambiguous time periods during daylight saving time (DST) transitions. When clocks spring forward or fall back, certain times either don't exist or are ambiguous, causing errors in pandas timezone operations.

Problem

When localizing or converting datetime values in timezones with DST, the tz_localize() method would raise errors for:

  • Nonexistent times: Times that don't exist during spring forward transitions (e.g., 2:30 AM when clocks jump from 2:00 AM to 3:00 AM)
  • Ambiguous times: Times that occur twice during fall back transitions (e.g., 2:30 AM when clocks fall back from 3:00 AM to 2:00 AM)

Solution

Added proper handling for nonexistent and ambiguous times by configuring the tz_localize() method with appropriate parameters across all affected nodes:

  • nonexistent="shift_forward": Automatically shifts nonexistent times to the next valid time
  • ambiguous="NaT": Returns NaT (Not a Time) for ambiguous times, or handles them appropriately per node context

Changes Made

Modified Nodes

1. ToISOFormat

Updated the convert_to_iso internal function to handle DST edge cases when converting string dates to ISO format.

2. DifferenceBetweenDates

Updated the replace_invalid internal function to handle DST edge cases when localizing naive timestamps for date difference calculations.

@pedroigorjs pedroigorjs self-assigned this Dec 1, 2025
@pedroigorjs pedroigorjs added the bug Something isn't working label Dec 1, 2025
@github-actions
Copy link

github-actions bot commented Dec 1, 2025

coverage

Coverage Report
FileStmtsMissCoverMissing
retrack/engine
   base.py50492%35, 97, 104, 107
   executor.py1221092%62, 74–75, 144, 208, 252, 267, 271, 283, 291
   request_manager.py67790%17, 36, 55, 70–71, 154, 157
   rule.py50394%87, 109, 116
retrack/nodes
   check.py41393%23, 26, 89
   constants.py92397%122, 125, 170
   math.py43198%75
retrack/nodes/dynamic
   base.py16194%26
   csv_table.py43295%59, 62
   flow.py44198%18
retrack/utils
   component_registry.py1051883%77–78, 81–82, 85–86, 91–92, 101, 112–123
   exceptions.py28389%106–127
   graph.py60788%35, 46, 59, 61, 63, 84, 86
   registry.py34682%23–26, 38, 43, 51
retrack/validators
   base.py4175%14
   node_exists.py15287%36, 38
   node_validator.py33391%31, 56–57
TOTAL14027595% 

Tests Skipped Failures Errors Time
90 0 💤 0 ❌ 0 🔥 3.997s ⏱️

@github-actions
Copy link

github-actions bot commented Dec 1, 2025

coverage

Coverage Report
FileStmtsMissCoverMissing
retrack/engine
   base.py50492%35, 97, 104, 107
   executor.py1221092%62, 74–75, 144, 208, 252, 267, 271, 283, 291
   request_manager.py67790%17, 36, 55, 70–71, 154, 157
   rule.py50394%87, 109, 116
retrack/nodes
   check.py41393%23, 26, 89
   constants.py92397%122, 125, 170
   math.py43198%75
retrack/nodes/dynamic
   base.py16194%26
   csv_table.py43295%59, 62
   flow.py44198%18
retrack/utils
   component_registry.py1051883%77–78, 81–82, 85–86, 91–92, 101, 112–123
   exceptions.py28582%17–18, 106–127
   graph.py60788%35, 46, 59, 61, 63, 84, 86
   registry.py34682%23–26, 38, 43, 51
retrack/validators
   base.py4175%14
   node_exists.py15287%36, 38
   node_validator.py33391%31, 56–57
TOTAL14027795% 

Tests Skipped Failures Errors Time
90 0 💤 0 ❌ 0 🔥 5.333s ⏱️

@github-actions
Copy link

github-actions bot commented Dec 1, 2025

coverage

Coverage Report
FileStmtsMissCoverMissing
retrack/engine
   base.py50492%35, 97, 104, 107
   executor.py1221092%62, 74–75, 144, 208, 252, 267, 271, 283, 291
   request_manager.py67790%17, 36, 55, 70–71, 154, 157
   rule.py50394%87, 109, 116
retrack/nodes
   check.py41393%23, 26, 89
   constants.py92397%122, 125, 170
   math.py43198%75
retrack/nodes/dynamic
   base.py16194%26
   csv_table.py43295%59, 62
   flow.py44198%18
retrack/utils
   component_registry.py1051883%77–78, 81–82, 85–86, 91–92, 101, 112–123
   exceptions.py28389%106–127
   graph.py60788%35, 46, 59, 61, 63, 84, 86
   registry.py34682%23–26, 38, 43, 51
retrack/validators
   base.py4175%14
   node_exists.py15287%36, 38
   node_validator.py33391%31, 56–57
TOTAL14027595% 

Tests Skipped Failures Errors Time
90 0 💤 0 ❌ 0 🔥 3.887s ⏱️

@github-actions
Copy link

github-actions bot commented Dec 1, 2025

coverage

Coverage Report
FileStmtsMissCoverMissing
retrack/engine
   base.py50492%35, 97, 104, 107
   executor.py1221092%62, 74–75, 144, 208, 252, 267, 271, 283, 291
   request_manager.py67790%17, 36, 55, 70–71, 154, 157
   rule.py50394%87, 109, 116
retrack/nodes
   check.py41393%23, 26, 89
   constants.py92397%122, 125, 170
   math.py43198%75
retrack/nodes/dynamic
   base.py16194%26
   csv_table.py43295%59, 62
   flow.py44198%18
retrack/utils
   component_registry.py1051883%77–78, 81–82, 85–86, 91–92, 101, 112–123
   exceptions.py28582%17–18, 106–127
   graph.py60788%35, 46, 59, 61, 63, 84, 86
   registry.py34682%23–26, 38, 43, 51
retrack/validators
   base.py4175%14
   node_exists.py15287%36, 38
   node_validator.py33391%31, 56–57
TOTAL14027795% 

Tests Skipped Failures Errors Time
90 0 💤 0 ❌ 0 🔥 5.274s ⏱️

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

coverage

Coverage Report
FileStmtsMissCoverMissing
retrack/engine
   base.py50492%35, 97, 104, 107
   executor.py1221092%62, 74–75, 144, 208, 252, 267, 271, 283, 291
   request_manager.py67790%17, 36, 55, 70–71, 154, 157
   rule.py50394%87, 109, 116
retrack/nodes
   check.py41393%23, 26, 89
   constants.py92397%122, 125, 170
   math.py43198%75
retrack/nodes/dynamic
   base.py16194%26
   csv_table.py43295%59, 62
   flow.py44198%18
retrack/utils
   component_registry.py1051883%77–78, 81–82, 85–86, 91–92, 101, 112–123
   exceptions.py28582%17–18, 106–127
   graph.py60788%35, 46, 59, 61, 63, 84, 86
   registry.py34682%23–26, 38, 43, 51
retrack/validators
   base.py4175%14
   node_exists.py15287%36, 38
   node_validator.py33391%31, 56–57
TOTAL14027795% 

Tests Skipped Failures Errors Time
90 0 💤 0 ❌ 0 🔥 5.431s ⏱️

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

coverage

Coverage Report
FileStmtsMissCoverMissing
retrack/engine
   base.py50492%35, 97, 104, 107
   executor.py1221092%62, 74–75, 144, 208, 252, 267, 271, 283, 291
   request_manager.py67790%17, 36, 55, 70–71, 154, 157
   rule.py50394%87, 109, 116
retrack/nodes
   check.py41393%23, 26, 89
   constants.py92397%122, 125, 170
   math.py43198%75
retrack/nodes/dynamic
   base.py16194%26
   csv_table.py43295%59, 62
   flow.py44198%18
retrack/utils
   component_registry.py1051883%77–78, 81–82, 85–86, 91–92, 101, 112–123
   exceptions.py28389%106–127
   graph.py60788%35, 46, 59, 61, 63, 84, 86
   registry.py34682%23–26, 38, 43, 51
retrack/validators
   base.py4175%14
   node_exists.py15287%36, 38
   node_validator.py33391%31, 56–57
TOTAL14027595% 

Tests Skipped Failures Errors Time
90 0 💤 0 ❌ 0 🔥 4.119s ⏱️

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

coverage

Coverage Report
FileStmtsMissCoverMissing
retrack/engine
   base.py50492%35, 97, 104, 107
   executor.py1221092%62, 74–75, 144, 208, 252, 267, 271, 283, 291
   request_manager.py67790%17, 36, 55, 70–71, 154, 157
   rule.py50394%87, 109, 116
retrack/nodes
   check.py41393%23, 26, 89
   constants.py92397%122, 125, 170
   math.py43198%75
retrack/nodes/dynamic
   base.py16194%26
   csv_table.py43295%59, 62
   flow.py44198%18
retrack/utils
   component_registry.py1051883%77–78, 81–82, 85–86, 91–92, 101, 112–123
   exceptions.py28582%17–18, 106–127
   graph.py60788%35, 46, 59, 61, 63, 84, 86
   registry.py34682%23–26, 38, 43, 51
retrack/validators
   base.py4175%14
   node_exists.py15287%36, 38
   node_validator.py33391%31, 56–57
TOTAL14027795% 

Tests Skipped Failures Errors Time
90 0 💤 0 ❌ 0 🔥 4.877s ⏱️

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

coverage

Coverage Report
FileStmtsMissCoverMissing
retrack/engine
   base.py50492%35, 97, 104, 107
   executor.py1221092%62, 74–75, 144, 208, 252, 267, 271, 283, 291
   request_manager.py67790%17, 36, 55, 70–71, 154, 157
   rule.py50394%87, 109, 116
retrack/nodes
   check.py41393%23, 26, 89
   constants.py92397%122, 125, 170
   math.py43198%75
retrack/nodes/dynamic
   base.py16194%26
   csv_table.py43295%59, 62
   flow.py44198%18
retrack/utils
   component_registry.py1051883%77–78, 81–82, 85–86, 91–92, 101, 112–123
   exceptions.py28389%106–127
   graph.py60788%35, 46, 59, 61, 63, 84, 86
   registry.py34682%23–26, 38, 43, 51
retrack/validators
   base.py4175%14
   node_exists.py15287%36, 38
   node_validator.py33391%31, 56–57
TOTAL14027595% 

Tests Skipped Failures Errors Time
90 0 💤 0 ❌ 0 🔥 3.920s ⏱️

@pedroigorjs pedroigorjs merged commit 3f4b0fe into main Dec 2, 2025
3 checks passed
@pedroigorjs pedroigorjs deleted the b/fix-non-existent-time-error branch December 2, 2025 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants