Skip to content

Conversation

@beepubapu
Copy link

What are you trying to accomplish?

I am changing Semian behavior on Redis OOM errors to not open the circuit in order to better allow Redis to recover (and since there is not much benefit here form circuit breaking anyways).

What approach did you choose and why?

In https://github.com/Shopify/job-platform/issues/1377, we identified that not opening the circuit is preferable here. The current behavior concern was anticipated early on in #172.

Per @Mangara :
| We've seen this multiple times, both from HedwigUI and from Hedwig itself, when it tries to unsafely dequeue to recover from OOM.

What should reviewers focus on?

  • Are there any concerns with every write attempt now hitting Redis? We think this is okay since it is a fast fail.

@beepubapu beepubapu added the ruby Pull requests that update Ruby code label Dec 18, 2025
@beepubapu
Copy link
Author

CI failures appear to be unrelated and a similar pattern can be seen on recent PRs. The Redis part of the matrix seems happy.

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR changes Redis OOM (Out Of Memory) error handling to prevent circuit breaking, allowing Redis to recover more effectively from memory pressure situations. The change recognizes that OOM errors are fast failures where circuit breaking provides no benefit and can actually hinder recovery by blocking read and dequeue operations.

  • Adds marks_semian_circuits? method override returning false to OutOfMemoryError classes in both redis and redis_client adapters
  • Updates test expectations to verify circuits remain closed after OOM errors exceed the error threshold
  • Includes clear documentation explaining the rationale: OOM errors are fast failures and blocking reads/dequeues prevents recovery

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
lib/semian/redis.rb Adds marks_semian_circuits? method to Redis::OutOfMemoryError class to prevent circuit opening for Redis v4
lib/semian/redis_client.rb Adds marks_semian_circuits? method to RedisClient::OutOfMemoryError using class_eval to prevent circuit opening
test/adapters/redis_test.rb Updates two OOM test cases to verify circuits do NOT open after exceeding error threshold, with clear explanatory comments

Note: The implementation correctly handles both Redis v4 (via lib/semian/redis.rb) and RedisClient (via lib/semian/redis_client.rb). However, for completeness, lib/semian/redis/v5.rb (not modified in this PR) should also be updated to add the same marks_semian_circuits? override for Redis::OutOfMemoryError to ensure consistent behavior when Redis v5+ translates RedisClient::OutOfMemoryError to Redis::OutOfMemoryError.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ruby Pull requests that update Ruby code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant