Skip to content

Conversation

@tonatoz
Copy link
Contributor

@tonatoz tonatoz commented Jul 16, 2025

Problem Solved

Previously, WebSocket subscriptions in Dshackle lacked proper cleanup mechanisms, leading to potential memory leaks and resource waste. When clients disconnected, subscriptions would remain active upstream even when no longer needed, causing:

  • Accumulation of orphaned subscriptions
  • Unnecessary resource consumption
  • Potential memory leaks
  • No tracking of which clients were using which subscriptions

New Architecture Overview

This PR introduces a comprehensive client lifecycle tracking system for WebSocket subscriptions with automatic cleanup:

Phase 1: Reference Counting Infrastructure

  • SubscriptionHolder: Manages individual subscriptions with client tracking instead of simple reference counting
  • SubscriptionCleanup: Interface for handling cleanup operations when subscriptions are no longer needed
  • Updated GenericIngressSubscription to support proper subscription lifecycle management

Phase 2: Client Lifecycle Tracking

  • SubscriptionClient: Interface representing a client with unique identification and disconnect callbacks
  • DefaultSubscriptionClient: Default implementation with automatic cleanup on disconnect
  • ClientAwareSubscriptionConnect: Wrapper that monitors client connection state and triggers cleanup on disconnect/cancel/error
  • Enhanced SubscriptionHolder to track individual clients rather than just reference counts

Phase 3: Comprehensive Testing

  • Added extensive test coverage for the new subscription management system
  • Tests verify proper client tracking, cleanup behavior, and edge cases
  • Updated existing tests to work with the new architecture

Key Features

  1. Automatic Cleanup: Subscriptions are automatically cleaned up when the last client disconnects
  2. Client Tracking: Each client gets a unique ID and is tracked individually
  3. Robust Disconnect Handling: Handles disconnection via cancellation, completion, errors, and manual disconnect
  4. Resource Management: Prevents memory leaks by properly unsubscribing from upstream when no clients remain
  5. Monitoring Support: Provides subscription statistics and client information for monitoring
  6. Thread Safety: All operations are thread-safe using concurrent data structures

Implementation Details

  • Client ID Generation: Unique IDs based on timestamp, thread ID, and random component
  • Subscription Sharing: Multiple clients can share the same subscription (topic + params)
  • Graceful Cleanup: Automatic unsubscription from upstream using appropriate methods (e.g., eth_unsubscribe, accountUnsubscribe)
  • Error Handling: Robust error handling during cleanup operations with proper logging

@tonatoz tonatoz force-pushed the auto-unsubscribe branch from bc6b1f4 to 3aec7f5 Compare July 16, 2025 12:22
@tonatoz tonatoz requested a review from KirillPamPam July 16, 2025 12:23
@tonatoz tonatoz force-pushed the auto-unsubscribe branch from 3aec7f5 to cb4c78a Compare July 16, 2025 12:34
tonatoz added 8 commits July 22, 2025 12:47
- Add SubscriptionHolder class with reference counting
- Add SubscriptionCleanup interface for cleanup operations
- Update GenericIngressSubscription with reference counting support
- Fix validateErigonBug typo in ChainOptions and EthereumChainSpecific
- Add proper cleanup mechanism for WebSocket subscriptions
- Add SubscriptionClient interface for client identification
- Create DefaultSubscriptionClient with disconnect callbacks
- Add ClientAwareSubscriptionConnect wrapper for tracking client lifecycle
- Update SubscriptionHolder to track individual clients instead of ref counting
- Implement automatic cleanup when clients disconnect
- Add client ID generation for unique client identification
…ription

Add client tracking and auto-cleanup capabilities to EthereumWsIngressSubscription:
- Integrate with SubscriptionHolder and DefaultSubscriptionClient for lifecycle tracking
- Add cleanup logic with proper eth_unsubscribe calls
- Add subscription statistics and monitoring capabilities
- Implement proper client reference counting to trigger cleanup when last client disconnects

This establishes the foundation for Ethereum subscription auto-cleanup similar to other blockchain implementations.
Create EthereumSubscriptionConnect class for client-tracked Ethereum subscriptions:
- Support for newHeads, logs, and pendingTransactions via eth_subscribe/eth_unsubscribe
- Integrate with GenericPersistentConnect pattern for automatic cleanup
- Expand EthereumWsIngressSubscription to handle all Ethereum subscription types
- Add proper parameter handling for different subscription methods (logs with filters, etc.)
- Maintain subscription ID tracking for cleanup coordination

This provides the connection infrastructure needed for Ethereum subscription auto-cleanup.
Wire EthereumEgressSubscription to use EthereumWsIngressSubscription for auto-cleanup:
- Add IngressSubscription parameter to EthereumEgressSubscription constructor
- Route subscription requests through client-tracked IngressSubscription when available
- Fall back to direct connections for backward compatibility (legacy mode)
- Update EthereumChainSpecific to wire IngressSubscription to EgressSubscription
- Add backward compatible constructor to maintain test compatibility
- Log mode (client-tracked vs legacy) for debugging

This completes the integration allowing Ethereum subscriptions to benefit from automatic cleanup when clients disconnect.
- Test EthereumWsIngressSubscription client tracking and cleanup logic
- Verify multiple clients can share same subscription with proper reference counting
- Test automatic cleanup when last client disconnects
- Verify eth_unsubscribe is called on subscription cleanup
- Test different subscription topics are handled independently
- Confirm unique client ID generation and subscription statistics

These tests validate the core auto-cleanup functionality works correctly for Ethereum subscriptions.
@tonatoz tonatoz force-pushed the auto-unsubscribe branch from cb4c78a to cd1580c Compare July 22, 2025 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants