Skip to content

feat(gs): add debug endpoint for secure database queries #2770

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
TaprootFreak wants to merge 8 commits into
base: develop
Choose a base branch
from
Open

feat(gs): add debug endpoint for secure database queries #2770

TaprootFreak wants to merge 8 commits into from

Conversation

@TaprootFreak
Copy link
Collaborator

@TaprootFreak TaprootFreak commented Dec 31, 2025 •
edited
Loading

Summary

  • Add DEBUG user role for developer database access
  • Add POST /gs/debug endpoint for secure read-only SQL queries
  • Add POST /gs/debug/logs endpoint for Azure App Insights log queries
  • 12-layer security validation with AST-based SQL parsing

Endpoints

POST /gs/debug - SQL Query

Execute read-only SQL queries against the database with automatic PII filtering.

POST /gs/debug/logs - Log Query

Query Azure App Insights logs using predefined KQL templates.

Security Layers (SQL Endpoint)

# Layer Protection
1 AST Parsing node-sql-parser validates syntax
2 SELECT Only No INSERT/UPDATE/DELETE
3 Single Statement No multi-statement queries
4 No UNION UNION/INTERSECT/EXCEPT blocked
5 No SELECT INTO Write operations blocked
6 Schema Blocking sys, information_schema, master, msdb, tempdb
7 Function Blocking OPENROWSET, OPENQUERY, OPENDATASOURCE, OPENXML
8 No FOR XML/JSON Data exfiltration blocked
9 Column Blocking 56 PII columns blocked pre-execution
10 TOP Validation Max 10,000 results via AST
11 Result Limit OFFSET/FETCH enforced
12 Post-Masking Defense-in-depth column masking

KQL Templates (Log Endpoint)

  • traces-by-operation - Traces by operation ID
  • traces-by-message - Traces by message filter
  • exceptions-recent - Recent exceptions
  • request-failures - Failed requests
  • dependencies-slow - Slow dependencies
  • custom-events - Custom events by name

Access Control

Role Access
DEBUG ✅ /gs/debug, /gs/debug/logs
ADMIN ✅ All DEBUG + /gs/db
SUPER_ADMIN ✅ All

@Yannick1712
Copy link
Member

Yannick1712 commented Dec 31, 2025

@TaprootFreak TaprootFreak force-pushed the feature/debug-endpoint branch 4 times, most recently from fc86ace to 6495c2d Compare December 31, 2025 12:08
@TaprootFreak
feat(gs): add debug endpoint for secure database queries
1b1b934 
Add new DEBUG user role for developer database access with POST /gs/debug
endpoint for executing read-only SQL queries.

Security layers:
- Role-based access (DEBUG/ADMIN/SUPER_ADMIN only)
- SQL parsing with node-sql-parser (AST validation)
- Only single SELECT statements allowed
- Blocked: UNION/INTERSECT/EXCEPT, SELECT INTO, FOR XML/JSON
- Blocked: OPENROWSET, OPENQUERY, OPENDATASOURCE (external connections)
- Pre-execution column checking (blocks alias bypass)
- Input validation with MaxLength(10000)
- Post-execution PII column masking (defense in depth)
- Full audit trail with user identification

Blocked columns: mail, email, firstname, surname, iban, ip, apiKey, etc.
@TaprootFreak TaprootFreak force-pushed the feature/debug-endpoint branch from 6495c2d to 1b1b934 Compare December 31, 2025 12:13
@TaprootFreak
fix(gs): resolve eslint warnings in debug endpoint
9e2baad 
- Remove unused catch variable (use bare catch)
- Remove unnecessary eslint-disable directive
Yannick1712 and others added 2 commits December 31, 2025 13:57
@Yannick1712 @TaprootFreak
[NOTASK] add more blockedCols
a2a3cc1
@TaprootFreak
fix(gs): move restricted columns from ADMIN to DEBUG blocking
07e241d 
- Remove organization.name, bank_tx.name, kyc_step.result from RestrictedColumns
- Add 'name' and 'result' to DebugBlockedColumns
- ADMIN can now see these columns on /gs/db
- DEBUG role has these blocked on /gs/debug
@TaprootFreak TaprootFreak mentioned this pull request Dec 31, 2025
Merged
5 tasks
@TaprootFreak
fix(gs): implement table-specific column blocking for debug endpoint (#…
b28dab1 
…2782)

* fix(gs): implement table-specific column blocking for debug endpoint

Replace generic DebugBlockedColumns list with table-specific blocking:
- TableBlockedColumns: Record<string, string[]> maps each table to its blocked columns
- Pre-execution: Check columns against their specific tables
- Post-execution for SELECT *: Mask columns from all query tables

Examples:
- SELECT name FROM asset → ALLOWED (asset has no blocked columns)
- SELECT name FROM bank_tx → BLOCKED (bank_tx.name contains personal data)
- SELECT * FROM bank_tx → name masked post-execution

Tables with blocked columns:
- user_data: mail, phone, firstname, surname, etc.
- bank_tx: name, iban, addressLine1, etc.
- bank_data: name, iban, label, comment
- kyc_step: result (contains names, birthday)
- organization: name, allBeneficialOwnersName, etc.

* fix(gs): always run post-execution masking for defense in depth

The previous implementation only masked post-execution for SELECT * queries.
This was a security risk: if pre-execution column extraction failed (catch block),
non-wildcard queries would not be masked.

Now post-execution masking always runs, ensuring blocked columns are masked
even if the SQL parser fails to detect them pre-execution.

* fix(gs): add missing blocked columns from original list

Add columns that were in the original DebugBlockedColumns but missing
in the new table-specific structure:

- user_data: countryId, verifiedCountryId, nationalityId
- user: signature
- fiat_output: accountNumber
- checkout_tx: cardName (new table)
- bank_account: accountNumber (new table)

* fix(gs): add missing tables with sensitive columns

Add tables that were missed when converting from global DebugBlockedColumns
to table-specific TableBlockedColumns:

- ref: ip (user IP for referral tracking)
- ip_log: ip, country (user IP logging)
- checkout_tx: ip (user IP during checkout, cardName already present)
- buy: iban (user IBAN for buy routes)
- deposit_route: iban (user IBAN for sell routes via Single Table Inheritance)

These columns were blocked globally in the original implementation but
were not added to all relevant tables in the table-specific version.

* fix(gs): add additional sensitive columns found in codebase review

Add missing blocked columns discovered during comprehensive entity scan:

- buy_crypto: chargebackIban (user IBAN for refunds)
- kyc_log: ipAddress (TfaLog), result (KYC data)
- bank_tx_return: chargebackIban, recipientMail, chargebackRemittanceInfo
- bank_tx_repeat: chargebackIban, chargebackRemittanceInfo
- limit_request: recipientMail
- ref_reward: recipientMail

* fix(gs): add additional sensitive columns from codebase review

Extend existing tables with missing blocked columns:
- checkout_tx: cardBin, cardLast4, cardFingerPrint, cardIssuer, cardIssuerCountry, raw
- buy_crypto: chargebackRemittanceInfo, siftResponse
- buy_fiat: remittanceInfo, usedBank, info
- crypto_input: senderAddresses
- user_data: relatedUsers
- limit_request: fundOriginText
- bank_tx_return: info

Add new tables with sensitive columns:
- transaction_risk_assessment: reason, methods, summary, result (AML/KYC assessments)
- support_issue: name, information (support tickets with user data)
- support_message: message, fileUrl (message content and files)
- sift_error_log: requestPayload (Sift API requests with PII)

* fix(gs): add webhook.data to blocked columns

* fix(gs): add notification.data to blocked columns

* fix(gs): add kyc_step.data to blocked columns
@TaprootFreak
feat(gs): add App Insights log query endpoint with template-based sec…
abec1b0 
…urity (#2778)

Add POST /gs/debug/logs endpoint for querying Azure Application Insights
logs using predefined, safe KQL templates.

Security features:
- Template-based queries only (no free-form KQL input)
- Strict parameter validation via class-validator (GUID, alphanumeric)
- All KQL-relevant special characters blocked in user input
- Defense-in-depth string escaping
- Result limits per template (200-500 rows)
- Full audit logging of queries

Available templates:
- traces-by-operation: Traces for specific operation ID
- traces-by-message: Traces filtered by message pattern
- exceptions-recent: Recent exceptions
- request-failures: Failed HTTP requests
- dependencies-slow: Slow external dependencies (by duration threshold)
- custom-events: Custom events by name

Infrastructure:
- AppInsightsQueryService: OAuth2 client with token caching
- Proper error handling and logging
- Mock responses for LOC mode

Requires UserRole.DEBUG and APPINSIGHTS_APP_ID env variable.
@TaprootFreak
fix(gs): add security hardening for debug SQL endpoint
24c3311 
Security improvements:

1. Block system tables and schemas:
   - Added BlockedSchemas list: sys, information_schema, master, msdb, tempdb
   - checkForBlockedSchemas() validates FROM clause and subqueries
   - Prevents access to sys.sql_logins, INFORMATION_SCHEMA.TABLES, etc.

2. Fix TOP validation to use AST instead of regex:
   - Previous regex /\btop\s+(\d+)/ missed TOP(n) with parentheses
   - Now uses stmt.top?.value from AST for accurate detection
   - Both TOP 100 and TOP(100) are correctly validated

3. Extend dangerous function check to all clauses:
   - Previous check only validated FROM clause
   - Now recursively checks SELECT columns and WHERE clauses
   - checkForDangerousFunctionsRecursive() traverses entire AST
   - Blocks OPENROWSET, OPENQUERY, OPENDATASOURCE, OPENXML everywhere
@TaprootFreak
refactor(gs): improve code professionalism in debug endpoints
9389c74 
- Remove commented debug code
- Fix return type any[] → Record<string, unknown>[]
- Remove redundant try-catch in controller (service handles errors)
- Rename misleading parameter userMail → userIdentifier
- Standardize comment style
@TaprootFreak
Copy link
Collaborator Author

TaprootFreak commented Jan 2, 2026

✅ Ready to Merge - Security Review Complete

This PR has undergone comprehensive security review and is production-ready.

Security Implementation (12 Validation Layers)

SQL Debug Endpoint (POST /gs/debug):

  1. AST-based parsing via node-sql-parser (not regex)
  2. SELECT-only enforcement
  3. Single statement validation
  4. UNION/INTERSECT blocking
  5. SELECT INTO prevention
  6. System schema blocking (sys, information_schema, master, msdb, tempdb)
  7. Dangerous function detection - recursive across all clauses (OPENROWSET, OPENQUERY, OPENDATASOURCE, OPENXML)
  8. FOR XML/JSON blocking
  9. Pre-execution column validation
  10. AST-based TOP validation (handles both TOP n and TOP(n))
  11. Result limiting (max 10,000 rows)
  12. Post-execution PII masking (56 blocked columns)

App Insights Log Query (POST /gs/debug/logs):

  • Template-based approach (6 predefined KQL queries)
  • Strict parameter validation via class-validator
  • KQL string escaping
  • OAuth2 token caching with expiry buffer
  • Full audit logging

Access Control

  • Both endpoints require DEBUG role
  • Triple guard protection: AuthGuard, RoleGuard, UserActiveGuard
  • Role hierarchy: DEBUG → ADMIN/SUPER_ADMIN

Code Quality

  • No commented-out debug code
  • Proper TypeScript types (Record<string, unknown>[] instead of any[])
  • Consistent naming conventions
  • Full audit trail with user identification

All identified security issues have been addressed. The defense-in-depth approach ensures multiple layers of protection against SQL injection, data exfiltration, and unauthorized access.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@davidleomay davidleomay Awaiting requested review from davidleomay

@Yannick1712 Yannick1712 Awaiting requested review from Yannick1712

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@TaprootFreak @Yannick1712