Fix Fake-TLS ClientHello detection to support TLS 1.2/1.3

[Copilot]

Fake-TLS connections fail when using the -D domain option because the TLS ClientHello detection only matches TLS 1.0. Modern Telegram clients use TLS 1.2/1.3, causing connections to be incorrectly proxied to the fallback domain instead of being handled as MTProxy traffic.

Changes

TLS version detection (line 1115)

// Before: Only matches TLS 1.0 (0x16 0x03 0x01)
(packet_len & 0xFFFFFF) == 0x010316

// After: Matches TLS 1.0/1.2/1.3 (0x16 0x03 0x??)
(packet_len & 0xFF) == 0x16 && ((packet_len >> 8) & 0xFF) == 0x03

Debug logging

• Added packet_len, ext_secret_cnt, allow_only_tls, and C_IS_TLS flags to connection type detection logging
• Added TLS ClientHello detection confirmation with version info

Technical Details

TLS ClientHello record header structure:

• Byte 0: 0x16 (Handshake record type)
• Byte 1-2: Protocol version (0x0301 = TLS 1.0, 0x0303 = TLS 1.2/1.3)
• Byte 3-4: Record length

When read as little-endian uint32:

• TLS 1.0: 0x??010316
• TLS 1.2/1.3: 0x??030316

The fixed condition checks bytes 0-1 only, accepting any TLS minor version.

Original prompt

Problem

When using the -D (domain) option to enable Fake-TLS mode, incoming Telegram client connections are not being handled correctly. Instead of processing them as MTProxy Fake-TLS connections, the server proxies them to the configured domain (e.g., Google).

Reproduction Steps

1. Generate secret: head -c 16 /dev/urandom | xxd -ps
2. Get domain hex: echo -n www.google.com | xxd -plain → 7777772e676f6f676c652e636f6d
3. Run server: ./mtproto-proxy -u nobody -p 8888 -H 443 -S <secret> -D www.google.com --aes-pwd proxy-secret proxy-multi.conf -M 1
4. Connect in Telegram with secret: ee<secret>7777772e676f6f676c652e636f6d

Expected Behavior

The Telegram client should connect successfully through the MTProxy.

Actual Behavior

From the verbose log, we can see:

[1442] trying to determine type of connection from {CLIENT_IP}:34344
[1442] New outbound connection #89 {HOST_IP}:48230 -> 142.250.179.132:443

The server is proxying connections to Google (142.250.179.132) instead of handling them as MTProxy traffic. Notably, there's no "TLS type with domain..." log message, which means the TLS ClientHello parsing block at line 1113 in net/net-tcp-rpc-ext-server.c is not being entered.

Root Cause Analysis

The TLS ClientHello handling block at line 1113 requires this condition:

} else if ((packet_len & 0xFFFFFF) == 0x010316 && (packet_len >> 24) >= 2 && ext_secret_cnt > 0 && allow_only_tls) {

Looking at the log output, we see:

1. Domain check passes: "Successfully checked domain www.google.com"
2. Secrets are loaded (via -S option)
3. But connections are still proxied

The issue is that connections are reaching line 1251-1253:

if (allow_only_tls && !(c->flags & C_IS_TLS)) {
  vkprintf (1, "Expected TLS-transport\n");
  RETURN_TLS_ERROR(default_domain_info);
}

This happens because the TLS ClientHello parsing block isn't being entered. Possible causes:

1. The condition (packet_len & 0xFFFFFF) == 0x010316 may not match the actual TLS ClientHello header from Telegram clients
2. There may be a byte-order issue with the packet length check

Requested Fix

1. Add debug logging to identify exactly which condition is failing at line 1113
2. Verify the TLS ClientHello detection logic matches what Telegram clients send
3. Fix the condition so Fake-TLS connections are properly recognized and handled

Related Issues

• Fixes Fake TSL not working #23 (Fake TLS not working)
• Related to PR Fix Fake-TLS (EE mode) rejected when domain option specified #24 (which fixed a different part of the problem but didn't solve the root cause)

This pull request was created from Copilot chat.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[dvershinin]

@copilot FYI, a user reported that everything worked fine with ee before using commit f9d39fb and broke when they used 6f20cec