From 35d2e3832429c9416e9d67748d294f49976c015e Mon Sep 17 00:00:00 2001 From: Wenqi Mou Date: Fri, 15 Aug 2025 02:48:51 -0400 Subject: [PATCH 1/4] This is an automated cherry-pick of #21109 Signed-off-by: ti-chi-bot --- br/br-pitr-manual.md | 150 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 150 insertions(+) diff --git a/br/br-pitr-manual.md b/br/br-pitr-manual.md index c873fb2789725..afdef7fe35797 100644 --- a/br/br-pitr-manual.md +++ b/br/br-pitr-manual.md @@ -498,3 +498,153 @@ tiup br restore point --pd="${PD_IP}:2379" --master-key-crypter-method aes128-ctr --master-key "local:///path/to/master.key" ``` +<<<<<<< HEAD +======= + +### Restore data using filters + +Starting from TiDB v9.0.0, you can use filters during PITR to restore specific databases or tables, enabling more fine-grained control over the data to be restored. + +The filter patterns follow the same [table filtering syntax](/table-filter.md) as other BR operations: + +- `'*.*'`: matches all databases and tables. +- `'db1.*'`: matches all tables in the database `db1`. +- `'db1.table1'`: matches the specific table `table1` in the database `db1`. +- `'db*.tbl*'`: matches databases starting with `db` and tables starting with `tbl`. +- `'!mysql.*'`: excludes all tables in the `mysql` database. + +Usage examples: + +```shell +# restore specific databases +tiup br restore point --pd="${PD_IP}:2379" \ +--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--start-ts "2025-06-02 00:00:00+0800" \ +--restored-ts "2025-06-03 18:00:00+0800" \ +--filter 'db1.*' --filter 'db2.*' + +# restore specific tables +tiup br restore point --pd="${PD_IP}:2379" \ +--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--start-ts "2025-06-02 00:00:00+0800" \ +--restored-ts "2025-06-03 18:00:00+0800" \ +--filter 'db1.users' --filter 'db1.orders' + +# restore using pattern matching +tiup br restore point --pd="${PD_IP}:2379" \ +--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--start-ts "2025-06-02 00:00:00+0800" \ +--restored-ts "2025-06-03 18:00:00+0800" \ +--filter 'db*.tbl*' +``` + +> **Note:** +> +> - Before restoring data using filters, ensure that the target cluster does not contain any databases or tables that match the filter. Otherwise, the restore will fail with an error. +> - The filter options apply during the restore phase for both snapshot and log backups. +> - You can specify multiple `--filter` options to include or exclude different patterns. +> - PITR filtering does not support system tables yet. If you need to restore specific system tables, use the `br restore full` command with filters instead. Note that this command restores only the snapshot backup data (not log backup data). + +### Concurrent restore operations + +Starting from TiDB v9.0.0, you can run multiple PITR restore tasks concurrently. This feature allows you to restore different datasets in parallel, improving efficiency for large-scale restore scenarios. + +Usage example for concurrent restores: + +```shell +# terminal 1 - restore database db1 +tiup br restore point --pd="${PD_IP}:2379" \ +--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--start-ts "2025-06-02 00:00:00+0800" \ +--restored-ts "2025-06-03 18:00:00+0800" \ +--filter 'db1.*' + +# terminal 2 - restore database db2 (can run simultaneously) +tiup br restore point --pd="${PD_IP}:2379" \ +--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--start-ts "2025-06-02 00:00:00+0800" \ +--restored-ts "2025-06-03 18:00:00+0800" \ +--filter 'db2.*' +``` + +> **Note:** +> +> Each concurrent restore operation must target a different database or a non-overlapping set of tables. Attempting to restore overlapping datasets concurrently will result in an error. + +### Compatibility between ongoing log backup and snapshot restore + +Starting from v9.0.0, when a log backup task is running, if all of the following conditions are met, you can still perform snapshot restore (`br restore [full|database|table]`) and allow the restored data to be properly recorded by the ongoing log backup (hereinafter referred to as "log backup"): + +- The node performing backup and restore operations has the following necessary permissions: + - Read access to the external storage containing the backup source, for snapshot restore + - Write access to the target external storage used by the log backup +- The target external storage for the log backup is Amazon S3 (`s3://`), Google Cloud Storage (`gcs://`), or Azure Blob Storage (`azblob://`). +- The data to be restored uses the same type of external storage as the target storage for the log backup. +- Neither the data to be restored nor the log backup has enabled local encryption. For details, see [log backup encryption](#encrypt-the-log-backup-data) and [snapshot backup encryption](/br/br-snapshot-manual.md#encrypt-the-backup-data). + +If any of the above conditions are not met, you can restore the data by following these steps: + +1. [Stop the log backup task](#stop-a-log-backup-task). +2. Perform the data restore. +3. After the restore is complete, perform a new snapshot backup. +4. [Restart the log backup task](#restart-a-log-backup-task). + +> **Note:** +> +> When restoring a log backup that contains records of snapshot (full) restore data, you must use BR v9.0.0 or later. Otherwise, restoring the recorded full restore data might fail. + +### Compatibility between ongoing log backup and PITR operations + +Starting from TiDB v9.0.0, you can perform PITR operations while a log backup task is running by default. The system automatically handles compatibility between these operations. + +#### Important limitation for PITR with ongoing log backup + +When you perform the PITR operations while a log backup is running, the restored data will also be recorded in the ongoing log backup. However, due to the nature of log restore operations, data inconsistencies might occur within the restore window. The system writes metadata to external storage to mark both the time range and data range where consistency cannot be guaranteed. + +If such inconsistency occurs during the time range `[t1, t2)`, you cannot directly restore data from this period. Instead, choose one of the following alternatives: + +- Restore data up to `t1` (to retrieve data before the inconsistent period). +- Perform a new snapshot backup after `t2`, and use it as the base for future PITR operations. + +### Abort restore operations + +If a restore operation fails, you can use the `tiup br abort` command to clean up registry entries and checkpoint data. This command automatically locates and removes relevant metadata based on the original restore parameters, including entries in the `mysql.tidb_restore_registry` table and checkpoint data (regardless of whether it is stored in a local database or external storage). + +> **Note:** +> +> The `abort` command only cleans up metadata. You need to manually delete any actual restored data from the cluster. + +The examples of aborting restore operations using the same parameters as the original restore command are as follows: + +```shell +# Abort a PITR operation +tiup br abort restore point --pd="${PD_IP}:2379" \ +--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' + +# Abort a PITR operation with filters +tiup br abort restore point --pd="${PD_IP}:2379" \ +--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--filter 'db1.*' + +# Abort a full restore +tiup br abort restore full --pd="${PD_IP}:2379" \ +--storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' + +# Abort a database restore +tiup br abort restore db --pd="${PD_IP}:2379" \ +--storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--db database_name + +# Abort a table restore +tiup br abort restore table --pd="${PD_IP}:2379" \ +--storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ +--db database_name --table table_name +``` +>>>>>>> 5182861b86 (br: pitr filter feature release doc (#21109)) From ba1bf560b04bc7c2713f05129ac850009d3c1a69 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Wed, 24 Dec 2025 17:44:53 +0800 Subject: [PATCH 2/4] br: resolve conflict --- br/br-pitr-manual.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/br/br-pitr-manual.md b/br/br-pitr-manual.md index afdef7fe35797..2a4ed544b3549 100644 --- a/br/br-pitr-manual.md +++ b/br/br-pitr-manual.md @@ -498,8 +498,6 @@ tiup br restore point --pd="${PD_IP}:2379" --master-key-crypter-method aes128-ctr --master-key "local:///path/to/master.key" ``` -<<<<<<< HEAD -======= ### Restore data using filters @@ -647,4 +645,3 @@ tiup br abort restore table --pd="${PD_IP}:2379" \ --storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \ --db database_name --table table_name ``` ->>>>>>> 5182861b86 (br: pitr filter feature release doc (#21109)) From 0b86d7c1a06058cda8193457166a6a80d9e25d78 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Wed, 24 Dec 2025 17:45:49 +0800 Subject: [PATCH 3/4] Update v9.0.0 to v8.5.5 --- br/br-pitr-manual.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/br/br-pitr-manual.md b/br/br-pitr-manual.md index 2a4ed544b3549..bd7c5bb067029 100644 --- a/br/br-pitr-manual.md +++ b/br/br-pitr-manual.md @@ -501,7 +501,7 @@ tiup br restore point --pd="${PD_IP}:2379" ### Restore data using filters -Starting from TiDB v9.0.0, you can use filters during PITR to restore specific databases or tables, enabling more fine-grained control over the data to be restored. +Starting from TiDB v8.5.5, you can use filters during PITR to restore specific databases or tables, enabling more fine-grained control over the data to be restored. The filter patterns follow the same [table filtering syntax](/table-filter.md) as other BR operations: @@ -548,7 +548,7 @@ tiup br restore point --pd="${PD_IP}:2379" \ ### Concurrent restore operations -Starting from TiDB v9.0.0, you can run multiple PITR restore tasks concurrently. This feature allows you to restore different datasets in parallel, improving efficiency for large-scale restore scenarios. +Starting from TiDB v8.5.5, you can run multiple PITR restore tasks concurrently. This feature allows you to restore different datasets in parallel, improving efficiency for large-scale restore scenarios. Usage example for concurrent restores: @@ -576,9 +576,9 @@ tiup br restore point --pd="${PD_IP}:2379" \ ### Compatibility between ongoing log backup and snapshot restore -Starting from v9.0.0, when a log backup task is running, if all of the following conditions are met, you can still perform snapshot restore (`br restore [full|database|table]`) and allow the restored data to be properly recorded by the ongoing log backup (hereinafter referred to as "log backup"): +Starting from v8.5.5, when a log backup task is running, if all of the following conditions are met, you can still perform snapshot restore (`br restore [full|database|table]`) and allow the restored data to be properly recorded by the ongoing log backup (hereinafter referred to as "log backup"): -- The node performing backup and restore operations has the following necessary permissions: +- The node performing backup and restore operations has the following necessary permissions: - Read access to the external storage containing the backup source, for snapshot restore - Write access to the target external storage used by the log backup - The target external storage for the log backup is Amazon S3 (`s3://`), Google Cloud Storage (`gcs://`), or Azure Blob Storage (`azblob://`). @@ -594,11 +594,11 @@ If any of the above conditions are not met, you can restore the data by followin > **Note:** > -> When restoring a log backup that contains records of snapshot (full) restore data, you must use BR v9.0.0 or later. Otherwise, restoring the recorded full restore data might fail. +> When restoring a log backup that contains records of snapshot (full) restore data, you must use BR v8.5.5 or later. Otherwise, restoring the recorded full restore data might fail. ### Compatibility between ongoing log backup and PITR operations -Starting from TiDB v9.0.0, you can perform PITR operations while a log backup task is running by default. The system automatically handles compatibility between these operations. +Starting from TiDB v8.5.5, you can perform PITR operations while a log backup task is running by default. The system automatically handles compatibility between these operations. #### Important limitation for PITR with ongoing log backup From c5954ab6c0e72eaa7e9636c80c6f552bdf77bff9 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Wed, 24 Dec 2025 17:46:53 +0800 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- br/br-pitr-manual.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-pitr-manual.md b/br/br-pitr-manual.md index bd7c5bb067029..1b1bf688d35bc 100644 --- a/br/br-pitr-manual.md +++ b/br/br-pitr-manual.md @@ -617,7 +617,7 @@ If a restore operation fails, you can use the `tiup br abort` command to clean u > > The `abort` command only cleans up metadata. You need to manually delete any actual restored data from the cluster. -The examples of aborting restore operations using the same parameters as the original restore command are as follows: +The following examples show how to abort restore operations using the same parameters as the original restore command: ```shell # Abort a PITR operation