Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 27 additions & 19 deletions docs/_snippets/_S3_authentication_and_bucket.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,37 +20,41 @@
<details>
<summary>Create S3 buckets and an IAM user</summary>

This article demonstrates the basics of how to configure an AWS IAM user, create an S3 bucket and configure ClickHouse to use the bucket as an S3 disk. You should work with your security team to determine the permissions to be used, and consider these as a starting point.
This article demonstrates the basics of how to configure an AWS IAM user, create an S3 bucket and configure ClickHouse to use the bucket as an S3 disk.
You should work with your security team to determine the permissions to be used, and consider these as a starting point.

### Create an AWS IAM user {#create-an-aws-iam-user}
In this procedure, we'll be creating a service account user, not a login user.
1. Log into the AWS IAM Management Console.

2. In "users", select **Add users**
In the following steps you'll be creating a service account user (not a login user).

1. Log into the AWS IAM Management Console.

2. In the `Users` menu, select `Create user`

<Image size="md" img={s3_1} alt="AWS IAM Management Console - Adding a new user" border force/>

3. Enter the user name and set the credential type to **Access key - Programmatic access** and select **Next: Permissions**
3. Enter the username and set the credential type to `Access key - Programmatic access` and select `Next: Permissions`

<Image size="md" img={s3_2} alt="Setting user name and access type for IAM user" border force/>

4. Do not add the user to any group; select **Next: Tags**
4. Do not add the user to any group; select `Next: Tags`

Check notice on line 40 in docs/_snippets/_S3_authentication_and_bucket.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Contractions

Suggestion: Use 'don't' instead of 'Do not'.

<Image size="md" img={s3_3} alt="Skipping group assignment for IAM user" border force/>

5. Unless you need to add any tags, select **Next: Review**
5. Unless you need to add any tags, select `Next: Review`

<Image size="md" img={s3_4} alt="Skipping tag assignment for IAM user" border force/>

6. Select **Create User**
6. Select `Create User`

:::note
The warning message stating that the user has no permissions can be ignored; permissions will be granted on the bucket for the user in the next section
:::
:::note
The warning message stating that the user has no permissions can be ignored; permissions will be granted on the bucket for the user in the next section
:::

<Image size="md" img={s3_5} alt="Creating the IAM user with no permissions warning" border force/>

7. The user is now created; click on **show** and copy the access and secret keys.
7. The user is now created; click on `show` and copy the access and secret keys.

:::note
Save the keys somewhere else; this is the only time that the secret access key will be available.
:::
Expand All @@ -66,49 +70,53 @@
<Image size="md" img={s3_8} alt="Copying the ARN of the IAM user" border force/>

### Create an S3 bucket {#create-an-s3-bucket}
1. In the S3 bucket section, select **Create bucket**

1. In the S3 bucket section, select `Create bucket`

<Image size="md" img={s3_9} alt="Starting the S3 bucket creation process" border force/>

2. Enter a bucket name, leave other options default

:::note
The bucket name must be unique across AWS, not just the organization, or it will emit an error.
:::

3. Leave `Block all Public Access` enabled; public access is not needed.

<Image size="md" img={s3_a} alt="Configuring the S3 bucket settings with public access blocked" border force/>

4. Select **Create Bucket** at the bottom of the page
4. Select `Create Bucket` at the bottom of the page

<Image size="md" img={s3_b} alt="Finalizing S3 bucket creation" border force/>

5. Select the link, copy the ARN, and save it for use when configuring the access policy for the bucket.

6. Once the bucket has been created, find the new S3 bucket in the S3 buckets list and select the link
6. Once the bucket has been created, find the new S3 bucket in the S3 bucket list and select the link

<Image size="md" img={s3_c} alt="Finding the newly created S3 bucket in the buckets list" border force/>

7. Select **Create folder**
7. Select `Create folder`

<Image size="md" img={s3_d} alt="Creating a new folder in the S3 bucket" border force/>

8. Enter a folder name that will be the target for the ClickHouse S3 disk and select **Create folder**
8. Enter a folder name that will be the target for the ClickHouse S3 disk and select `Create folder`

<Image size="md" img={s3_e} alt="Setting the folder name for ClickHouse S3 disk usage" border force/>

9. The folder should now be visible on the bucket list

<Image size="md" img={s3_f} alt="Viewing the newly created folder in the S3 bucket" border force/>

10. Select the checkbox for the new folder and click on **Copy URL** Save the URL copied to be used in the ClickHouse storage configuration in the next section.
10. Select the checkbox for the new folder and click on `Copy URL` Save the URL copied to be used in the ClickHouse storage configuration in the next section.

<Image size="md" img={s3_g} alt="Copying the S3 folder URL for ClickHouse configuration" border force/>

11. Select the **Permissions** tab and click on the **Edit** button in the **Bucket Policy** section
11. Select the `Permissions` tab and click on the `Edit` button in the `Bucket Policy` section

<Image size="md" img={s3_h} alt="Accessing the S3 bucket policy configuration" border force/>

12. Add a bucket policy, example below:

```json
{
"Version" : "2012-10-17",
Expand Down
79 changes: 45 additions & 34 deletions docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,45 +10,55 @@
import Image from '@theme/IdealImage';
import secure_s3 from '@site/static/images/cloud/security/secures3.png';
import s3_info from '@site/static/images/cloud/security/secures3_arn.png';
import s3_output from '@site/static/images/cloud/security/secures3_output.jpg';
import s3_output from '@site/static/images/cloud/security/secures3_output.png';

This article demonstrates how ClickHouse Cloud customers can leverage role-based access to authenticate with Amazon Simple Storage Service (S3) and access their data securely.

## Introduction {#introduction}

Before diving into the setup for secure S3 access, it is important to understand how this works. Below is an overview of how ClickHouse services can access private S3 buckets by assuming into a role within customers' AWS account.

<Image img={secure_s3} size="lg" alt="Overview of Secure S3 Access with ClickHouse"/>
<br/>
<Image img={secure_s3} size="md" alt="Overview of Secure S3 Access with ClickHouse"/>
<br/>

This approach allows customers to manage all access to their S3 buckets in a single place (the IAM policy of the assumed-role) without having to go through all of their bucket policies to add or remove access.
In the section below, you will learn how to set this up.

## Setup {#setup}

### Obtaining the ClickHouse service IAM role ARN {#obtaining-the-clickhouse-service-iam-role-arn}
## Obtain the IAM role ARN of your ClickHouse service {#obtaining-the-clickhouse-service-iam-role-arn}

1 - Login to your ClickHouse cloud account.
1. Login to your ClickHouse cloud account.

2 - Select the ClickHouse service you want to create the integration
2. Select the ClickHouse service you want to create the integration

3 - Select the **Settings** tab
3. Select the **Settings** tab

4 - Scroll down to the **Network security information** section at the bottom of the page
4. Scroll down to the **Network security information** section at the bottom of the page

5 - Copy the **Service role ID (IAM)** value belong to the service as shown below.
5. Copy the **Service role ID (IAM)** value belong to the service as shown below.

<Image img={s3_info} size="lg" alt="Obtaining ClickHouse service IAM Role ARN" border />

### Setting up IAM assume role {#setting-up-iam-assume-role}
## Set up IAM assume role {#setting-up-iam-assume-role}

#### Option 1: Deploying with CloudFormation stack {#option-1-deploying-with-cloudformation-stack}
The IAM assume role can be setup in one of two ways:
- [Using CloudFormation stack](#option-1-deploying-with-cloudformation-stack)
- [Manually creating an IAM role](#option-2-manually-create-iam-role)

1 - Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role.
### Deploying with CloudFormation stack {#option-1-deploying-with-cloudformation-stack}

2 - Visit [this url](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/quickcreate?templateURL=https://s3.us-east-2.amazonaws.com/clickhouse-public-resources.clickhouse.cloud/cf-templates/secure-s3.yaml&stackName=ClickHouseSecureS3) to populate the CloudFormation stack.
1. Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role.

3 - Enter (or paste) the **IAM Role** belong to the ClickHouse service
2. Visit the following [CloudFormation URL](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/quickcreate?templateURL=https://s3.us-east-2.amazonaws.com/clickhouse-public-resources.clickhouse.cloud/cf-templates/secure-s3.yaml&stackName=ClickHouseSecureS3) to populate the CloudFormation stack.

4 - Configure the CloudFormation stack. Below is additional information about these parameters.
3. Enter (or paste) the **service role ID (IAM)** for your service that you obtained earlier into the input titled "ClickHouse Instance Roles"
You can paste the service role ID exactly as it appears in Cloud console.

4. Enter your bucket name in the input titled "Bucket Names". If your bucket URL is `https://ch-docs-s3-bucket.s3.eu-central-1.amazonaws.com/clickhouseS3/` then the bucket name is `ch-docs-s3-bucket`.

:::note
Do not put the full bucket ARN but instead just the bucket name only.
:::

5. Configure the CloudFormation stack. Below is additional information about these parameters.

| Parameter | Default Value | Description |
| :--- | :----: | :---- |
Expand All @@ -58,29 +68,27 @@
| Bucket Access | Read | Sets the level of access for the provided buckets. |
| Bucket Names | | Comma separated list of **bucket names** that this role will have access to. |

*Note*: Do not put the full bucket Arn but instead just the bucket name only.

5 - Select the **I acknowledge that AWS CloudFormation might create IAM resources with custom names.** checkbox
6. Select the **I acknowledge that AWS CloudFormation might create IAM resources with custom names.** checkbox

6 - Click **Create stack** button at bottom right
7. Click the **Create stack** button at the bottom right

7 - Make sure the CloudFormation stack completes with no error.
8. Make sure the CloudFormation stack completes with no error.

8 - Select the **Outputs** of the CloudFormation stack
9. Select the newly created Stack then select the **Outputs** tab of the CloudFormation stack

9 - Copy the **RoleArn** value for this integration. This is what needed to access your S3 bucket.
10. Copy the **RoleArn** value for this integration, which is what you need to access your S3 bucket.

<Image img={s3_output} size="lg" alt="CloudFormation stack output showing IAM Role ARN" border />

#### Option 2: Manually create IAM role {#option-2-manually-create-iam-role}
### Manually create IAM role {#option-2-manually-create-iam-role}

1 - Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role.
1. Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role.

2 - Browse to IAM Service Console
2. Browse to the IAM Service Console

3 - Create a new IAM role with the following IAM & Trust policy.
3. Create a new IAM role with the following IAM & Trust policy

Trust policy (Please replace `{ClickHouse_IAM_ARN}` with the IAM Role arn belong to your ClickHouse instance):
Trust policy (Please replace `{ClickHouse_IAM_ARN}` with the IAM Role arn belong to your ClickHouse instance):

Check notice on line 91 in docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Wordy

Suggestion: Use 'please' only if we've inconvenienced the user.

```json
{
Expand Down Expand Up @@ -127,22 +135,25 @@
}
```

4 - Copy the new **IAM Role Arn** after creation. This is what needed to access your S3 bucket.
4. Copy the new **IAM Role Arn** after creation, which is what is needed to access your S3 bucket.

Check notice on line 138 in docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Contractions

Suggestion: Use 'what's' instead of 'what is'.

## Access your S3 bucket with the ClickHouseAccess role {#access-your-s3-bucket-with-the-clickhouseaccess-role}

ClickHouse Cloud has a new feature that allows you to specify `extra_credentials` as part of the S3 table function. Below is an example of how to run a query using the newly created role copied from above.
ClickHouse Cloud allows you to specify `extra_credentials` as part of the S3 table function.
Below is an example of how to run a query using the newly created role copied from above.

```sql
DESCRIBE TABLE s3('https://s3.amazonaws.com/BUCKETNAME/BUCKETOBJECT.csv','CSVWithNames',extra_credentials(role_arn = 'arn:aws:iam::111111111111:role/ClickHouseAccessRole-001'))
```

Below is an example query that uses the `role_session_name` as a shared secret to query data from a bucket. If the `role_session_name` is not correct, this operation will fail.
Below is an example query that uses the `role_session_name` as a shared secret to query data from a bucket.
If the `role_session_name` is not correct, this operation will fail.

```sql
DESCRIBE TABLE s3('https://s3.amazonaws.com/BUCKETNAME/BUCKETOBJECT.csv','CSVWithNames',extra_credentials(role_arn = 'arn:aws:iam::111111111111:role/ClickHouseAccessRole-001', role_session_name = 'secret-role-name'))
```

:::note
We recommend that your source S3 is in the same region as your ClickHouse Cloud Service to reduce on data transfer costs. For more information, refer to [S3 pricing]( https://aws.amazon.com/s3/pricing/)
We recommend that your source S3 is in the same region as your ClickHouse Cloud Service to reduce on data transfer costs.
For more information, refer to [S3 pricing]( https://aws.amazon.com/s3/pricing/)
:::
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
sidebar_label: 'ClickHouse OSS'
sidebar_label: 'Using remoteSecure'
slug: /cloud/migration/clickhouse-to-cloud
title: 'Migrating between self-managed ClickHouse and ClickHouse Cloud'
description: 'Page describing how to migrate between self-managed ClickHouse and ClickHouse Cloud'
Expand All @@ -16,13 +16,13 @@ import self_managed_04 from '@site/static/images/integrations/migration/self-man
import self_managed_05 from '@site/static/images/integrations/migration/self-managed-05.png';
import self_managed_06 from '@site/static/images/integrations/migration/self-managed-06.png';

# Migrating between self-managed ClickHouse and ClickHouse Cloud
# Migrating between self-managed ClickHouse and ClickHouse Cloud using remoteSecure

<Image img={self_managed_01} size='lg' alt='Migrating Self-managed ClickHouse'/>

This guide will show how to migrate from a self-managed ClickHouse server to ClickHouse Cloud, and also how to migrate between ClickHouse Cloud services. The [`remoteSecure`](/sql-reference/table-functions/remote) function is used in `SELECT` and `INSERT` queries to allow access to remote ClickHouse servers, which makes migrating tables as simple as writing an `INSERT INTO` query with an embedded `SELECT`.

## Migrating from Self-managed ClickHouse to ClickHouse Cloud {#migrating-from-self-managed-clickhouse-to-clickhouse-cloud}
## Migrating from self-managed ClickHouse to ClickHouse Cloud {#migrating-from-self-managed-clickhouse-to-clickhouse-cloud}

<Image img={self_managed_02} size='lg' alt='Migrating Self-managed ClickHouse' />

Expand Down
Loading