Skip to content

docs: Update manual AWS Batch launch template scripts #993

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adamrtalbot wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docs: Update manual AWS Batch launch template scripts #993

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 38 additions & 15 deletions platform-cloud/docs/enterprise/advanced-topics/manual-aws-batch-setup.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,18 +127,22 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:
owner: root
content: |
#!/usr/bin/env bash
exec > >(tee /var/log/tower-forge.log|logger -t TowerForge -s 2>/dev/console) 2>&1
##
yum install -q -y jq sed wget unzip nvme-cli lvm2
wget -q https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
## install CloudWatch agent
curl -s https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm -o amazon-cloudwatch-agent.rpm
rpm -U ./amazon-cloudwatch-agent.rpm
rm -f ./amazon-cloudwatch-agent.rpm
curl -s https://nf-xpack.seqera.io/amazon-cloudwatch-agent/custom-v0.1.json \
# | sed 's/custom-id/<your custom ID>/g' \
curl -s https://nf-xpack.seqera.io/amazon-cloudwatch-agent/config-v0.4.json \
# | sed 's/$FORGE_ID/<your custom ID>/g' \
> /opt/aws/amazon-cloudwatch-agent/bin/config.json
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
-a fetch-config \
-m ec2 \
-s \
-c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
## format NVMe storage for Fusion
mkdir -p /scratch/fusion
NVME_DISKS=($(nvme list | grep 'Amazon EC2 NVMe Instance Storage' | awk '{ print $1 }'))
NUM_DISKS=${#NVME_DISKS[@]}
Expand All @@ -155,18 +159,26 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:
fi
fi
chmod a+w /scratch/fusion
## ECS configuration
mkdir -p /etc/ecs
echo ECS_IMAGE_PULL_BEHAVIOR=once >> /etc/ecs/ecs.config
echo ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true >> /etc/ecs/ecs.config
echo ECS_ENABLE_SPOT_INSTANCE_DRAINING=true >> /etc/ecs/ecs.config
echo ECS_CONTAINER_CREATE_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_CONTAINER_START_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_CONTAINER_STOP_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_MANIFEST_PULL_TIMEOUT=10m >> /etc/ecs/ecs.config
## stop docker
systemctl stop docker
## install AWS CLI
mkdir -p /home/ec2-user
curl -s https://nf-xpack.seqera.io/miniconda-awscli/miniconda-awscli.tar.gz \
| tar xz -C /home/ec2-user
curl -s https://nf-xpack.seqera.io/miniconda-awscli/miniconda-25.3.1-awscli-1.40.12.tar.gz \
| tar xz -C /
export PATH=$PATH:/home/ec2-user/miniconda/bin
ln -s /home/ec2-user/miniconda/bin/aws /usr/bin/aws
## restart docker
systemctl start docker
systemctl enable --now --no-block ecs
## kernel settings to prevent OOM
echo "1258291200" > /proc/sys/vm/dirty_bytes
echo "629145600" > /proc/sys/vm/dirty_background_bytes

Expand All @@ -176,7 +188,7 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:
--//--
```

1. To prepend a custom identifier to the CloudWatch log streams for AWS resources created by your manual compute environment, uncomment the `| sed 's/custom-id/<your custom ID>/g' \` line and replace `<your custom ID>` with your custom ID. If ommitted, this defaults to `custom-id`.
1. To prepend a custom identifier to the CloudWatch log streams for AWS resources created by your manual compute environment, uncomment the `| sed 's/$FORGE_ID/<your custom ID>/g' \` line and replace `<your custom ID>` with your custom identifier. If omitted, `$FORGE_ID` remains as-is in the config.
1. Save the template with the name `seqera-launchtemplate`.

</TabItem>
Expand All @@ -199,30 +211,41 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:
owner: root
content: |
#!/usr/bin/env bash
exec > >(tee /var/log/tower-forge.log|logger -t TowerForge -s 2>/dev/console) 2>&1
##
yum install -q -y jq sed wget unzip nvme-cli lvm2
wget -q https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
## install CloudWatch agent
curl -s https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm -o amazon-cloudwatch-agent.rpm
rpm -U ./amazon-cloudwatch-agent.rpm
rm -f ./amazon-cloudwatch-agent.rpm
curl -s https://nf-xpack.seqera.io/amazon-cloudwatch-agent/custom-v0.1.json \
# | sed 's/custom-id/<your custom ID>/g' \
curl -s https://nf-xpack.seqera.io/amazon-cloudwatch-agent/config-v0.4.json \
# | sed 's/$FORGE_ID/<your custom ID>/g' \
> /opt/aws/amazon-cloudwatch-agent/bin/config.json
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
-a fetch-config \
-m ec2 \
-s \
-c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
## ECS configuration
mkdir -p /etc/ecs
echo ECS_IMAGE_PULL_BEHAVIOR=once >> /etc/ecs/ecs.config
echo ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true >> /etc/ecs/ecs.config
echo ECS_ENABLE_SPOT_INSTANCE_DRAINING=true >> /etc/ecs/ecs.config
echo ECS_CONTAINER_CREATE_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_CONTAINER_START_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_CONTAINER_STOP_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_MANIFEST_PULL_TIMEOUT=10m >> /etc/ecs/ecs.config
## stop docker
systemctl stop docker
## install AWS CLI v2
mkdir -p /home/ec2-user
curl -s https://nf-xpack.seqera.io/miniconda-awscli/miniconda-awscli.tar.gz \
| tar xz -C /home/ec2-user
## install AWS CLI
curl -s https://nf-xpack.seqera.io/miniconda-awscli/miniconda-25.3.1-awscli-1.40.12.tar.gz \
| tar xz -C /
export PATH=$PATH:/home/ec2-user/miniconda/bin
ln -s /home/ec2-user/miniconda/bin/aws /usr/bin/aws
## restart docker
systemctl start docker
systemctl enable --now --no-block ecs
## kernel settings to prevent OOM
echo "1258291200" > /proc/sys/vm/dirty_bytes
echo "629145600" > /proc/sys/vm/dirty_background_bytes

Expand All @@ -232,7 +255,7 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:
--//--
```

1. To prepend a custom identifier to the CloudWatch log streams for AWS resources created by your manual compute environment, uncomment the `| sed 's/custom-id/<your custom ID>/g' \` line and replace `<your custom ID>` with your custom ID. If ommitted, this defaults to `custom-id`.
1. To prepend a custom identifier to the CloudWatch log streams for AWS resources created by your manual compute environment, uncomment the `| sed 's/$FORGE_ID/<your custom ID>/g' \` line and replace `<your custom ID>` with your custom identifier. If omitted, `$FORGE_ID` remains as-is in the config.
1. Save the template with the name `seqera-launchtemplate`.

</TabItem>
Expand Down
63 changes: 43 additions & 20 deletions platform-enterprise_docs/enterprise/advanced-topics/manual-aws-batch-setup.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,18 +163,22 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:
owner: root
content: |
#!/usr/bin/env bash
exec > >(tee /var/log/tower-forge.log|logger -t TowerForge -s 2>/dev/console) 2>&1
##
yum install -q -y jq sed wget unzip nvme-cli lvm2
wget -q https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
## install CloudWatch agent
curl -s https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm -o amazon-cloudwatch-agent.rpm
rpm -U ./amazon-cloudwatch-agent.rpm
rm -f ./amazon-cloudwatch-agent.rpm
curl -s https://nf-xpack.seqera.io/amazon-cloudwatch-agent/custom-v0.1.json \
# | sed 's/custom-id/<your custom ID>/g' \
curl -s https://nf-xpack.seqera.io/amazon-cloudwatch-agent/config-v0.4.json \
# | sed 's/$FORGE_ID/<your custom ID>/g' \
> /opt/aws/amazon-cloudwatch-agent/bin/config.json
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
-a fetch-config \
-m ec2 \
-s \
-c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
-c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
## format NVMe storage for Fusion
mkdir -p /scratch/fusion
NVME_DISKS=($(nvme list | grep 'Amazon EC2 NVMe Instance Storage' | awk '{ print $1 }'))
NUM_DISKS=${#NVME_DISKS[@]}
Expand All @@ -191,18 +195,26 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:
fi
fi
chmod a+w /scratch/fusion
## ECS configuration
mkdir -p /etc/ecs
echo ECS_IMAGE_PULL_BEHAVIOR=once >> /etc/ecs/ecs.config
echo ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true >> /etc/ecs/ecs.config
echo ECS_ENABLE_SPOT_INSTANCE_DRAINING=true >> /etc/ecs/ecs.config
echo ECS_CONTAINER_CREATE_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_CONTAINER_START_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_CONTAINER_STOP_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_MANIFEST_PULL_TIMEOUT=10m >> /etc/ecs/ecs.config
## stop docker
systemctl stop docker
## install AWS CLI
mkdir -p /home/ec2-user
curl -s https://nf-xpack.seqera.io/miniconda-awscli/miniconda-awscli.tar.gz \
| tar xz -C /home/ec2-user
curl -s https://nf-xpack.seqera.io/miniconda-awscli/miniconda-25.3.1-awscli-1.40.12.tar.gz \
| tar xz -C /
export PATH=$PATH:/home/ec2-user/miniconda/bin
ln -s /home/ec2-user/miniconda/bin/aws /usr/bin/aws
## restart docker
systemctl start docker
systemctl enable --now --no-block ecs
## kernel settings to prevent OOM
echo "1258291200" > /proc/sys/vm/dirty_bytes
echo "629145600" > /proc/sys/vm/dirty_background_bytes

Expand All @@ -212,7 +224,7 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:
--//--
```

1. To prepend a custom identifier to the CloudWatch log streams for AWS resources created by your manual compute environment, uncomment the `| sed 's/custom-id/<your custom ID>/g' \` line and replace `<your custom ID>` with your custom ID. If ommitted, this defaults to `custom-id`.
1. To prepend a custom identifier to the CloudWatch log streams for AWS resources created by your manual compute environment, uncomment the `| sed 's/$FORGE_ID/<your custom ID>/g' \` line and replace `<your custom ID>` with your custom identifier. If omitted, `$FORGE_ID` remains as-is in the config.
1. Save the template with the name `seqera-launchtemplate`.

</TabItem>
Expand All @@ -235,30 +247,41 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:
owner: root
content: |
#!/usr/bin/env bash
exec > >(tee /var/log/tower-forge.log|logger -t TowerForge -s 2>/dev/console) 2>&1
##
yum install -q -y jq sed wget unzip nvme-cli lvm2
wget -q https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
## install CloudWatch agent
curl -s https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm -o amazon-cloudwatch-agent.rpm
rpm -U ./amazon-cloudwatch-agent.rpm
rm -f ./amazon-cloudwatch-agent.rpm
curl -s https://nf-xpack.seqera.io/amazon-cloudwatch-agent/custom-v0.1.json \
# | sed 's/custom-id/<your custom ID>/g' \
curl -s https://nf-xpack.seqera.io/amazon-cloudwatch-agent/config-v0.4.json \
# | sed 's/$FORGE_ID/<your custom ID>/g' \
> /opt/aws/amazon-cloudwatch-agent/bin/config.json
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
-a fetch-config \
-m ec2 \
-s \
-c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
## ECS configuration
mkdir -p /etc/ecs
echo ECS_IMAGE_PULL_BEHAVIOR=once >> /etc/ecs/ecs.config
echo ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true >> /etc/ecs/ecs.config
echo ECS_ENABLE_SPOT_INSTANCE_DRAINING=true >> /etc/ecs/ecs.config
echo ECS_CONTAINER_CREATE_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_CONTAINER_START_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_CONTAINER_STOP_TIMEOUT=10m >> /etc/ecs/ecs.config
echo ECS_MANIFEST_PULL_TIMEOUT=10m >> /etc/ecs/ecs.config
## stop docker
systemctl stop docker
## install AWS CLI v2
mkdir -p /home/ec2-user
curl -s https://nf-xpack.seqera.io/miniconda-awscli/miniconda-awscli.tar.gz \
| tar xz -C /home/ec2-user
## install AWS CLI
curl -s https://nf-xpack.seqera.io/miniconda-awscli/miniconda-25.3.1-awscli-1.40.12.tar.gz \
| tar xz -C /
export PATH=$PATH:/home/ec2-user/miniconda/bin
ln -s /home/ec2-user/miniconda/bin/aws /usr/bin/aws
## restart docker
systemctl start docker
systemctl enable --now --no-block ecs
## kernel settings to prevent OOM
echo "1258291200" > /proc/sys/vm/dirty_bytes
echo "629145600" > /proc/sys/vm/dirty_background_bytes

Expand All @@ -268,15 +291,15 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:
--//--
```

1. To prepend a custom identifier to the CloudWatch log streams for AWS resources created by your manual compute environment, uncomment the `| sed 's/custom-id/<your custom ID>/g' \` line and replace `<your custom ID>` with your custom ID. If ommitted, this defaults to `custom-id`.
1. To prepend a custom identifier to the CloudWatch log streams for AWS resources created by your manual compute environment, uncomment the `| sed 's/$FORGE_ID/<your custom ID>/g' \` line and replace `<your custom ID>` with your custom identifier. If omitted, `$FORGE_ID` remains as-is in the config.
1. Save the template with the name `seqera-launchtemplate`.

</TabItem>
</Tabs>

### Create the Batch compute environments

:::caution
:::caution
AWS Graviton instances (ARM64 CPU architecture) are not supported in manual compute environments. To use Graviton instances, create your AWS Batch compute environment with [Batch Forge](../../compute-envs/aws-batch#automatic-configuration-of-batch-resources).
:::

Expand Down Expand Up @@ -310,7 +333,7 @@ The head queue requires an on-demand compute environment. Do not select **Use Sp
:::
1. Expand **Additional configuration** and select the `seqera-launchtemplate` from the Launch template dropdown.
1. Configure VPCs, subnets, and security groups on the next page as needed.
1. Review your configuration and select **Create compute environment**.
1. Review your configuration and select **Create compute environment**.

</TabItem>
<TabItem value="Compute queue with Spot instances" label="Compute queue with Spot instances" default>
Expand All @@ -320,11 +343,11 @@ Create this compute environment to use Spot instances for your workflow compute
1. In the [Batch Console](https://eu-west-1.console.aws.amazon.com/batch/home), select **Create** on the Compute environments page.
1. Select **Amazon EC2** as the compute environment configuration.
1. Enter a name of your choice, and apply the `seqera-servicerole` and `seqera-instancerole`.
1. Select **Enable using Spot instances** to use Spot instances and save computing costs.
1. Select **Enable using Spot instances** to use Spot instances and save computing costs.
1. Select the `seqera-fleetrole` and enter vCPU limits and instance types, if needed.
1. Expand **Additional configuration** and select the `seqera-launchtemplate` from the Launch template dropdown.
1. Configure VPCs, subnets, and security groups on the next page as needed.
1. Review your configuration and select **Create compute environment**.
1. Review your configuration and select **Create compute environment**.

</TabItem>
</Tabs>
Expand Down
Loading