Skip to content

Commit 9ee0fa1

Browse files
committed
refactor Atlas baseline to use fresh hosted Supabase schema
1 parent b2c24c6 commit 9ee0fa1

19 files changed

+960
-102
lines changed

nx.json

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -103,10 +103,6 @@
103103
"local": true,
104104
"cache": true
105105
},
106-
"dump-realtime-schema": {
107-
"local": true,
108-
"cache": true
109-
},
110106
"gen-types": {
111107
"local": true,
112108
"cache": true

pkgs/core/.gitignore

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1 @@
11
atlas/schema.sql
2-
atlas/.supabase-baseline-schema.sql

pkgs/core/ATLAS.md

Lines changed: 0 additions & 32 deletions
This file was deleted.

pkgs/core/README.md

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,6 @@ PostgreSQL-native workflow engine for defining, managing, and tracking DAG-based
55
> [!NOTE]
66
> This project and all its components are licensed under [Apache 2.0](./LICENSE) license.
77
8-
> [!WARNING]
9-
> This project uses [Atlas](https://atlasgo.io/docs) to manage the schemas and migrations.
10-
> See [ATLAS.md](ATLAS.md) for more details.
11-
128
## Table of Contents
139

1410
- [Overview](#overview)
@@ -49,8 +45,7 @@ The actual execution of workflow tasks is handled by the [Edge Worker](../edge-w
4945

5046
## Requirements
5147

52-
> [!IMPORTANT]
53-
> **pgmq Version Requirement** (since v0.8.0)
48+
> [!IMPORTANT] > **pgmq Version Requirement** (since v0.8.0)
5449
>
5550
> pgflow v0.8.0 and later requires **pgmq 1.5.0 or higher**. This version of pgflow will NOT work with pgmq 1.4.x or earlier.
5651
>

pkgs/core/atlas/.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Temp Supabase project for baseline dumping
2+
supabase/

pkgs/core/atlas/Dockerfile

Lines changed: 55 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
# Use official PostgreSQL 17
1+
# Atlas Postgres image with extensions for pgflow
2+
#
3+
# Extension versions aligned with Supabase Postgres 17.6.1.054
4+
# Reference: https://github.com/supabase/postgres/blob/17.6.1.054/Dockerfile-17
5+
26
FROM postgres:17
37

48
# Set environment variables for postgres
@@ -10,22 +14,65 @@ RUN apt-get update && \
1014
apt-get install -y \
1115
build-essential \
1216
git \
13-
postgresql-server-dev-17
17+
postgresql-server-dev-17 \
18+
libcurl4-openssl-dev \
19+
libsodium-dev \
20+
libicu-dev \
21+
pkg-config
1422

15-
# Clone and install pgmq
23+
# Clone and install pgmq v1.5.1 (not in Supabase image, pgflow dependency)
1624
RUN mkdir -p /usr/share/postgresql/17/extension && \
1725
git clone https://github.com/tembo-io/pgmq.git /tmp/pgmq && \
1826
cd /tmp/pgmq/pgmq-extension && \
1927
git checkout v1.5.1 && \
20-
# Copy extension files manually to PostgreSQL 17 extensions directory
2128
make && \
2229
cp pgmq.control /usr/share/postgresql/17/extension/ && \
23-
cp sql/pgmq--*.sql /usr/share/postgresql/17/extension/
30+
cp sql/pgmq--*.sql /usr/share/postgresql/17/extension/ && \
31+
rm -rf /tmp/pgmq
32+
33+
# Install pg_cron v1.6.4 (v1.6.3+ required for PG17 compatibility)
34+
RUN git clone https://github.com/citusdata/pg_cron.git /tmp/pg_cron && \
35+
cd /tmp/pg_cron && \
36+
git checkout v1.6.4 && \
37+
make && \
38+
make install && \
39+
rm -rf /tmp/pg_cron
40+
41+
# Install pg_net v0.20.2 (PG17 compatible, requires libcurl)
42+
RUN git clone https://github.com/supabase/pg_net.git /tmp/pg_net && \
43+
cd /tmp/pg_net && \
44+
git checkout v0.20.2 && \
45+
make && \
46+
make install && \
47+
rm -rf /tmp/pg_net
2448

25-
# Clean up
26-
RUN apt-get remove -y build-essential git postgresql-server-dev-17 && \
49+
# Install pgsodium v3.1.6 (requires libsodium)
50+
RUN git clone https://github.com/michelp/pgsodium.git /tmp/pgsodium && \
51+
cd /tmp/pgsodium && \
52+
git checkout v3.1.6 && \
53+
make && \
54+
make install && \
55+
rm -rf /tmp/pgsodium
56+
57+
# Install supabase_vault v0.2.8 (depends on pgsodium)
58+
RUN git clone https://github.com/supabase/vault.git /tmp/vault && \
59+
cd /tmp/vault && \
60+
git checkout v0.2.8 && \
61+
make && \
62+
make install && \
63+
rm -rf /tmp/vault
64+
65+
# Clean up build dependencies (keep runtime libs: libcurl4t64, libsodium23, libicu)
66+
RUN apt-get remove -y build-essential git postgresql-server-dev-17 \
67+
libcurl4-openssl-dev libsodium-dev libicu-dev pkg-config && \
2768
apt-get autoremove -y && \
28-
rm -rf /var/lib/apt/lists/* /tmp/pgmq
69+
apt-get update && \
70+
apt-get install -y --no-install-recommends libcurl4t64 libsodium23 && \
71+
rm -rf /var/lib/apt/lists/*
72+
73+
# Configure shared_preload_libraries for extensions that require it
74+
# pg_cron and pg_net need to be preloaded (pgsodium NOT needed for Atlas dev)
75+
RUN echo "shared_preload_libraries = 'pg_cron,pg_net'" >> /usr/share/postgresql/postgresql.conf.sample
2976

3077
# We need to ensure we don't create any init scripts that will run on startup
3178
# We must also remove any existing init scripts to ensure a clean database for Atlas

pkgs/core/atlas/README.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Atlas Migration Management
2+
3+
This directory contains configuration for generating pgflow database migrations using [Atlas](https://atlasgo.io/).
4+
5+
## Files
6+
7+
- `atlas.hcl` - Atlas configuration
8+
- `Dockerfile` - Custom Postgres image with required extensions
9+
- `supabase-baseline-schema.sql` - Schema baseline representing a fresh hosted Supabase project
10+
- `fresh-extensions.txt` - Record of extensions in fresh Supabase (for reference)
11+
- `dump-fresh-baseline.sh` - Script to regenerate baseline
12+
13+
## Docker Image
14+
15+
The `Dockerfile` builds a custom Postgres 17 image with extensions matching hosted Supabase.
16+
17+
**Reference:** [Supabase Postgres 17.6.1.054 Dockerfile](https://github.com/supabase/postgres/blob/17.6.1.054/Dockerfile-17)
18+
19+
### Installed Extensions
20+
21+
| Extension | Git Tag | Ext Version | Notes |
22+
|-----------|---------|-------------|-------|
23+
| pgmq | v1.5.1 | 1.5.1 | pgflow dependency (not in Supabase image) |
24+
| pg_cron | v1.6.4 | 1.6 | Scheduled jobs (v1.6.3+ for PG17) |
25+
| pg_net | v0.7.1 | 0.20.2 | HTTP requests from SQL |
26+
| pgsodium | v3.1.6 | 3.1.9 | Cryptography primitives |
27+
| supabase_vault | v0.2.8 | 0.3.1 | Secrets management (depends on pgsodium) |
28+
29+
### Building the Image
30+
31+
```bash
32+
cd pkgs/core
33+
./scripts/build-atlas-postgres-image
34+
./scripts/push-atlas-postgres-image
35+
```
36+
37+
## Baseline Schema
38+
39+
The baseline represents what a **fresh hosted Supabase project** looks like before pgflow migrations.
40+
41+
### Pre-installed Extensions (hosted Supabase)
42+
43+
| Extension | Status |
44+
|-----------|--------|
45+
| `supabase_vault` | Pre-installed |
46+
| `pgmq` | Available, not installed |
47+
| `pg_cron` | Available, not installed |
48+
| `pg_net` | Available, not installed |
49+
50+
Our migrations must `CREATE EXTENSION` for pgmq, pg_cron, and pg_net.
51+
52+
### Local CLI Difference
53+
54+
The local Supabase CLI pre-installs `pg_net`, but hosted Supabase does not. The `dump-fresh-baseline.sh` script accounts for this by dropping `pg_net` before dumping to match hosted behavior.
55+
56+
## Regenerating the Baseline
57+
58+
Run this when:
59+
- Supabase CLI version changes significantly
60+
- Adding new Supabase-provided schemas to baseline
61+
62+
```bash
63+
cd pkgs/core/atlas
64+
./dump-fresh-baseline.sh
65+
```
66+
67+
The script will:
68+
1. Create a fresh Supabase project in `supabase/` (temp, gitignored)
69+
2. Drop `pg_net` to match hosted Supabase defaults
70+
3. Verify extension assumptions match hosted Supabase
71+
4. Dump schema and extension list
72+
5. Clean up temp project
73+
74+
## Generating Migrations
75+
76+
After updating SQL in `supabase/migrations/`, run from `pkgs/core/`:
77+
78+
```bash
79+
pnpm nx atlas:migrate:diff core --name="description_of_change"
80+
```
81+
82+
This compares the current schema against the baseline + existing migrations.

pkgs/core/atlas/atlas.hcl

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,10 @@ docker "postgres" "pgflow" {
1515
# image = "postgres:17"
1616
# custom image is built and pushed to speed up schema verification,
1717
# otherwise it takes around 30s
18-
image = "jumski/postgres-17-pgmq:latest"
19-
baseline = file(".supabase-baseline-schema.sql")
18+
# contains: pgmq, pg_cron, pg_net, pgsodium, supabase_vault
19+
# tag matches Supabase Postgres version we align extensions with
20+
image = "jumski/atlas-postgres-pgflow:17.6.1.054"
21+
baseline = file("supabase-baseline-schema.sql")
2022
build {
2123
dockerfile = "atlas/Dockerfile"
2224
context = "."
Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
#!/bin/bash
2+
# Dump baseline schema from a fresh Supabase instance
3+
#
4+
# This script:
5+
# 1. Creates a fresh Supabase project (no pgflow)
6+
# 2. Verifies extension availability matches our assumptions
7+
# 3. Dumps the schema as our baseline
8+
# 4. Cleans up
9+
#
10+
# Run this when:
11+
# - Supabase CLI version changes
12+
# - Adding new Supabase-provided schemas to baseline
13+
14+
set -euo pipefail
15+
16+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
17+
cd "$SCRIPT_DIR"
18+
19+
# Use npx to run supabase directly in this directory (not via pnpm which runs from pkgs/core)
20+
SUPA="npx supabase"
21+
22+
echo "=== Cleaning up any previous temp project ==="
23+
rm -rf supabase/ || true
24+
25+
echo "=== Initializing fresh Supabase project ==="
26+
$SUPA init --force
27+
28+
echo "=== Stopping any running Supabase instances ==="
29+
$SUPA stop --all || true
30+
31+
echo "=== Starting fresh Supabase ==="
32+
$SUPA start
33+
34+
echo "=== Getting database URL ==="
35+
DB_URL=$($SUPA status --output json | jq -r '.DB_URL')
36+
echo "DB_URL: $DB_URL"
37+
38+
echo "=== Matching hosted Supabase defaults ==="
39+
# Local CLI pre-installs pg_net, but hosted Supabase doesn't
40+
# Drop it to match hosted behavior (our migrations will create it)
41+
echo "Dropping pg_net to match hosted Supabase defaults..."
42+
psql "$DB_URL" -c "DROP EXTENSION IF EXISTS pg_net CASCADE;" 2>/dev/null || true
43+
44+
echo "=== Verifying extension assumptions ==="
45+
# Query extension status (using --no-psqlrc to avoid extra output)
46+
EXTENSION_STATUS=$(PAGER='' psql "$DB_URL" --no-psqlrc -t -A -F'|' -c "
47+
SELECT
48+
name,
49+
CASE WHEN installed_version IS NOT NULL THEN 'installed' ELSE 'available' END as status
50+
FROM pg_available_extensions
51+
WHERE name IN ('pgmq', 'pg_cron', 'pg_net', 'supabase_vault')
52+
ORDER BY name;
53+
")
54+
55+
echo "Extension status:"
56+
echo "$EXTENSION_STATUS"
57+
58+
# Expected state matching HOSTED Supabase (not local CLI)
59+
# - supabase_vault: pre-installed
60+
# - pgmq, pg_cron, pg_net: available but NOT installed
61+
EXPECTED="pg_cron|available
62+
pg_net|available
63+
pgmq|available
64+
supabase_vault|installed"
65+
66+
if [ "$EXTENSION_STATUS" != "$EXPECTED" ]; then
67+
echo ""
68+
echo "ERROR: Extension status does not match hosted Supabase assumptions!"
69+
echo ""
70+
echo "Expected (matching hosted Supabase):"
71+
echo "$EXPECTED"
72+
echo ""
73+
echo "Got:"
74+
echo "$EXTENSION_STATUS"
75+
echo ""
76+
echo "If hosted Supabase changed defaults, update this script's expectations."
77+
$SUPA stop
78+
rm -rf supabase/
79+
exit 1
80+
fi
81+
82+
echo ""
83+
echo "Extension assumptions verified (matches hosted Supabase)!"
84+
85+
echo "=== Dumping full \\dx output ==="
86+
psql "$DB_URL" -c '\dx' > fresh-extensions.txt
87+
echo "Saved to fresh-extensions.txt"
88+
89+
echo "=== Dumping baseline schema ==="
90+
# Only dump schemas we need:
91+
# - realtime: we reference realtime.send()
92+
# - vault: supabase_vault is pre-installed here
93+
# Skip extensions schema - it contains Supabase internal event triggers
94+
# that reference roles not available in Atlas dev container
95+
atlas schema inspect \
96+
--schema realtime,vault \
97+
-u "$DB_URL?sslmode=disable" \
98+
--format "{{ sql . }}" > supabase-baseline-schema.sql
99+
100+
# Strip VERSION strings (they change between Supabase versions)
101+
sed -i 's/ VERSION "[^"]*"//g' supabase-baseline-schema.sql
102+
103+
# Strip date-specific partitions (they change daily, we don't reference them)
104+
# e.g., messages_2025_12_04, messages_2025_12_05, etc.
105+
sed -i '/messages_20[0-9]\{2\}_[0-9]\{2\}_[0-9]\{2\}/d' supabase-baseline-schema.sql
106+
107+
echo "Saved to supabase-baseline-schema.sql"
108+
109+
echo "=== Stopping Supabase ==="
110+
$SUPA stop
111+
112+
echo "=== Cleaning up temp project ==="
113+
rm -rf supabase/
114+
115+
echo ""
116+
echo "=== Done! ==="
117+
echo "Baseline regenerated from fresh Supabase."
118+
echo "Commit the updated supabase-baseline-schema.sql and fresh-extensions.txt"
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
Null display is "[NULL]".
2+
Expanded display is used automatically.
3+
List of installed extensions
4+
Name | Version | Schema | Description
5+
--------------------+---------+------------+------------------------------------------------------------------------
6+
pg_graphql | 1.5.11 | graphql | pg_graphql: GraphQL support
7+
pg_stat_statements | 1.11 | extensions | track planning and execution statistics of all SQL statements executed
8+
pgcrypto | 1.3 | extensions | cryptographic functions
9+
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
10+
supabase_vault | 0.3.1 | vault | Supabase Vault Extension
11+
uuid-ossp | 1.1 | extensions | generate universally unique identifiers (UUIDs)
12+
(6 rows)
13+

0 commit comments

Comments
 (0)