ZVPG: ZFS Versioned PostgreSQL Engine

Empowering PostgreSQL with Git-like branching and snapshotting capabilities, powered by the rock-solid ZFS file system.

ZVPG (ZFS Versioned PostgreSQL Engine) revolutionizes database development and testing workflows. It leverages the native power of ZFS snapshots and branches to create instantaneous, space-efficient, and fully independent copies of your PostgreSQL databases. Think of it as git for your database, allowing you to create branches for features, test schemas, or run isolated experiments without impacting your primary instance or consuming significant disk space.

New in v2.0: ZVPG now uses a modern container-based architecture! Instead of managing PostgreSQL instances directly, ZVPG leverages container runtimes (Docker, Podman, or nerdctl) to provide better isolation, easier management, and more flexible configuration. Each branch runs in its own PostgreSQL container with custom configuration files stored in your ~/.zvpg/ directory.

Key Features

Instantaneous Snapshots: Create atomic, read-only snapshots of your entire PostgreSQL data directory in milliseconds.
Zero-Cost Branches: Spin up multiple, fully-functional, writeable PostgreSQL instances from any snapshot, instantly. Branches are copy-on-write, meaning they only store the differences, saving immense disk space.
Git-like Branching: Manage different database states with a familiar branching model. Create a branch, make schema changes, test, and then merge or discard your work.
Container-Based Isolation: Every branch runs as an independent PostgreSQL container with its own port, ensuring complete isolation between development, testing, and production environments.
Flexible Container Runtime: Support for multiple container runtimes including Docker, Podman, and nerdctl.
User-Managed Configuration: PostgreSQL configuration files (postgresql.conf, pg_hba.conf) are stored in ~/.zvpg/ directory for easy customization.
Comprehensive CLI: A powerful and intuitive command-line interface with subcommands for all operations:
- zvpg init - Environment initialization and health checking
- zvpg snapshot / zvpg commit - Snapshot/commit management
- zvpg branch - Git-like branch operations with PostgreSQL container management
- zvpg status - System monitoring and health checks
Flexible Configuration: JSON-based configuration with sensible defaults and extensive customization options.

Architecture

ZVPG acts as a control plane, orchestrating ZFS and containerized PostgreSQL instances to provide its versioning capabilities. The new container-based architecture provides better isolation and easier management:

graph TD
    subgraph User
        A[CLI: zvpg]
        U[~/.zvpg/config.json]
        C[~/.zvpg/postgresql.conf]
        H[~/.zvpg/pg_hba.conf]
    end

    subgraph ZVPG Engine
        B[Command Parser]
        S[Service: Snapshot, Branch]
    end

    subgraph System
        D[ZFS Filesystem]
        R[Container Runtime]
        P[PostgreSQL Containers]
    end

    A -- Manages --> B
    A -- Configures --> U
    A -- Customizes --> C
    A -- Customizes --> H
    B -- Executes --> S
    S -- zfs snapshot --> D
    S -- zfs branch --> D
    S -- container run --> R
    R -- creates --> P
    D <-- mounts data --> P
    U -- config --> S
    C <-- postgresql.conf --> P
    H <-- pg_hba.conf --> P

The user interacts with the zvpg CLI.
The command is parsed and passed to the appropriate Service (e.g., SnapshotService, BranchService).
The service layer executes low-level ZFS commands to create snapshots or branches of the main data volume.
When a branch with PostgreSQL is started, the service uses the container runtime to start a new PostgreSQL container, mounting the branch data directory and using custom configuration from ~/.zvpg/.
Each PostgreSQL container runs on a unique port for complete isolation.

Container Workflow

The container-based architecture provides several advantages:

sequenceDiagram
    participant User
    participant ZVPG
    participant ZFS
    participant Runtime as Container Runtime
    participant Container as PostgreSQL Container

    User->>ZVPG: zvpg branch create feature/auth --port 6001
    ZVPG->>ZFS: zfs clone snapshot feature/auth
    ZVPG->>Runtime: docker run -d postgres:17
    Note right of Runtime: Mount ZFS branch data
    Note right of Runtime: Use ~/.zvpg/postgresql.conf
    Note right of Runtime: Expose port 6001:5432
    Runtime->>Container: Start PostgreSQL
    Container-->>ZVPG: Container ready
    ZVPG-->>User: Branch created with container
    
    User->>ZVPG: zvpg branch stop feature/auth
    ZVPG->>Runtime: docker stop zvpg_feature_auth_6001
    Runtime->>Container: Stop container
    ZVPG->>Runtime: docker rm zvpg_feature_auth_6001
    ZVPG-->>User: Container stopped and removed

Getting Started

Prerequisites

Before you begin, ensure you have the following dependencies installed and configured on your system:

ZFS: The ZFS filesystem must be installed and a storage pool created.
Container Runtime: One of the supported container runtimes (Docker, Podman, or nerdctl) must be installed and accessible.
PostgreSQL Client Tools: The PostgreSQL client tools (psql, pg_isready) must be in your PATH for health checks.
Deno: ZVPG is built on the Deno runtime.

ZVPG Configuration Reference

This document describes all configuration options available in ZVPG v2.0.

Configuration File Location

By default, ZVPG looks for configuration in ~/.zvpg/config.json. You can override this with the --config flag:

zvpg --config /path/to/config.json branch create test

Configuration Options

ZFS Settings

Option	Type	Default	Description
`zfsPool`	string	`"zvpg_pool"`	Name of the ZFS pool to use
`mountDir`	string	`"/var/lib/zvpg"`	Base mount directory for ZFS datasets
`dataSubdir`	string	`"data"`	Subdirectory for main data
`clonesSubdir`	string	`"clones"`	Subdirectory for clones
`socketSubdir`	string	`"sockets"`	Subdirectory for sockets

PostgreSQL Settings

Option	Type	Default	Description
`postgresUser`	string	`"postgres"`	Default PostgreSQL user
`postgresDb`	string	`"postgres"`	Default PostgreSQL database

Container Settings

Option	Type	Default	Description
`containerRuntime`	string	`"docker"`	Container runtime to use (`docker`, `podman`, `nerdctl`)
`pgBaseImage`	string	`"postgres:17"`	PostgreSQL container image
`pgConfigDir`	string	`"~/.zvpg"`	Directory for PostgreSQL configuration files
`pgHbaPath`	string	`"~/.zvpg/pg_hba.conf"`	Path to pg_hba.conf file
`pgIdentPath`	string	`"~/.zvpg/pg_ident.conf"`	Path to pg_ident.conf file
`pgConfPath`	string	`"~/.zvpg/postgresql.conf"`	Path to postgresql.conf file

Network Settings

Option	Type	Default	Description
`branchPortStart`	number	`6001`	Starting port for PostgreSQL containers
`branchPortEnd`	number	`6099`	Ending port for PostgreSQL containers

Branch Settings

Option	Type	Default	Description
`branchDefault`	string	`"main"`	Default branch name
`branchNamingPattern`	string	`FOLLOWS GIT`	Regex pattern for branch names

git is using ^(?!\/|\.|.*([\/.]\.|\/\/|\.lock$))[\p{L}\p{N}\-_\/]+$ for parsing branch name

Logging Settings

Option	Type	Default	Description
`logLevel`	string	`"INFO"`	Log level (`DEBUG`, `INFO`, `WARN`, `ERROR`)
`logDir`	string	`"/var/log/zvpg"`	Directory for log files

Example Configurations

Docker with Custom Image

{
  "containerRuntime": "docker",
  "pgBaseImage": "postgres:16-alpine",
  "pgConfigDir": "~/.zvpg",
  "branchPortStart": 7001,
  "branchPortEnd": 7099
}

Podman with Rootless Containers

{
  "containerRuntime": "podman",
  "pgBaseImage": "docker.io/postgres:17",
  "pgConfigDir": "~/.config/zvpg",
}

nerdctl with Custom Pool

{
  "containerRuntime": "nerdctl",
  "zfsPool": "nvme_pool",
  "pgBaseImage": "postgres:17",
  "mountDir": "/mnt/nvme/zvpg"
}

PostgreSQL Configuration Files

postgresql.conf

Located at ~/.zvpg/postgresql.conf by default. This file is mounted into each PostgreSQL container. Key settings:

listen_addresses = '*' - Allow connections from all interfaces
port = 5432 - Internal container port (mapped to host port)
shared_buffers - Memory allocation for PostgreSQL
log_* - Logging configuration

pg_hba.conf

Located at ~/.zvpg/pg_hba.conf by default. Controls client authentication:

local all all trust - Trust local connections
host all all 127.0.0.1/32 trust - Trust localhost connections
host all all 0.0.0.0/0 md5 - Require password for remote connections

Environment Variables

ZVPG respects the following environment variables:

ZVPG_CONFIG_DIR - Override config directory (default: ~/.zvpg)
ZVPG_LOG_LEVEL - Override log level
ZVPG_CONTAINER_RUNTIME - Override container runtime

Security Considerations

Network Security: Configure pg_hba.conf appropriately for your network
Container Security: Use specific image tags instead of latest
File Permissions: Ensure config files have appropriate permissions
Firewall: Configure firewall rules for the port range
Container Images: Use trusted PostgreSQL images
Secrets: Avoid storing passwords in configuration files

Performance Tuning

ZFS Tuning

Use compression=lz4 for better performance
Set atime=off to reduce I/O
Use dedicated storage for ZFS pools

Container Tuning

Set appropriate shared_buffers in postgresql.conf
Use local storage for container runtime
Limit container memory usage if needed

Network Tuning

Use smaller port ranges for fewer containers
Configure appropriate timeouts
Use localhost connections when possible

PostgreSQL Configuration Templates

ZVPG includes template configuration files that you can customize:

Edit the configuration files to match your requirements:

~/.zvpg/postgresql.conf: Main PostgreSQL configuration
~/.zvpg/pg_hba.conf: Client authentication configuration
~/.zvpg/config.json: ZVPG system configuration

Usage

ZVPG provides a simple yet powerful set of commands to manage your database versions.

Snapshots

zvpg commit is an alias of zvpg snapshot

Snapshots are the foundation of ZVPG. They are immutable, point-in-time images of your database.

Create a snapshot:

zvpg snapshot create my_first_snapshot -m "Initial database schema"

List all snapshots:
```
zvpg snapshot list
```
Get snapshot information:
```
zvpg snapshot info my_first_snapshot
```
Delete a snapshot:
```
zvpg snapshot delete my_first_snapshot
```

Commits

Commits are an alias for snapshots, providing a Git-like interface.

Create a commit:

zvpg commit create my_commit -m "Add user authentication"

List all commits:
```
zvpg commit list
```
Show commit information:
```
zvpg commit show my_commit
```
Remove a commit:
```
zvpg commit remove my_commit
```

Branches

Branches provide a comprehensive abstraction for managing development workflows, similar to Git. They use the dedicated ZFS-based branch system and can optionally run PostgreSQL instances.

Create a branch from current state (latest snapshot):
```
zvpg branch create feature/new-api
```

Create a branch from a specific snapshot:

zvpg branch create feature/new-api --from my_snapshot

Create a branch with a specific parent:

zvpg branch create feature/new-api --parent development

Create a branch with specific PostgreSQL port:

zvpg branch create feature/new-api --port 6433

Start PostgreSQL container for an existing branch:
```
zvpg branch start feature/new-api --port 6433
```
Stop PostgreSQL container for a branch:
```
zvpg branch stop feature/new-api
```
List branches:
```
zvpg branch list
```
Get branch information:
```
zvpg branch info feature/new-api
```

Connect to a branch's PostgreSQL container:

psql -h localhost -p 6433 -U postgres postgres

Create a snapshot from a branch:

zvpg branch snapshot feature/new-api milestone-1 -m "First milestone completed"

Delete a branch:
```
zvpg branch delete feature/new-api
```

Status

Get a comprehensive overview of the entire system.

Show complete system status:
```
zvpg status system
```
Show detailed branches status:
```
zvpg status branches
```
Show detailed snapshots status:
```
zvpg status snapshots
```
Perform system health check:
```
zvpg status health
```

Configuration

ZVPG is configured via a JSON file (usually located at ~/.zvpg/config.json). The init command will generate a default configuration file.

Key configuration options include:

zfsPool: The name of the ZFS pool to use.
mountDir: The base directory where ZFS datasets are mounted.
branchPortStart / branchPortEnd: The range of network ports to use for branch PostgreSQL instances.
logLevel: The logging verbosity.

Command Reference

Initialize Commands

zvpg init env [options]              # Initialize the ZVPG environment
zvpg init check [options]            # Check environment initialization status

Snapshot Commands

zvpg snapshot create <name> [options]    # Create a new snapshot
zvpg snapshot list [options]             # List all snapshots
zvpg snapshot info <name> [options]      # Show snapshot information
zvpg snapshot delete <name> [options]    # Delete a snapshot

Commit Commands (Snapshot Aliases)

zvpg commit create <name> [options]      # Create a new commit
zvpg commit list [options]               # List all commits
zvpg commit show <name> [options]        # Show commit information
zvpg commit remove <name> [options]      # Remove a commit

Branch Commands

zvpg branch create <name> [options]               # Create a new branch (PostgreSQL auto-started)
zvpg branch list [options]                        # List all branches
zvpg branch info <name> [options]                 # Show branch information
zvpg branch delete <name> [options]               # Delete a branch
zvpg branch start <name> [options]                # Start PostgreSQL instance for branch
zvpg branch stop <name> [options]                 # Stop PostgreSQL instance for branch
zvpg branch snapshot <branch> <snapshot> [options] # Create snapshot from branch

Status Commands

zvpg status system [options]             # Show complete system status
zvpg status branches [options]           # Show detailed branches status
zvpg status snapshots [options]          # Show detailed snapshots status
zvpg status health [options]             # Perform system health check

Common Options

-c, --config <config>: Configuration file path
-f, --format <format>: Output format (table|json) for list commands
-m, --message <message>: Message for snapshot/commit creation
-p, --port <port>: Port number for PostgreSQL instances
--from <snapshot>: Create branch from specific snapshot
--parent <parent>: Parent branch name (default: main)
--force: Force operations (where applicable)

Workflow Examples

Basic Development Workflow

Initialize the environment:
```
zvpg init env
```

Create a snapshot of your current database:

zvpg snapshot create baseline -m "Initial production state"

Create a development branch with PostgreSQL instance:
```
zvpg branch create feature/user-auth --from baseline
```

Work on your branch (PostgreSQL automatically started):

# Connect to the branch's PostgreSQL instance (port shown in creation output)
zvpg branch info feature/user-auth  # Check the port
psql -p <port> -U postgres -h localhost

Create a snapshot from your branch:

zvpg branch snapshot feature/user-auth milestone-1 -m "User authentication milestone"

Stop the PostgreSQL instance when done:
```
zvpg branch stop feature/user-auth
```
Clean up when no longer needed:
```
zvpg branch delete feature/user-auth
```

Testing Workflow

Create a testing branch:

zvpg branch create testing --from baseline

Run your tests against the branch:

# Get the automatically assigned port
PORT=$(zvpg branch info testing | grep "PostgreSQL Port" | awk '{print $3}')
# Your tests connect to the assigned port
npm test -- --db-port=$PORT

Reset the test environment:

zvpg branch stop testing
zvpg branch delete testing
zvpg branch create testing --from baseline

Monitoring and Maintenance

Check system status:
```
zvpg status system
```
Monitor branches:
```
zvpg branch list
```
Check branch details:
```
zvpg branch info my-branch
```
Check snapshots usage:
```
zvpg status snapshots
```
Perform health check:
```
zvpg status health
```

Troubleshooting

Container Runtime Issues

Container runtime not found:

# Check if your container runtime is installed
docker --version
podman --version
nerdctl --version

# Update configuration to use available runtime
vim ~/.zvpg/config.json

Port conflicts:

# Check which ports are in use
zvpg status

# Stop conflicting containers
docker ps
docker stop <container_name>

Permission issues:

# Ensure user can access container runtime
sudo usermod -aG docker $USER
newgrp docker

# For podman (rootless)
systemctl --user enable podman.socket

PostgreSQL Container Issues

Container fails to start:

# Check container logs
docker logs zvpg_<branch>_<port>

# Check if data directory is properly mounted
zvpg branch info <branch>

Cannot connect to PostgreSQL:

# Check if container is running
docker ps | grep zvpg

# Test connection
pg_isready -h localhost -p <port>

# Check firewall/network settings
telnet localhost <port>

Configuration issues:

# Check configuration files
cat ~/.zvpg/postgresql.conf
cat ~/.zvpg/pg_hba.conf

# Reset to defaults
rm ~/.zvpg/postgresql.conf ~/.zvpg/pg_hba.conf
zvpg branch stop <branch>
zvpg branch start <branch>

ZFS Issues

ZFS dataset not found:

# Check if ZFS pool exists
zfs list

# Recreate environment
zvpg init env

Snapshot cleanup:

# List all snapshots
zvpg snapshot list

# Clean up old snapshots
zvpg snapshot delete <snapshot_name>

Contributing

We welcome contributions from the community! Whether it's a bug report, a new feature, or a documentation improvement, please feel free to open an issue or submit a pull request.

Please read our CONTRIBUTING.md for detailed guidelines.

License

ZVPG is licensed under the Apache License 2.0.

Name		Name	Last commit message	Last commit date
Latest commit History 7 Commits
.vscode		.vscode
commands		commands
services		services
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
ZVPG.svg		ZVPG.svg
config.ts		config.ts
deno.json		deno.json
deno.lock		deno.lock
utils.ts		utils.ts
version.ts		version.ts
zvpg.ts		zvpg.ts

License

LemonHX/ZVPG

Folders and files

Latest commit

History

Repository files navigation

ZVPG: ZFS Versioned PostgreSQL Engine

Key Features

Architecture

Container Workflow

Getting Started

Prerequisites

ZVPG Configuration Reference

Configuration File Location

Configuration Options

ZFS Settings

PostgreSQL Settings

Container Settings

Network Settings

Branch Settings

Logging Settings

Example Configurations

Docker with Custom Image

Podman with Rootless Containers

nerdctl with Custom Pool

PostgreSQL Configuration Files

postgresql.conf

pg_hba.conf

Environment Variables

Security Considerations

Performance Tuning

ZFS Tuning

Container Tuning

Network Tuning

PostgreSQL Configuration Templates

Usage

Snapshots

Commits

Branches

Status

Configuration

Command Reference

Initialize Commands

Snapshot Commands

Commit Commands (Snapshot Aliases)

Branch Commands

Status Commands

Common Options

Workflow Examples

Basic Development Workflow

Testing Workflow

Monitoring and Maintenance

Troubleshooting

Container Runtime Issues

PostgreSQL Container Issues

ZFS Issues

Contributing

License

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages