Test reorg

[johngrantuk]

After various attempts to improve the test reliability have settled on the following:

• Add new scripts for tests
  • all: All tests
  • no-integration: No integration tests (these are usually culprit of flaky tests on CI). This still runs network validation test which is likely enough as a cover all test at this stage.
  • no-v2: No V2 tests.
• Updated CI
  • Checks workflow uses no-integration, should make for more reliable runs as only tests a small subset.
  • Added test-all workflow. Can be manually triggered to run.
  • Added test-no-v2 workflow. Can be manually triggered to run.
• Update GitHub Actions workflows for performance based off AI advice (see below for full detail)

Update GitHub Actions Workflows for Performance

Overview

Update all GitHub Actions workflows to use the latest action versions (v4), optimize caching strategies, remove redundant operations, and improve job parallelism to reduce CI/CD execution time.

Files to Update

1. .github/actions/setup/action.yml

Changes:

• Remove redundant actions/checkout@v3 step (workflows already checkout)
• Remove redundant ~/.pnpm-store cache (handled by setup-node with cache: 'pnpm')
• Update actions/setup-node@v3 → actions/setup-node@v4
• Update actions/cache@v3 → actions/cache@v4
• Add conditional install step (skip if node_modules cache hit)
• Add restore-keys to node_modules cache for better cache hit rate

Key improvements:

• Eliminates duplicate checkout operations
• Removes redundant pnpm store caching
• Conditional dependency installation saves time on cache hits

2. .github/workflows/checks.yml

Changes:

• Change runs-on: ubuntu-latest → runs-on: macos-latest in all jobs
• Remove install job (not effectively used, causes sequential execution)
• Update actions/checkout@v3 → actions/checkout@v4 in all jobs
• Update actions/cache@v3 → actions/cache@v4 (for Foundry caching)
• Update foundry-rs/foundry-toolchain@v1 → foundry-rs/foundry-toolchain@v1.4.2 (verify latest version)
• Add Foundry caching step in test job:
• Cache ~/.foundry directory before installing Foundry
• Use cache key based on Foundry version/toolchain action version
• Add restore-keys for partial cache matches
• Make code-quality, test, and build jobs run in parallel (remove needs: install)

Key improvements:

• Jobs run in parallel instead of sequentially
• Faster overall workflow execution
• Removes unnecessary job dependency
• Foundry binaries cached to avoid re-downloading on each run

3. .github/workflows/test-no-v2.yml

Changes:

• Change runs-on: ubuntu-latest → runs-on: macos-latest
• Update actions/checkout@v3 → actions/checkout@v4
• Update actions/cache@v3 → actions/cache@v4 (for Foundry caching)
• Update foundry-rs/foundry-toolchain@v1 → foundry-rs/foundry-toolchain@v1.4.2 (verify latest version)
• Add Foundry caching step:
• Cache ~/.foundry directory before installing Foundry
• Use cache key based on Foundry version/toolchain action version
• Add restore-keys for partial cache matches

Key improvements:

• Foundry binaries cached to avoid re-downloading on each run
• Faster test execution on subsequent runs

4. .github/workflows/test-all.yml

Changes:

• Change runs-on: ubuntu-latest → runs-on: macos-latest
• Update actions/checkout@v3 → actions/checkout@v4
• Update actions/cache@v3 → actions/cache@v4 (for Foundry caching)
• Update foundry-rs/foundry-toolchain@v1 → foundry-rs/foundry-toolchain@v1.4.2 (verify latest version)
• Add Foundry caching step:
• Cache ~/.foundry directory before installing Foundry
• Use cache key based on Foundry version/toolchain action version
• Add restore-keys for partial cache matches

Key improvements:

• Foundry binaries cached to avoid re-downloading on each run
• Faster test execution on subsequent runs

5. .github/workflows/release.yml

Changes:

• Change runs-on: ubuntu-latest → runs-on: macos-latest
• Update actions/checkout@v3 → actions/checkout@v4
• Update changesets/action@v1 → changesets/action@v1.4.0 (verify latest version)

Implementation Details

Setup Action Optimization

The setup action currently:

1. Checks out code (redundant - workflows do this)
2. Manually caches ~/.pnpm-store (redundant - setup-node handles this)
3. Caches node_modules (useful, but should be conditional)

After update:

• Only handles Node.js/pnpm setup and dependency installation
• Uses built-in pnpm caching from setup-node
• Conditionally installs dependencies based on cache hit

Workflow Parallelization

The checks.yml workflow currently runs:

1. install job (does nothing useful)
2. code-quality waits for install
3. test waits for install
4. build waits for install

After update:

• All three jobs (code-quality, test, build) run in parallel
• Each job handles its own setup independently
• Faster overall execution time

Foundry Caching Strategy

Foundry installation can take 30-60 seconds. Adding explicit caching:

• Cache path: ~/.foundry (default Foundry installation directory on Linux/macOS)
• Cache key: foundry-${{ runner.os }}-${{ hashFiles('.github/workflows/*.yml') }}-v1.4.2
• Restore keys: foundry-${{ runner.os }}- (for partial matches)
• Cache step placed before foundry-rs/foundry-toolchain action
• The toolchain action will skip installation if binaries already exist

Note: The foundry-rs/foundry-toolchain action has built-in RPC response caching (~/.foundry/cache/rpc), but caching the binaries themselves provides additional performance benefits.

macOS Runner Implementation

All workflows will use macos-latest instead of ubuntu-latest for better performance:macOS Runner Specs:

• 3-core M1 processor (vs Ubuntu's 2-core) - 50% more CPU
• 7 GB RAM (same as Ubuntu)
• 14 GB storage
• Free and unlimited for public repositories

Implementation:

• All workflows updated to use runs-on: macos-latest
• Foundry cache path remains ~/.foundry (same on macOS)
• All dependencies (Foundry, Node.js, pnpm) support macOS/ARM architecture

Considerations:

• Better for CPU-intensive tasks (builds, tests with parallel execution)
• M1 architecture (ARM) - all tools have ARM-compatible versions
• For private repos: macOS consumes 10x minutes (not applicable for public repos)

Version Verification

Before finalizing, verify the latest versions:

• foundry-rs/foundry-toolchain: Check GitHub releases for latest v1.x tag
• changesets/action: Check GitHub releases for latest v1.x tag

Expected Performance Improvements

• Faster action execution: v4 actions have performance improvements
• Better caching: Conditional installs skip work when cache hits
• Foundry caching: Saves 30-60 seconds per run when cache hits (binaries don't need re-downloading)
• Parallel execution: Removed sequential dependency allows jobs to run concurrently
• Reduced redundancy: Eliminated duplicate checkout and caching operations
• macOS runners: 50% more CPU cores (3 vs 2) speeds up CPU-intensive tasks like builds and parallel test execution

Testing Strategy

[brunoguerios]

Is the idea to rely on devs running integration tests locally to make sure things are working as expected?
Other than that, maybe a nice thing would be to create low priority tasks to add a few extra unit tests to cover basic implementation.

[johngrantuk]

Is the idea to rely on devs running integration tests locally to make sure things are working as expected? Other than that, maybe a nice thing would be to create low priority tasks to add a few extra unit tests to cover basic implementation.

Yeah, I think thats possibly the best solution for now. pnpm test:all passes locally for me a large percentage of the time. Let me know if you don't think this is the right approach though.