Testing in CI

The plain-language idea

A test suite proves something works — but only at the moment and on the machine it ran. If tests only ever run locally, "the suite is green" is a claim about one developer's laptop five minutes before they pushed, not a claim about the code sitting on main right now. Someone else's untested change, a dependency bump, an environment difference — any of these can break things between "it worked for me" and "it's in production." Running the suite in CI, on every push and every pull request, turns that local-and-temporary claim into something that's actually verified against the real, current state of the branch.

This project doesn't use a separate test framework at all — no Jest, no Vitest configured. Its tests are plain .test.ts files under lib/, each one a self-contained script with real assertions, run directly with tsx. That's an unusually lean setup, but the CI principle is identical to any framework: find every test file, run it, and fail the whole pipeline the moment one fails.

See it in a real thing: this repo's test step

- name: Run lib/ test suite
  run: |
    set -e
    for f in $(find lib -name "*.test.ts"); do
      echo "--- $f ---"
      pnpm exec tsx "$f"
    done

set -e is doing real work here, not decoration: without it, a failing tsx command inside the loop would print its error and the loop would just keep going to the next file, and the step could exit successfully even though a test failed partway through. With set -e, the first non-zero exit from any test file kills the step immediately — the job fails, and the failure is attributed to the specific file whose --- path --- line printed last.

Because each test is just a standalone script, this loop runs every test sequentially, in one job, on one runner. For a project with a handful of fast lib/ tests that's fine. Once a suite grows into hundreds of tests or starts including slow integration or end-to-end tests, sequential-in-one-job stops scaling, and that's where a matrix strategy earns its keep — splitting the suite across several parallel jobs instead of one long one:

jobs:
  test:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pnpm install --frozen-lockfile
      - name: Run shard ${{ matrix.shard }} of 4
        run: pnpm exec playwright test --shard=${{ matrix.shard }}/4

Four jobs start simultaneously, each running roughly a quarter of the suite, and the whole test stage finishes in close to a quarter of the wall-clock time — at the cost of four runners' worth of compute instead of one.

What people usually miss

Watch out

• A flaky test is worse than a slow test, not just annoying in the same way. A slow test costs you time predictably, every run — you can plan around it. A flaky test (one that fails intermittently for reasons unrelated to the actual change) costs you trust: people start re-running failed pipelines out of habit instead of investigating, and eventually a real failure gets re-run away too, ignored as "probably just flaky again."
• Parallelizing tests requires them to be independent. If two test files share state — the same test database row, the same port, the same file on disk — running them in different jobs at the same time can produce failures that never happen when the suite runs sequentially, and those failures are usually much harder to reproduce locally than a straightforward logic bug.
• "Tests passed in CI" and "tests ran in CI" are not the same claim. A test step with a bug that makes it exit 0 no matter what — a missing set -e, a swallowed exception, an empty test glob — will show a green checkmark forever while testing nothing. Worth occasionally reading the actual step output, not just the pass/fail badge.

The genealogy: a gate is only as good as its enforcement

Running tests in CI is the first half of the story — the next lesson covers linting and type-checking as gates in the same pipeline, and the one after that covers how a passing check actually gets tied to whether a pull request is even allowed to merge. A test suite that runs in CI but that nobody is required to keep green is barely better than one that only runs locally.