Lighthouse CI and Automated Regression Detection

The plain-language idea

The previous lesson covered the general pattern of enforcing a performance budget in CI. Lighthouse CI is the tool most commonly used to actually implement that pattern for Lighthouse-measurable metrics, and it does meaningfully more than "run Lighthouse and check the score."

A single manual Lighthouse run has real run-to-run variance — machine load, background processes, network jitter can all shift a lab measurement by a noticeable amount between two runs of the exact same code. Trusting one run to gate a merge means occasionally failing a perfectly fine PR, or worse, occasionally passing a genuinely regressed one. Lighthouse CI runs multiple times per build and takes a median (or applies a statistical assertion across the runs), which is far more trustworthy than any single sample.

It also keeps history: results are stored per build, so when a metric does regress, it can be traced back to the specific commit where it happened, not just discovered as "somewhere in the last month, this got worse." And it can assert against individual metric thresholds, not just the rolled-up overall score — a PR fails because LCP specifically crossed its budget, with that fact stated directly, rather than because "the performance score dropped 4 points" for reasons a reviewer then has to go dig up.

See it in a real thing: an assertion config that fails for the right reason

// lighthouserc.json
{
  "ci": {
    "collect": {
      "numberOfRuns": 5,
      "url": ["https://preview-pr-482.example.com/"]
    },
    "assert": {
      "assertions": {
        "largest-contentful-paint": ["error", { "maxNumericValue": 2500 }],
        "cumulative-layout-shift": ["error", { "maxNumericValue": 0.1 }],
        "categories:performance": ["warn", { "minScore": 0.9 }]
      }
    }
  }
}

With numberOfRuns: 5, this config runs the audit five times against the PR's preview deployment and asserts against the median. LCP and CLS are hard failures against their specific numeric thresholds — the same Core Web Vitals "good" thresholds from earlier in this track — while the rolled-up performance score is only a warning. A reviewer looking at a failed check sees exactly which metric broke and by how much, not a single opaque number.

What people usually miss

Watch out

• Multiple runs exist specifically because lab measurements are noisy, not to be thorough for its own sake. A single run failing a PR is often just variance, not a real regression — that's the exact false-positive problem the median run count solves.
• Metric-level assertions are more actionable than an overall score. "Performance score dropped from 94 to 89" tells a reviewer nothing about what to fix. "LCP exceeded 2500ms" tells them exactly where to look.
• Historical storage is what makes attribution possible. Without stored per-build results, a regression is invisible until someone happens to notice it, and by then it's unclear which of many merged commits caused it.
• Passing Lighthouse CI doesn't mean production is fast. It means the app is fast under the specific conditions CI simulated. Real users on real networks, real devices, and real third-party scripts can still have a materially worse experience than what CI measured.

Why this matters in interviews

"How do you make a performance regression fail a PR the way a broken test would?" is a common systems-thinking question, and naming Lighthouse CI's median-of-multiple-runs and metric-level assertions (versus a fragile single manual run) is what separates someone who's actually operated this in a real pipeline from someone who's only run Lighthouse manually once.