Lesson 19 of 30

Notifications and Observability

Q: How would you design failure notifications for a CI pipeline, and how would you handle a test that fails intermittently instead of just re-running it?

A strong answer covers: Notify on the failure transition (or on genuine break), not on every single run — alert fatigue makes a channel get muted, which is functionally no notification at all; Route the alert to a channel/person who can actually act on it, with enough context to act (commit, branch, link to the run) rather than a bare 'CI failed'; A flaky test should be tracked over time (failure rate, which test, how often) rather than silently re-run and forgotten; Recognize the cost of the re-run habit: it erodes trust in what a red build actually means, beyond just wasting CI minutes.

A red pipeline nobody sees is just a slower failure. Real observability means a human gets pinged fast, and flaky tests get tracked instead of just re-run.

15 min read

Check your understanding

Every option explains itself — why the right one is right, and why each wrong one is wrong. Miss one and it comes back later on a spaced-repetition schedule.

No quiz for this one — it's a practical topic

This topic is about judgment and practical tradeoffs more than a single right answer — a mock interview is the better way to test yourself on it.

Be interviewed on this →

Stuck? Ask the AI tutor

Hints without spoilers, and a plain explanation of why a wrong answer was wrong — answered only from this lesson. Free account, no card.

Interview questions on Notifications and Observability

What interviewers actually ask about this, and what a strong answer has to cover.

How would you design failure notifications for a CI pipeline, and how would you handle a test that fails intermittently instead of just re-running it?

A strong answer covers

Notify on the failure transition (or on genuine break), not on every single run — alert fatigue makes a channel get muted, which is functionally no notification at all
Route the alert to a channel/person who can actually act on it, with enough context to act (commit, branch, link to the run) rather than a bare 'CI failed'
A flaky test should be tracked over time (failure rate, which test, how often) rather than silently re-run and forgotten
Recognize the cost of the re-run habit: it erodes trust in what a red build actually means, beyond just wasting CI minutes

Likely follow-ups

Your team already has a Slack channel that pings on every CI run and everyone has muted it. How would you fix that without losing visibility into real failures?
How would you actually detect that a test is flaky versus genuinely broken, from CI data alone?
Once you've identified a flaky test, what do you do with it — fix it immediately, quarantine it, delete it? What decides which?

Practice these with the AI interviewer →

The plain-language idea

A pipeline that fails silently isn't actually protecting anything — it's just delaying the bad news until someone happens to check, usually a reviewer wondering why their PR is stuck, or worse, a user hitting a bug that a red build would have caught hours earlier. Notifications close that gap: the pipeline tells a human, fast, through a channel they already watch, the moment something breaks.

But notifications alone only solve "did I find out." Observability is the bigger habit: treating your CI history as data, not just a pass/fail gate you glance at and forget. The clearest example is flaky tests — a test that fails maybe one run in twenty, for no code-related reason. The lazy response is "just re-run the job," and most teams do exactly that, click by click, for months. The observable response is to notice that the same test has now failed on unrelated PRs eleven times this quarter, quarantine it, and actually fix or delete it — because a flaky test that gets re-run instead of fixed doesn't just waste minutes, it trains the whole team to distrust red builds, which is the exact opposite of what a pipeline is for.

See it in a real thing: a Slack notification on failure

This repo's own pipeline (.github/workflows/ci.yml) doesn't currently notify anyone — it runs type-check, lint, tests, content validation, and build, and stops there. Adding a failure notification is a small addition to the same job:

      - name: Production build
        run: pnpm build

      - name: Notify Slack on failure
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          channel-id: "#ci-alerts"
          slack-message: |
            :red_circle: CI failed on `${{ github.ref_name }}`
            Commit: ${{ github.sha }}
            By: ${{ github.actor }}
            <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run>
        env:
          SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

if: failure() is doing the real work here — it means this step only runs when a previous step in the job failed, so a healthy pipeline stays quiet and only a broken one pings the channel. That's the difference between a notification and noise: alert on the state change, not on every run.

Tracking flakiness instead of just re-running is a separate habit, and it needs a place to write to — a step that appends every test's pass/fail/duration to a dashboard or a simple database table (test_name, run_id, outcome, timestamp) turns "this test feels flaky" into "this test has an 8% failure rate over the last 50 runs, and it's always the same assertion," which is something you can actually act on.

What people usually miss

Watch out

A dashboard nobody looks at isn't observability, it's a wiki page. Build a dashboard because someone has a standing reason to check it — a weekly flaky-test triage, an on-call rotation — not because "we should have metrics."
Re-running a flaky test and moving on is the failure mode, not a fix. Every re-run without investigation is a vote that this test's signal doesn't matter, and eventually the whole suite's red/green result stops meaning anything to anyone.
Alert fatigue is a real failure mode, not a minor annoyance. A channel that pings on every failed run — including ones a human is already actively fixing — gets muted, and a muted channel is functionally the same as no notification at all.

The genealogy: this is what turns a gate into a feedback loop

Earlier lessons treated tests and lint as gates that stop bad code from merging. Notifications and flaky-test tracking are what turn that gate into a feedback loop the team actually learns from over time — the difference between a pipeline that blocks one bad PR and a pipeline that gets more trustworthy every month because someone is watching its own failure patterns, not just its instant-by-instant verdict.