Monitoring & Observability with CloudWatch

The plain-language idea

CloudWatch is AWS's monitoring service, and almost everything else on AWS ships data into it automatically, in three shapes:

• Logs — raw text output, organized into log groups (usually one per service — one Lambda function, one ECS service) and log streams within them (usually one per running instance of that service). Lambda ships console.log output here automatically, with zero configuration.
• Metrics — numeric time series: request count, error rate, CPU utilization, duration. Namespaced by service (AWS/Lambda, AWS/RDS) and broken down by dimensions (which specific function, which specific database).
• Alarms — a threshold watching one metric, that triggers an action (send a notification, trigger auto-scaling) when crossed for a sustained period — not on a single noisy spike, but a defined evaluation window.

See it in a real thing: from a spike to a root cause

1. A CloudWatch alarm on a Lambda function's Errors metric crosses its threshold and fires — the first signal something is wrong, often before a single user complains.
2. The alarm's notification names the specific function and time window — not "something broke somewhere."
3. CloudWatch Logs Insights (a query language over log groups) filters that function's logs to the exact time window: fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc.
4. The actual error and its stack trace are sitting in the query results — the alarm found that something broke, the logs found what.

What people usually miss

Watch out

• An alarm needs a sustained breach, not a single data point, by design. A metric crossing a threshold for one evaluation period usually doesn't fire an alarm on its own — alarms are configured with an evaluation window (e.g. "3 out of 3 periods") specifically to avoid paging someone over one noisy, self-correcting spike.
• Log retention isn't infinite by default, and isn't free. CloudWatch Logs keeps data indefinitely unless a retention period is explicitly set on the log group — an easy-to-miss setting that either accumulates cost forever or silently loses old logs sooner than assumed, depending on which direction the default surprises you.
• CloudWatch Logs Insights queries cost money per gigabyte scanned, separate from log storage cost — an unbounded query with no time-range filter across a large log group can be a real, surprising line item, not just a slow query.
• Custom metrics are a real, first-class option, not just what AWS's own services report. An application can publish its own business-specific metric (e.g. "checkout completions") directly to CloudWatch — monitoring isn't limited to what infrastructure automatically reports.

Why this matters in interviews

"How would you find out why a Lambda function started failing at 2am?" tests whether a candidate has a concrete answer — check the alarm that fired, narrow the log group to that time window with Logs Insights, find the stack trace — versus a vague "I'd look at the logs," which doesn't say where, how, or in what order.