Generating sitemap.xml and robots.txt

The plain-language idea

sitemap.xml tells a crawler every URL worth indexing; robots.txt tells it which paths not to bother with. Both are conventionally static files sitting at a domain's root — but a static file has no way to know when your app adds a route, or when a route that used to be public becomes gated behind auth. Next.js's sitemap.ts and robots.ts file conventions replace the static text files with functions: app/sitemap.ts exports a function returning MetadataRoute.Sitemap (an array of URL entries), and app/robots.ts exports a function returning MetadataRoute.Robots (allow/disallow rules plus a pointer back to the sitemap). Next.js builds the actual /sitemap.xml and /robots.txt responses from whatever these functions return, every time they're requested (or at build time, for the static case).

Because these are real TypeScript functions, they can read from anything code can read from — a database, a content directory, an array of route constants — instead of a human remembering to update a hand-written list every time a route is added or removed.

See it in a real thing: deriving both files from one shared list

// app/robots.ts
import type { MetadataRoute } from "next";
import { SITE_URL } from "@/lib/core/site";

// The single source of truth both files draw from — routes an unauthenticated
// visitor (and a crawler) can never actually reach, because middleware 307s
// them to /login regardless of what the page component itself renders.
export const GATED_PREFIXES = ["/learn", "/dashboard", "/settings", "/genealogy", "/interview"];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: "*", allow: "/", disallow: [...GATED_PREFIXES, "/api/", "/auth/"] },
    sitemap: `${SITE_URL}/sitemap.xml`,
  };
}

// app/sitemap.ts
import type { MetadataRoute } from "next";
import { SITE_URL } from "@/lib/core/site";

export default function sitemap(): MetadataRoute.Sitemap {
  // Only the routes NOT covered by GATED_PREFIXES belong here — listing a gated
  // route in the sitemap would just point crawlers at a login redirect.
  return [
    { url: SITE_URL, changeFrequency: "weekly", priority: 1 },
    { url: `${SITE_URL}/how-it-works`, changeFrequency: "monthly", priority: 0.6 },
    { url: `${SITE_URL}/login`, changeFrequency: "yearly", priority: 0.3 },
  ];
}

Both files import the same SITE_URL constant rather than each hardcoding the domain — get that wrong (an env-driven value pointing at a preview deployment, say) and both outputs are wrong the same way at once, which is exactly the kind of bug that produces a build that passes and an SEO report that looks fine while quietly telling Google the wrong domain. sitemap.ts can also export a generateSitemaps function to split a very large sitemap (Google's practical limit is 50,000 URLs per file) into multiple numbered sitemap files automatically.

What people usually miss

Watch out

• These files don't know your app's routing rules on their own — you have to tell them. Neither convention automatically excludes an auth-gated route; if the actual gating logic lives in middleware (a PROTECTED_PREFIXES array, say), the sitemap and robots files have to reference that same list, or reimplement it, or they'll drift from what the app truly serves.
• sitemap.ts output is capped per file, not per site. Once a site has more than ~50,000 URLs, a single sitemap.ts isn't enough — generateSitemaps is the mechanism for splitting into a paginated sitemap index, not a manual loop writing multiple unrelated route files.
• robots.ts's sitemap field isn't optional decoration — it's how a crawler finds the sitemap at all without being told the URL out-of-band (e.g. via Search Console). Omitting it doesn't break anything visibly, it just means discovery relies entirely on other signals.
• A relative URL in either file resolves against nothing on its own. Unlike page metadata, which can lean on a root layout's metadataBase, sitemap.ts and robots.ts entries need a fully-qualified URL written out — there's no ambient base URL these two conventions inherit automatically.
• Both files still respect the same static/dynamic rendering rules as any other route. A sitemap.ts that reads no per-request data can be generated once at build time; one that queries a database for dynamic content is generated per-request (or on a schedule, with the right caching), with the same cost tradeoffs as any other dynamic route.

The genealogy: the metadata API's sibling conventions, not part of it

These aren't part of the metadata/generateMetadata resolution tree the metadata API lesson covers — that system resolves per-page <head> tags across nested layouts, while sitemap.ts and robots.ts are independent root-level file conventions producing two separate top-level responses. They share only the underlying idea: generate a crawler-facing artifact from code instead of a hand-maintained static file, so it can't silently drift from what the app actually serves.