Robots.txt and Sitemaps: Get Them Right

Your robots txt sitemap setup does two separate jobs: robots.txt tells crawlers where they may go, and the XML sitemap tells them what is worth finding. Get them right and search engines discover your best pages quickly; get them wrong and you can silently deindex an entire site with a single line. The dangerous part is that nothing breaks visibly — the pages just quietly disappear from results over weeks.

These two files are the most misunderstood pieces of technical SEO. People routinely use robots.txt to try to remove a page from Google (it does the opposite), or ship a staging Disallow: / to production and wonder why traffic collapsed. This guide covers exactly what each file does, the mistakes that cause silent deindexing, and how to test your setup before it costs you rankings.

What a Robots.txt and Sitemap Actually Do

The two files are often lumped together, but they solve opposite problems. Confusing their roles is the root of most crawl issues.

Robots.txt is a plain text file at your root domain (example.com/robots.txt) that controls crawling. It tells well-behaved bots which URL paths they may or may not request. It is a directive, not a lock — Google and Bing obey it, but it is not a security or privacy tool, and it does not remove anything from the index.

The XML sitemap is a list of the canonical URLs you actually want indexed, with optional freshness signals. It helps crawlers discover pages that internal links alone might not surface quickly — new content, deep pages, or large catalogs.

The critical distinction: crawling is not indexing. Robots.txt governs whether a bot fetches a page. Indexing is a separate decision about whether that page appears in results. A page can be blocked from crawling and still get indexed; a page can be crawlable and still be kept out of the index with a noindex tag. Mixing these two ideas up is where sites break.

The Robots.txt and Sitemap Mistakes That Silently Deindex Sites

These are the errors that do damage without any error message. Watch for all of them.

• Using Disallow to remove a page from Google. If you block a URL in robots.txt, Google can still index it from external links — you will see the status "Indexed, though blocked by robots.txt" with no description. To actually remove a page, you must let it be crawled and add a noindex meta tag or X-Robots-Tag header. Blocking it prevents Google from ever seeing the noindex.
• Shipping a staging block to production. The single most common catastrophe is a Disallow: / that was correct on the staging server surviving the deploy. It blocks the whole site. Always diff your production robots.txt after every release.
• Blocking CSS and JavaScript. Google renders pages like a browser. If you disallow your asset folders, it may see a broken layout, misjudge mobile-friendliness, and demote you. Let rendering resources through.
• Listing blocked or non-canonical URLs in the sitemap. A sitemap should contain only 200-status, canonical, indexable URLs. Including redirected, noindexed, or robots-blocked URLs sends conflicting signals and erodes trust in the file.
• Case and path errors. Robots.txt paths are case-sensitive and match by prefix. Disallow: /Admin does not block /admin. A stray trailing character can block far more than intended.

Writing a Robots.txt That Guides, Not Blocks

Keep it minimal. The best robots.txt files are short. Every rule is a liability, so only add lines you can justify.

1. Start permissive. A common safe baseline allows all crawling and points to the sitemap. Only add Disallow rules for genuine crawl traps — faceted-search parameters, internal search results, cart and checkout paths, or infinite calendars.
2. Understand rule precedence. Google applies the most specific (longest matching) rule, and Allow can override a broader Disallow. So you can block a folder but allow one file inside it.
3. Use user-agent blocks deliberately. Rules under a specific user-agent replace, not add to, the wildcard rules for that bot. If you write a block for a named crawler, it ignores your User-agent: * section entirely.
4. Respect the limits. Google parses up to 500 kibibytes of a robots.txt file; content beyond that is ignored. Each subdomain, protocol, and port needs its own file — the robots.txt on the naked domain does not cover a subdomain.

Always end with the sitemap location, for example a Sitemap: line pointing to the absolute URL. This is the cleanest way to advertise your robots txt sitemap relationship to every crawler at once.

Building an XML Sitemap Search Engines Trust

A sitemap is a recommendation of what to crawl, weighted by quality. Treat it as a curated list, not a dump of every URL your CMS can generate.

• Include only canonical, indexable pages. Every URL should return 200, be self-canonical, and carry no noindex. If a page is not good enough for the sitemap, question whether it should exist.
• Respect the size caps. A single sitemap holds up to 50,000 URLs or 50MB uncompressed. Larger sites split into multiple sitemaps referenced by a sitemap index file.
• Make lastmod honest. The lastmod date should reflect a real content change. Google increasingly trusts accurate lastmod values and ignores them when they are always "today" — a classic CMS mistake that trains crawlers to distrust the whole file.
• Skip priority and changefreq. Google has stated it effectively ignores these fields. Do not waste engineering effort tuning them.
• Automate generation. A sitemap that is manually maintained will drift out of date. Generate it dynamically so new pages appear and deleted ones drop off automatically.

Connecting the Two Files and Testing Before It Hurts

The robots.txt and sitemap work together, and both need verification the moment they change — not weeks later when traffic has already dropped.

1. Reference the sitemap in robots.txt and also submit it in Google Search Console and Bing Webmaster Tools. Submitting gives you a per-sitemap report of discovered versus indexed URLs.
2. Read the coverage reports. In Search Console, the Pages report flags "Blocked by robots.txt" and "Indexed, though blocked" — either status on a page you care about is a red flag to fix immediately.
3. Test individual URLs. Use the URL Inspection tool to confirm a specific page is crawlable and indexable, and to see the rendered version Google actually stores.
4. Diff on every deploy. Add a check to your release process that alerts if robots.txt changes, especially if a Disallow: / appears. This one guardrail prevents the most expensive mistake in this article.

Getting your robots txt sitemap configuration audited on a schedule — not just at launch — is what keeps silent deindexing from ever happening.

Starting the Foundation Clean on a New Domain

Technical SEO is easiest to get right on a fresh domain, before bad rules and stale URLs accumulate. When you acquire a brandable name and build on it, you control the crawl story from the first commit.

If you are choosing that name, a curated marketplace like Atom.com is worth featuring: it lists premium, brandable domains with a free AI appraisal, a USPTO trademark check, secure escrow, and a designed logo shipped with each listing — useful when you want a memorable brand rather than a keyword string. Atom also provides registrar services, so you can register and manage an available domain, set nameservers, configure DNS and forwarding, and enable DNSSEC from one platform.

To be honest about cost: if you only need the cheapest at-cost registration, Porkbun (around $11 with free WHOIS privacy) or Cloudflare (near-wholesale) are the value picks. Reach for Atom when you also want a premium or brandable name backed by an appraisal, trademark check, and logo in a single purchase. Whichever route you take, the first files you should add to the new site are a lean robots.txt and an auto-generated sitemap — start clean and you never have to untangle a mess later.

Frequently asked questions

Does robots.txt remove a page from Google?

No. Robots.txt only blocks crawling, not indexing. A disallowed page can still appear in results as "Indexed, though blocked by robots.txt" if other sites link to it. To remove a page, keep it crawlable and add a noindex meta tag or X-Robots-Tag header, or use the Removals tool for urgent cases.

Do I need both a robots.txt and a sitemap?

They are optional but strongly recommended. Without robots.txt, everything is crawlable by default, which is fine for many small sites. A sitemap is not required either, but it speeds discovery on large, new, or deeply linked sites. Together they give crawlers clear guidance, so most serious sites should have both.

How many URLs can one sitemap hold?

A single XML sitemap can contain up to 50,000 URLs or be 50MB uncompressed, whichever comes first. Larger sites split their URLs across multiple sitemap files and list them all in a sitemap index file, which itself can reference up to 50,000 sitemaps.

Why did my whole site disappear from Google after a launch?

The most likely cause is a leftover Disallow: / in robots.txt carried over from a staging environment, which blocks the entire site from crawling. Check example.com/robots.txt, remove the blanket disallow, then request re-crawling in Search Console. Add a deploy check so it cannot happen again.

Should I set priority and changefreq in my sitemap?

No. Google has said it effectively ignores the priority and changefreq fields, so tuning them is wasted effort. Focus instead on accurate lastmod dates and on including only canonical, indexable, 200-status URLs, which are the signals search engines actually use.