Home/Newsletter/Disallow Was Never a Privacy Setting
Edition #17

Disallow Was Never a Privacy Setting

Dan Toma·July 28, 2026· read
Key Takeaway

Robots.txt tells Google not to read a page. It never told Google not to list it. Shared AI conversations, including medical records, ended up in search results because those two instructions were confused.


FAQ

What is the difference between robots.txt Disallow and a noindex tag?

Disallow in robots.txt asks a crawler not to fetch a URL. Noindex, delivered either as a meta tag or an X-Robots-Tag header, tells a search engine not to list the page in results. They solve different problems, and using them together breaks the second one because the crawler never reads the header on a page it was told not to open.

Can Google index a page it has never crawled?

Yes. If Google discovers a URL through a link, a sitemap, or another public reference, it can list that URL in search results without ever fetching the content. The listing may show little more than the URL and anchor text, but the page is in the index and findable.

How do I actually keep a page out of Google?

Allow crawling and serve a noindex directive, so the crawler can read the instruction. If the content is genuinely sensitive, do not rely on crawler directives at all. Put it behind authentication, because any unlisted public URL is one shared link away from being discoverable.

Subscribe to The Weekly Vibe

Every Tuesday. 5-7 original takes on what matters in AI, Marketing, and Business Growth. No spam, no fluff, unsubscribe anytime.