Start with search evidence
Google's robots.txt guidance frames the file as crawler-access control; sensitive or non-indexed material still needs access control or index-management choices.
Public pages should separate robots rules from sitemap, canonical, noindex, login, and removal decisions.
Visibility is not demand
The useful question is not whether the page appeared somewhere; it is whether the search term, page promise, and next action fit the same reader job.
Check the page path
- Add one handling row per page type: allow crawl, disallow crawl, noindex, login protect, or remove
- Keep the test narrow: one priority page with clear topic, source links, internal links, and a conversion action
What still needs proof
Using robots.txt as a secrecy layer can leave sensitive URLs exposed in other ways. Keep the original source open so the announcement, the evidence, and this site's interpretation stay separate.