The Robots Exclusion Protocol became RFC 9309 in September 2022, and the standard itself states that its rules are not a form of access authorisation.
https://datatracker.ietf.org/doc/html/rfc9309The full answer on robots.txt versus noindex is here:
https://aiwebpageseo.com/blog/is-robots-txt-still-used-robots-vs-noindexHave you ever found a page blocked in robots.txt still showing in search results, and what did you do about it?
No replies yet.