I'm noticing Googlebot is not respecting my robots.txt. I'm seeing Googlebot's user agent crawling pages that have been in my robots.txt file for many months. Some of them are showing up in GSC as "Indexed, though blocked by robots.txt" with Last crawled dates indicated as recent as yesterday.
Additionally, I'm seeing Googlebot crawl my robots.txt file a few times a day, and the URLs are definitely blocked per the Google robots.txt tester.
My robots.txt is in the following format:
Sitemap: ...
User-agent: *
# ...
Disallow: ...
Disallow: ...
etc. ~ 40 lines
# ...
Disallow: ...
Disallow: ...
etc. ~ 60 lines
# ...
Disallow: ...
Disallow: ...
etc. ~ 20 lines
WordPress has always been praised as one of the most SEO friendly and optimized content management systems (CMS) out there – even right out-of-the-box! But there are still some very important and basic first steps you will want to take upon first installing WordPress on your blog or site to help optimize the on-page SEO […]
