Auto Generate Sitemap publish and unpublish

Hi @vincent and @harveywun thank you so much for reaching out and excellent questions!

Our auto generated sitemap does not remove pages from the sitemap.xml if you have a custom robots.txt record to not index. Referencing this article.

An XML sitemap shouldn’t override robots.txt. If you have Google Webmaster Tools setup, you will see warnings on the sitemaps page that pages being blocked by robots are being submitted.

Now, robots.txt does not prevent indexation, just crawling. So if the pages were indexed before they implemented robots.txt, they may continue to be indexed. Google will also display just the URL for pages that it’s discovered, but can’t crawl because of robots.txt.

I have gone ahead and changed this to a Wishlist item and created the item here to vote on: Add "don't index" switch to pages | Webflow Wishlist

​Please let me know if this is helpful, or if you have any additional questions, I’m happy to help further. :slight_smile:

1 Like