Same sitemap multiple times in Robots.txt

Publishing help SEO
1

Hi,

My question is very basic but I hope you can help :slight_smile:
I added a Custom Sitemap to my Webflow project, and added the Sitemap to my robots.txt. My robots.txt looks like this in Webflow:

user-agent: *
disallow: /product/
disallow: /category/
disallow: /checkout/
disallow: /order-confirmation/
disallow: /paypal-checkout/

Sitemap: https://www.caregame.com/sitemap.xml

However, going to https://www.caregame.com/robots.txt - the same sitemap appears two times.

Webflow University’s article about sitemap mentions:

Note: Do not enter your sitemap link into the robots.text field if you are using Webflow’s auto-generated sitemap, otherwise your robots.txt file will contain 2 sitemaps for search engines to crawl.

I am not using Webflow’s auto-generated sitemap - I entered my own and the ‘Auto-generate sitemap’ section is well on ‘NO’. But I still have two sitemaps on my robots.txt, what can I do? And what are the risks in terms of SEO?

Thanks!

Here is my site Read-Only: LINK

1 Like
2

It looks like you fixed it…what was the solution, if you don’t mind me asking?

3

Hi @Tarik_Naj,

Webflow automatically adds the sitemap URL to your robots.txt file so if you add the sitemap URL yourself into Project Settings > SEO > robots.txt you will see duplicate URLs in your robots.txt file.

If you’re seeing this, you’ll just need to remove the sitemap URL you added and republish your project!

Hope this helps.

4

I am having this issue and have selected the auto-generate sitemap. Any help is appreciated.