Normally, we host all of our sites on our own server, but a particular client needs features that we have decided to host them on webflow to provide.
We’re accustomed to creating our own robots.txt file and uploading it, but in Webflow, I see that on the SEO tab there is an “Indexing” section. I have pasted our robots.txt file there and published, however, Google Search Console is still showing it as non-existent. Can someone please explain if this is the proper place to upload the robots.txt file?
The sitemap worked and has been recognized/submitted. But I am just confused by the robots.txt portion. I’m sure it’s something basic I’m missing, and I’ve gone through the Webflow help documents to try to find my answer.
Under the root. /robots.txt
This is also where webflow system creates this file.
so first check if you find this file under (maybe you not publish the changes to the “live site”) www.mysite/robots.txt
If you want the page not to get into the index, it is better to use noindex in the robots meta tag. To do this, add the following meta tag to the page in the section:
This will allow you
remove the page from the index when you next visit the search robot (and it will not be necessary to manually delete this page via webmasters)
allow you to transfer the reference page weight robots.txt file is best to close from indexing:
site search results
login / login / password recovery page
Yes, you are correct. But to add a solution to my issue, and what I think will help people in the future.
Solution: Go to Project Settings / SEO / Indexing
Turn off “Disable Webflow Subdomain Indexing” to enable a robots.txt to be produced at https://yourwebflowdomain.com/robots.txt and adding your robots.txt file copy into the box labeled “robots.txt”. Save the changes and then publish your site.