Basic robots.txt question

Normally, we host all of our sites on our own server, but a particular client needs features that we have decided to host them on webflow to provide.

We’re accustomed to creating our own robots.txt file and uploading it, but in Webflow, I see that on the SEO tab there is an “Indexing” section. I have pasted our robots.txt file there and published, however, Google Search Console is still showing it as non-existent. Can someone please explain if this is the proper place to upload the robots.txt file?

The sitemap worked and has been recognized/submitted. But I am just confused by the robots.txt portion. I’m sure it’s something basic I’m missing, and I’ve gone through the Webflow help documents to try to find my answer.

Thanks!

1 Like

Bumping this. Hoping for a little help soon.

Under the root.
‎/robots.txt
This is also where webflow system creates this file.
so first check if you find this file under (maybe you not publish the changes to the “live site”)
www.mysite/robots.txt

maybe this is hosting issue:

webflow docs:

I’m hosting this site ON Webflow. The Search Console finds no robots.txt file. I thought Webflow generated one?

And, just in case, I inserted my own into the “indexing” portion on the site. Still nothing.

How is it a hosting issue? Again, I’m paying Webflow to host this site, just need to know how to get the robots.txt file recognizable by Google Search Console.

1 Like

Add url + Screenshot of your robots.txt area (In the UI)

First check if you have this file under
www.mysite/robots.txt

Example (webflow site from “showcase” area):
https://www.milch.com/de/robots.txt

If you want the page not to get into the index, it is better to use noindex in the robots meta tag. To do this, add the following meta tag to the page in the section:

<meta name = ”robots” content = ”noindex, follow”>.

This will allow you

remove the page from the index when you next visit the search robot (and it will not be necessary to manually delete this page via webmasters)
allow you to transfer the reference page weight
robots.txt file is best to close from indexing:

admin site
site search results
login / login / password recovery page

Did you find a solution? I have the same issue. I do not have robots.txt in the appropriate URL https://energyhomeinspection.com/robots.txt

https://preview.webflow.com/preview/energy-home-inspection?utm_medium=preview_link&utm_source=designer&utm_content=energy-home-inspection&preview=cf17d75af700ab928359af6ebeb282a7&workflow=preview

@Astrotheque - There is a robots.txt file present at the uri you provided. Maybe you had dns propagation issues or you did not publish your changes before.

1 Like

Yes, you are correct. But to add a solution to my issue, and what I think will help people in the future.
Solution: Go to Project Settings / SEO / Indexing
Turn off “Disable Webflow Subdomain Indexing” to enable a robots.txt to be produced at https://yourwebflowdomain.com/robots.txt and adding your robots.txt file copy into the box labeled “robots.txt”. Save the changes and then publish your site.