Webflow robots.txt setting not getting published - customer support is not able to help

Publishing help SEO
#1

Hey everyone :wave:

In the SEO settings, I removed the robots.txt rules after the Webflow customer support team asked me to do it after multiple back-and-forth messages.

Screen Shot 2022-12-20 at 10.46.21 AM
Screen Shot 2022-12-20 at 10.46.21 AM1586×736 34.3 KB

Then I published the site. However, the robots.txt file of our website is not getting updated.

Screen Shot 2022-12-20 at 10.45.37 AM
Screen Shot 2022-12-20 at 10.45.37 AM760×274 16.6 KB

Appreciate any help in this area. This is blocking search engines from indexing our website.

Update on 21st December

The support team asked me to unpublish and republish the site. Because they believe it is caching issue. Even after doing that, the issue persists.

#2

I’m not sure what you mean by support is unable to help; if you’ve removed the custom robots.txt and republished, and it’s not republishing, that’s a major bug they’d be chasing down.

In the meantime, I’d try these things;

  1. Remove it again, save it again, republish it again
  2. Try changing it to spaces, save and republish, see if that fixed it
  3. Make it an allow rule;

For Google, this works,

User-agent: *
Allow: /
#3

Thank you for your reply. Appreciate it :pray:

Even after removing the robots.txt rules and republishing, Webflow is not updating the actual robots.txt file. I agree, this is a major bug (something which is happening on my end at least).

I had already tried resaving and republishing many times. It’s not working.

The support team gives some redundant answers once every day.

Totally not happy with the way Webflow and their team are working.

I also tried the following:

User-agent: *
Allow: /

The customer support representative actually asked me to remove that and resave it.

#4

Yes that would be super difficult-
My guess is some form of caching or CDN issue.

I can see on your bettermode.webflow.io site, that it was published about 1h 30 ago from now, and that the /robots.txt is still showing the old content;

User-agent: *
Disallow: /

Look this is a terrible hack, but it might just get you out of the deep end for now.
Let’s see if adding a 301 redirect will at least prevent googlebot from accessing that bad data…

Try adding this to your redirect table, and republish everything.

image

In the test I just did, Webflow accepts it, and this redirect works. I’m hoping that if you then resubmit your site in Google Search Console, it will decide the robots.txt no longer exists, and that will resolve your issue.

Good luck.