Documents stored in any directory not showing up in Google search

Our website was built with Webflow and it’s been a great tool.

Until this week, all of our normal pages have been indexed by Google and appear in search listings. It was easy to get them indexed.

However, we recently created some blog articles in the Webflow CMS, and when we published them, Google console returned the following error:

Crawl allowed? error | No: blocked by robots.txt

We couldn’t tell if our inability to index these new pages was specifically related to something funky with the CMS directory, so to confirm that it was not CMS-related, we built a new and separate directory.

Then we created a test web page and placed it inside that new directory. When we went to Google Console and request indexing for the new URL, the same error appeared:

Crawl allowed? error | No: blocked by robots.txt

So it appears to be that anything in a directory isn’t getting indexed by Google.

Yesterday in the SEO section of Webflow dashboard, our settings were set to this:

Disable Webflow Subdomain Indexing:yes
**User-agent: * **
Disallow: /

However, today we changed it to this:

Disable Webflow Subdomain Indexing:yes
**User-agent: * **
Allow: /

Is this simply a matter of our needing to be patient and wait for Google to recognize these changes to our robots file?

Thanks for any feedback or insights.

User-agent: *
Disallow: /

This setting disallows the entire site. Never add this setting to a “live” website!!

In your case remove the entire code of robots.txt (No meaning for allow if you do not disable something).

After this - use URL Inspection tool to re-index your site faster:

If you want only Disable Webflow Subdomain (io) Indexing (NOT YOUR CUSTOM DOMAIN INDEXING):

1 Like

Thanks Ezra. Will give it a shot :ok_hand:

Any insight into how long it takes for the changes to resolve on Google’s end?

Followed your instructions and Google still says there’s a robots issue minutes after I updated.

Thanks again.

For those with a similar problem, this solution works, it just takes about 24 hours for Google’s servers to register the robots change. After that you can use Google Console to register the page.

1 Like

Great :slight_smile: Please mark my answer a solution for future searches about this topic - Thanks!

done - and thank you!

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.