Streaming live at 10am (PST)

Pages blocked in robots.txt showing in google index

Hi

My robots file looks like this:
http://leicestermortgagebroker.co.uk/robots.txt

but when I run site://leicestermortgagebroker.co.uk in google, the contact and privacy pages are showing in google’s index, which I don’t want.

Any idea why this is? Is the only way around it to remove these urls in google webmaster tools?

Webflow Support, any idea if/when the ability to noindex individual pages will be available as a feature? It would be really useful for seo purposes.

Thanks a lot

Graham

https://developers.google.com/webmasters/control-crawl-index/docs/faq?csw=1#h17

robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header.

Hi Vincent

Thanks very much for your reply.

How do I use the noindex robots meta tag in webflow on an individual page please. I thought this wasn’t possible in webflow yet?

Thanks

Graham

I am not sure you can for individual pages as the custom header code (where Meta goes), impact the whole site. Maybe ask Webflow support about that.

Was there any resolution to this question? I have the same need.

I too, have the same question.

Seems like you should be able to throw <META NAME="ROBOTS" CONTENT="NOINDEX"> into the Head tag section with no problem … as noted in the UI “code included here will only apply to this page”

1 Like

I really don’t know why I was under the impression that was not the case… must have been raelly tired :—) Thanks for clearing things up.

1 Like

No problem! Easy enough to do, especially when you’re tired : )