Currently were facing two problems regarding our project.
The first issue is that I am excluding some pages from the search results, however google still crawls them and displays them which should not occur.
The second issue is that Google is displaying a 404 link on the search results. The page doesn’t exist anymore and we don’t want it to be displayed. According to google you should just wait until the page gets crawled again, this happened already and the link still won’t disappear. I already gave google the sitemap and I even tried blocking the page through the google search console, which you actually shouldn’t do in such a case. However, yet nothing has worked out so far.
Does anyone of you have experience with those issues?
When you test anything on Google, don’t do it in your browser, because it will show results served for you, especially for you, with history taken into account etc. So in Chrome, you click on File > New Incognito window, and you do your searches tests from there. This way, you see the results as anybody will see them.
Now what you did to exclude the page from engine isn’t at all doing that, sorry. The option you’ve checked is to exclude content from your own site internal search engine, if you decide to use it. Not from Goolge and other engines.
There is actually nothing on Webflow that you can click or check to exclude a PAGE from search engines. You can do that for the entire site, in Settings (and only for .io staging) but not for PAGES.
To exclude a page from Google, you open the page settings, go to the HEAD custom code box and paste the following code:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW> This is telling robots to not index the page and to not follow the links from this page.
And wait a few days.
Sorry that you’ve mistaken Site search and public search engines. (@jmw I ping you here so you can decide if the copy in the UI is somehow misleading or not thanks for having a look)
Thanks for the input this will help us a lot. I already did some research about the robot.txt, do I need to follow any other rules in the custom code regarding programing for example setting a etc.
So far so good with that issue, I guess for the 404 link were just going to wait a bit more and see how it behaves throughout the next days
