I’ve got two versions of my site:
1 - the privately published version on webflow’s url
2 - The exported version hosted on my webhost and domain.
Can you confirm that google and the other search engines bots can’t see the private version please? Or if they can that it’s set to noindex, nofollow. I just want to check there’s no chance of google seeing two identical sites and getting penalized.
I think you may have misunderstood? you do not want to put that robot.txt code on your " 2 - The exported version hosted on my webhost and domain." site or google will not index it …
I am a bit confused at the response to you also as probably only webflow would be able to access their server root level to block their servers from bots and indexing … and I thought that if you added code to the settings area of the control panel for the site you create with them that injects that code into your exported files and if you host with them the files placed on their servers?
Webflow … how do you block the bots from crawling our creative files that we do not host with you that we created / design in the dashboard, site design area? I think this it the question being asked.
Thanks for your answers and my apologies for the confusion. To be clear, I was worried that both my exported site on my own domain/hosting and the privately published site on webflow’s subdomain would both be seen by Google. Thus causing an issue for ranking my exported site. I’ve added the user agent code in robots.txt filed that thesergie gave to the webflow version and republished that.
So, just to clarify, is it impossible to add the robots code to the webflow site ONLY and leave my custom domain alone? (It is also hosted through webflow.) If so, the main downside that I see is that I couldn’t have my portfolio site show up in my account profile next to other websites I’ve created unless I’m ok with Google ranking the webflow version alongside the custom domain.
Currently, it seems like my options are: 1. Privatize my webflow site, thus removing it from Googles’s rankings while simultaneously pulling it out of “websites I’ve created” list on my account profile. 2. Leave the webflow site public and either: a. Be ok with my webflow site coming up in Google’s ranking, OR b. Add robots to both the webflow site AND my custom domain removing both of them from Google’s ranking (aka: goodbye organic traffic).
Right now, it looks like my best option is to privatize the webflow version of my site. Yes?