I have tried multiple methods to make my client’s custom domain name appear on Google Search to no avail. However, the webflow staging site shows up just fine.
I tried disabling the index, removing the index of the staging site (waiting to process right now), and adding 301 redirect (which just creates a redirect loop and fails to load the custom domain).
On Google Search Console, the customer domain is not indexed. The live test showed that the site is blocked by robots.txt. I tried adding that before to disable access to the staging site, but that didn’t work on the staging site but rather the custom domain one. Now that I removed the robots.txt, the live test still showed it being blocked.
I have read other community posts and resources where a solution is to duplicate. However, it would involve buying another site plan and I’m not sure is client-friendly.
I’m honestly at my wit’s end. Any help is greatly appreciated.
That approach works well if you’re struggling to remove the webflow.io site from SERPs. But you don’t need to clone it and transfer the site plan ( which would be free ), you can just rename your webflow.io site.
You’ll find that setting at the top of the Publishing tab under site settings. Note that even with 404’s, it may take awhile for Google to de-list that staging site from SERPs.
I changed my webflow.io site name and it indeed did not show up when I searched. However, I tried adding the robots.txt like you suggested and it is still not indexed. Before there was no custom txt and it didn’t show up either. Is there any way to have my custom domain index by Google search engine?
I didn’t suggest anything- your robots.txt was fine as I showed you, do not change it.
Once you get your settings right, it will take days, weeks, or months for indexing, it’s not immediate.
The main thing is to use Google Search Console, submit your sitemap, and submit any problems you had for correction. It’s a process that involves a lot of patience and some learning.
I see. I have been looking at my site again and it seems that my robots.txt was wrong slightly as it has both http and https which confused the bots. After fixing that, I was able to submit the sitemap and request for indexing.
Thank you so so much for your quick and informative help. I am still very new at this so there’s a lot to learn
You’re absolutely right- I thought you were showing me Webflow’s default robots.txt file and I wasn’t paying attention. I only looked at the Disallow rule which is what most people mess up.
Apologies for that, it was a poor oversight.
Typically you want to use Webflow’s default robots.txt setting, which is a 404 response- however once you’ve set the robots .txt you can’t simply delete it.
What was most likely causing problems for you was that first line- it’s not a valid rule so Google would likely see the robots.txt as broken, and interpreted it as “robots not allowed”. This is why leaving it un-set is best.
Most likely when you do set a robots.txt, Webflow probably appends the sitemap line itself. That means in your setting panel you probably just want the rules;
User-Agent: *
Disallow:
Which says “for all robots, block nothing”
But you can leave it the way you have it, as long as GSC reports it can process the robots.txt you’re all good.
