Is it possible to use a single robots.txt which will be loaded on multiple domains to bot-block some of those domains and not others?
In webflow you do not get a r.txt per domain.
How can you block all of no-bots.domain.com, but allow bots to yes-bots.domain.com when you are limited to the single SEO r.txt spec and both domains are published under one project.
Thanks in advance.
Here is my site Read-Only: LINK
(how to share your site Read-Only link)
Webflow does not have the ability to serve different content for different domain names on a single site.
Your only means of doing that would be to setup a reverse proxy, and handle no.mydomain.com specially by returning a different robots.txt
However the problem you’re trying to solve is usually better solved by setting a primary domain, and letting Webflow 301 redirect. No blocking necessary, all of your traffic goes to one domain.
Alternatively, if you need two different domains pointing to the same site independently ( I have a few clients like this ), use canonical URLs to avoid SEO duplicate content issues. You’ll have both domains visible to your users, but all the indexing on Google will be on one of those two domains - same as if you had bot-blocked the others.
