We have encountered an issue where pages from the subdomain Webflow (custom 2nd one, so can’t just close it in the Webflow settings), which also serves as our production server (used to pull all content to our main site https://synder.com/), have started getting indexed.
You’re using Cloudflare, so you could create a worker on those paths and add the noindex.
I do a similar thing in our development setups to prevent indexing of our regular dev and test servers, and I add a robots.txt on those subdomains as well.
This is a common issue in Webflow that we’ve encountered with some of our clients. Webflow doesn’t allow you to select which of your added domains can be indexed, and custom code applies to all domains collectively. This can lead to unintended indexing of subdomains, causing duplicate content issues.
Here’s how you can address it with Cloudflare Workers:
Set up your domain in Cloudflare
Navigate to the Cloudflare dashboard and add your domain. Follow the prompts to verify ownership and configure initial settings.
Update your DNS records to point to Cloudflare
In your domain registrar’s settings, update the nameservers to those provided by Cloudflare. This process may take up to 24 hours to propagate (but in most cases takes only 10-15 mins).
Activate Cloudflare Workers
In the Cloudflare dashboard, go to the “Workers” section and create a new Worker.
Create a Worker to detect and handle unwanted domains/subdomains
In the Workers section, create a new Worker and paste the code provided below. Customize the variables at the top to match your specific needs.
For a detailed guide on setting up Cloudflare Workers with Webflow, check out our article about multiple Webflow projects on the same domain. While it covers a different topic, the process for setting up Workers is similar and can be useful.
Here’s a sample Cloudflare Worker code you can use:
// Customize these variables
const allowedDomains = ['synder.com']; // Domains you want to index
const noIndexDomains = ['e-commerce-page.synder.com', 'synder-home.synder.com', 'integrations-page.synder.com']; // Domains/subdomains you want to noindex
addEventListener('fetch', event => {
event.respondWith(handleRequest(event.request))
})
async function handleRequest(request) {
const url = new URL(request.url)
if (noIndexDomains.includes(url.hostname) || !allowedDomains.includes(url.hostname)) {
// Fetch the original response
let response = await fetch(request)
let body = await response.text()
// Add noindex, nofollow meta tag
body = body.replace('</head>', '<meta name="robots" content="noindex, nofollow">\n</head>')
// Create a new response with modified body
return new Response(body, {
status: response.status,
statusText: response.statusText,
headers: response.headers
})
}
// For allowed domains, serve the site as is
return fetch(request)
}
This Worker checks if the accessed domain is in the noIndexDomains list or not in the allowedDomains list. If either condition is true, it adds a noindex, nofollow meta tag. For your allowed domains, it serves the site as usual.
If anyone encounters similar issues in the future, feel free to reach out to us at BRIX Agency. We specialize in Webflow and technical SEO, and we’re here to help resolve these (or any other Webflow-related) types of problems to prevent duplicate content and its associated issues.