Hi folks. I have observed that my website’s robots.txt file has “x-robots-tag: noindex” in its header response. In fact, when I browse other Webflow-based sites that have robots.txt files, they do too. I’m two weeks after launch and not getting much indexing happening (fair enough; not many backlinks or credibility yet). But according to my research, having a noindex in the header for robots.txt may actually be instructing bots to skip the entire site, which would be very bad for SEO… And I have not found a single commercial site with the same noindex header for their robots.txt. Is it helping or hurting my site (or our sites)?
Thanks in advance