Webflow blocking Ahrefs crawler, robots.txt broken

[Noah_Blier]

Hi everyone,

I’m trying to figure out what’s going on with our robots.txt and crawlability. Google’s robots.txt report shows the following, but neither is our primary robots.txt (getnametag.com/robots.txt):

Our actual robots.txt page (getnametag.com/robots.txt) doesn’t load for me at all, even though I’ve configured it in Webflow. In addition, Ahrefs site audit crawl shows me the following errors in Project Settings → Scope:

I asked Ahrefs support what this means, and they gave me the following response:

“There is some type of block for us when we try to reach the server for this site; kindly check in with your hosting provider for more details, and to also request that they whitelist our IP ranges and user-agents (AhrefsBot & AhrefsSiteAudit).”

Unfortunately, Webflow report still hasn’t responded to my ticket, so here I am. Has anyone encountered something similar?

[memetican (Michael Wells)]

Hi Noah,

It looks as though your site is not hosted on Webflow, or else has a reverse proxy setup ( cloudfront? ). You’d want to check there for interference with your robots.txt and how it’s returned.

In particular, from your error message, I’d assume that the problem is that your proxy identifies the robots.txt as gzip encoded.