I’m trying to validate that all of the links on the page are valid, not broken, and don’t return a 404. I’m using a JavaScript tool to do this, but most of the pages return a timeout or 404 when I run the tool. My guess is that Webflow is blocking these calls because they are detected as bot requests. Does anyone know a way to get around this? Can I possibly change the userAgent or pass a parameter to disable this check?
It’s actually against WF’s terms of service and acceptable use policy to use bots on your WF-hosted site.
You might contact support and see what they recommend?
Is there some kind of automation built into WF to test key metrics of the website (which requires automated tools/bots) like tools that are found in litehouse for example, site availability and broken links that you know of?
I wouldn’t consider this a bot, it is a github action workflow that runs every week to validate key metics/health of the site. We do this on other providers by whitelisting our ip address.
Not natively.
I’d probably just use standard SEO tools like the free version of AHrefs to identify any unwanted 404s.
It’s possible that WF is blocking github actions explicitly. At one point tools like that were being abused to scrape WF’s free staging and host sites elsewhere.