How to Properly Add an llm.txt File in Webflow and Reference It in robots.txt

Community resources
1

Hi everyone,

I’m looking to add an llm.txt file to my Webflow site for LLM and AI crawler optimization.

Here’s what I’ve done so far:

  • I uploaded the llm.txt file as an asset in Webflow.
  • Then, I set up a redirect from /llm.txt to the uploaded file URL, since that was the only method I found mentioned on the Webflow forums.

My questions are:

  1. Is there a better or more direct way to add an llm.txt file in Webflow so it’s accessible at the root level (i.e., https://www.../llm.txt)?
  2. How can I make sure this file is properly discoverable by bots?
  3. Can we include a custom directive like this in the robots.txt file so the llm file is discoverable by bots?
    llm-discovery: https://www…/llm.txt

Would really appreciate any insights or suggestions!

Thanks in advance!

2

The upload a .txt file and redirect approach might work, however;

  • Some tools may not like the redirect, or the domain name change
  • It’s difficult to update
  • Updates must be made manually, rather than automatically
  • There is no support for the .md files that represent the page-level content of your site.

If you wanted something more professional and robust, I’ve just finished some tooling for a client site that is reverse proxy-based.

You can see it running on Sygnal’s site,
e.g. https://www.sygnal.com/llms.txt
e.g. https://www.sygnal.com/blog/llms-txt-webflow.md

I’m working on docs, but you’ll find partial docs here;

In case you or anyone coming across this need this kind of premium setup, the setup, config, and licensing are currently $497, one-time. No ongoing costs. Your DNS does need to be on Cloudflare, but it’s easy to move if it’s not already, and there are no costs there as well. Setup cost also covers the setup of your LLMS master page in the designer ( you have full control over the LLMS content in the designer itself ).

3

Hi, thank you so much for your detailed response — really appreciate you taking the time!

I completely agree that the redirect method I’m using isn’t ideal, and I’m aware that some tools and bots may not follow it properly, which is definitely a concern when it comes to discoverability.

As for the reverse proxy solution you mentioned — while it looks very robust and well thought out, I have to admit it might be a bit too complex and heavy-handed for what I was hoping to achieve. The pricing and Cloudflare requirement also make it feel a bit out of scope for what should ideally be a simple and native implementation within Webflow.

I was really hoping for a more lightweight Webflow-native solution or even something officially supported by Webflow itself. But thank you again for sharing your setup — it’s great to know there are advanced options out there for teams who need them.

Thanks again!

4

Yeah, at this point, LLMS.TXT is extremely new.

Google began indexing sites around 1996, but sitemap.xml wasn’t introduced as a standard until 2005. Between that was a wild west of HTML sitemaps that were designed for both humans and bots.

That’s the period we’re in with LLM-based indexing and site crawling. No real specification, just some loose conventions that will need to evolve and settle over the next few years.

I expect that LLMS.TXT would need to be strongly adopted by the industry and strongly supported by LLM’s before Webflow would build infrastructure to support it. Even then, it could be minimalistic. Sitemap.xml still doesn’t support lastMod, priority or freq, so I add those in with reverse proxies as well when clients need it.

I’d say do what you can with the redirects approach, see how it works for you.

1 Like