HOW to create robots.txt

[arslan.2010 (shan)]

hi guys, my website is new, i was checking Semrush site audit report, it says, your website has no robots.txt file. may i know how to create this, i wan google to index all of my website pages.

[yakub_yilmaz (YY)]

Hello,

Solution:

1. Go to Project Settings / SEO / Indexing
2. Turn off “Disable Webflow Subdomain Indexing” to enable a robots.txt
3. Write:
   User-agent: *
   Disallow:

maybe this link can help you:

Setting up a robots.txt file on Webflow (digidop.fr)

[arslan.2010 (shan)]

is it okay? now google will index and crawl all my website…right?

[yakub_yilmaz (YY)]

Change the text to
User-Agent: *
Disallow:

Then check by adding /robots.txt to your url to see it that worked

[arslan.2010 (shan)]

WORKING FINE…thanks for your precious time…