Hi, I recently submitted the above site (my first in Webflow) to a company called ‘TotalSEO’ it scored as I expected in most areas but they have raised a couple of issues that concern me. I’ll post a topic for each!
They say:-
Robots.txt is missing
It is important to ensure your website has a robots.txt file.
A robots.txt file allows you to restrict the access of search engine robots that crawl the web and it can prevent these robots from accessing specific
directories and pages you may not want them to see. It also specifies where the XML sitemap file is located.
One of the first things we do when we start a new SEO campaign is ensure the robots.txt file is in place and correctly set up.
I thought this was built in to Webflows functionality? Could someone please let me know how to rectify this?
Are you exporting the site and using your own hosting solution or are you hosting on Webflow? Webflow will create a robots.txt file automatically if you have content typed in the SEO > Robots.txt section for hosted sites.
Under Site Settings, press SEO:
Scroll down to Robots.txt and enter your content. You can type Allow: / if you want a vanilla Robots.txt entry.
Thanks @McGuire, I am using Webflow - so do I need to type Allow:/ or is that just for other hosts? Also, forgive my stupidity what do you mean by vanilla?
There’s a more important issue here: The robots.txt file is NOT required for the vast majority of sites. It sounds like this SEO tool is making you jump through extra hoops. There’s tons of literature online about this, and even Google says it in plain English: https://developers.google.com/webmasters/control-crawl-index/docs/faq#h01
Thank you @McGuire amazingly helpful and I will read up on the resources provides. Really appreciated. I think most of these SEO companies try to make it more complicated to justify their fee; however when they offered a free review I used it as a learning tool and I’ve learnt some valuable things - Thank you
