SEO Issues - Robots.txt & Sitemap

Hi there,

I’m having an issue with a couple of sites and SEO. All the meta titles and meta descriptions are entered on the pages, however, they are not being indexed correctly on the SERPs. Currently, these are the settings on the SEO tab within Webflow:

Can someone help with advice on how to make sure robots.txt and the sitemap are set-up correctly for appropriate indexing? Is addition set-up needed within the DNS or Google Search Console to have things index correctly? I tried the video on verifying with the HTML tag, but Google Search Console seems to have changed and it would not let me verify.

Any advice is greatly appreciated! Thank you!

For additional context, the domain host is Network Solutions. Originally I had to add a TXT file to the DNS to verify with Google. Could that be causing the issue? When I inspect in the console it says that the url is blocked by robots.txt.


Set your robot.txt with this if it ok for them to crawl all your site

User-agent: *

you can also revert to the old Search console to follow the video in webflow

You only need robots.txt if you need to disallow urls and folders.

Check your canonical url . Inspect the live url/s in search console as see what issues are reported.

I try to verify at a dns / registrar level. It is less likely to be accidently removed which is the problem with website code verification.