Non-indexable Pages

Publishing help SEO
1

Hi - When I use the Screaming Frog SEO spider, I get the following results. It is showing all of my pages to be non-indexable. I am an SEO newbie. Is this how it is supposed to be?

Screenshot 2022-11-15 at 2.43.53 PM
Screenshot 2022-11-15 at 2.43.53 PM3260Ă—958 383 KB

Here are the changes that I have made:

I have inserted a robot.txt file as follows:

User-agent: *
Allow: /

I have also disabled Webflow subdomain indexing from the settings page.

Here is the sitemap that I inserted that was generated by Screaming Frog application:

<?xml version="1.0" encoding="utf-8"?> https://www.neighbors.ai/ 2022-11-08 daily 0.9 https://www.neighbors.ai/mobile-workforce 2022-11-08 daily 0.8 https://www.neighbors.ai/products 2022-11-08 daily 0.8 https://www.neighbors.ai/oems 2022-11-08 daily 0.8 https://www.neighbors.ai/contact-us 2022-11-08 daily 0.8 https://www.neighbors.ai/business 2022-11-08 daily 0.8 https://www.neighbors.ai/about 2022-11-08 daily 0.8 https://www.neighbors.ai/channel-partners 2022-11-08 daily 0.8 https://www.neighbors.ai/insights 2022-11-08 daily 0.8

Finally, I have set my global canonical tag URL to be: http://www.neighbors.ai

Thank you for your help!

Here is my site Read-Only: LINK
(how to share your site Read-Only link)

2

No idea what screaming frog considers indexable, but Google search console will be your actual authority on that.

I’d recommend using Webflow’s auto-generated sitemap. It’s much more relevant, as it updates automatically whenever you add a blog post, etc.

Your canonical URL is not on your published site, and it should be;
https://www.neighbors.ai
With the s

3

Thank you so much Michael @memetican! I have started to use Google search console, and made the changes you suggested. It has fixed the issues that I was seeing. Just to learn about it, why is the https a better canonical URL than http version?

4

Also, one more thing: Should I leave the robots.txt blank or insert something there?

5

Canonical means “correct, formal”, and your site is SSL, so the https: version is the right one.

In general, you never want google to follow a Canonical link and then be redirected somewhere else.

1 Like
6

I usually find leaving it blank gives me the best results, but on some google search consoles I’ve seen other people report errors and problems. I’m doing some tests on this now to see if Google’s preferred behavior has changed.

1 Like