Google indexing failing

Publishing help SEO
1

Hi, help, please :sweat_smile:

I am trying to index my site on google but I get the following error when I make the request saying my site can not be crawled because the robots.txt is blocking it.

Screenshot 2022-08-19 at 12.46.00
Screenshot 2022-08-19 at 12.46.00931×654 44.1 KB

My current robots.txt file is empty so I do not understand why I get this error. I have also tried allowing all bots access to everything.

What other methods can I use to trouble shoot this?

Here is my site Read-Only: https://preview.webflow.com/preview/jamtmw401?utm_medium=preview_link&utm_source=dashboard&utm_content=jamtmw401&preview=b1b17e27a3eceaabbf048f436793af2a&workflow=preview

The live site is www.jamtmw.com

3

your robots txt seems to disallow everything:

User-agent: *
Disallow:

It’s not because the field in your site’s setting is empty that the robots.txt is. If you have previously forbidden robots, and just erase the content of the field, then the robots.txt tends not to be refreshed.

Try to specifically allow all robots by writing new values in your robots.txt field.

4

Hello Everyone, Google crawling is too late. I upload article on a daily basis on my site but still they are not indexed. Is google follow any algorithum according to that it index or something wrong in my site due to which some of my articles are not indexed.

Here is my Live site: https://truepaintballer.com

5 article I had upload in last month but they are still not indexed. need your help.

5

Hi @vincent, that configuration actually looks right according to the spec;. I’ve always found robots.txt syntax a bit unwieldy.

But this should allow all bots to crawl;

User-agent: *
Disallow:

Googlebot appears to support some additional constructions like;

User-agent: *
Allow: /

However that’s considered nonstandard.

@StuartM did the problem resolve? Perhaps it was a caching issue of some kind.

6

Hey,
I did not know yet how google crawling work. I’m working on one of my client blog website and when I request google for crawling then it does not crawl it.

Last 5 days I’m trying to index this article https://myhomewatertreatment.com/review/springwell-water-softener-reviews/ but unable to index it yet.

How can I quickly index it and what strategy I need to apply on it?
Waiting to your response.

7

Hey Tippu, your site is not running on Webflow-
It’s a wordpress site with a Yoast SEO config. You should find some good guides there for what you’re trying to do.