Sitemap & Google indexing: what to spider and what to avoid?

[dakz]

Edit Edit Edit

[kalgary1]

Hey dakz,

I know this has been awhile since you posted this, but I have noticed the same thing.

Google is indexing a cid that doesn't exist...

Did anyone find a way to control this?

[Jhonnyf]

Use a Robot.txt file
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360

Spanish
http://www.google.com/support/webmasters/bin/answer.py?hl=es&answer=40360

[MontG]

Yeah, in the Robot.txt file you can set restrictions to where the crawlers can go.

And talking about spiders, I took one look at your profile image and was about to give a flick to the bug I thought was sitting on my screen. :D What's with the bug man?

- S

[kalgary1]

ok... my robot.txt file never contained the following:
User-agent: GooglebotDisallow: /*?Is this a good fix?What it (would) affect(s):1. Contact Page (disallowed)
2. Terms, Privacy, About, Currency Converter (disallowed):D

[kalgary1]

ok... my robot.txt file never contained the following:
User-agent: GooglebotDisallow: /*?Is this a good fix?What it (would) affect(s):1. Contact Page (disallowed)
2. Terms, Privacy, About, Currency Converter (disallowed):D

That didn't look right.
ok... my robot.txt file never contained the following:

User-agent: Googlebot
Disallow: /*?

Is this a good fix?
What it (would) affect(s):

1. Contact Page (disallowed)
2. Terms, Privacy, About, Currency Converter (disallowed):D

[Peter]

I don't know enough about robots.txt to give sound advice. But I do know one thing: Google is a great stream of potential customers. I wouldn't disallow the googlebot to visit your site because this is Google's "actual" method of indexing sites on their search index. If a googlebot cannot connect to your site, your content does not get indexed.

It's just a scary thought. I would however take a few minutes to visit a popular google forum to learn ways to properly address areas you do not want google and others to view. However, always keep in mind that the ILance "front/client side" does not give "guests or visitors" any control to bid buy and sell. So they must use the login menu (or be remembered via cookies) to actually login and see member - driven content, place bids, post listings, etc.

Just a heads up... please becareful.

-peter