Comments on: “Robots.txt”

By: admin

admin — Wed, 24 Sep 2008 20:36:02 +0000

Hey Joe,

Thanks fer commenting – excellent advice, I concur !

best

tarky7

Joe — Tue, 23 Sep 2008 15:37:05 +0000

META NAME=”robots” CONTENT=”FOLLOW,INDEX”
meta name=”googlebot” content=”index, follow, all”

Joe — Tue, 23 Sep 2008 15:36:27 +0000

my meta is

Joe — Tue, 23 Sep 2008 15:35:13 +0000

I understand thats the meta but the robots.txt is actually a file

User-agent: *
Disallow: /cgi-bin/
Disallow: /secure/

the above is just an example you can change it to however you need it.

So you really dont need the tag. I dont have any secure parts so I use this as my meta.
The Google Bot i just put in there, you dont need it.

and my robots.txt file located on the root of my server looks like this…

# Allows all robots

User-agent: *
Disallow: