So there it is, the source of all we know about the growing AI that makes up Google. The only way to stop the avaricious engine of the Internet from indexing everything in it’s path is this simple phrase.
It is a fascinating thing to drill down into, the deeper workings on the spiders, bots and agents swarming the web in search of content. How do these things work, and what is really going on. I have read that Google has been building these vast server farms with independent power sources and resources at certain strategic places around the globe to ‘mirror’ the entire Internet.
Anyway, grand conspiracy theories aside, if you want to check the lexicon of search agents, check out robotstxt !
He’s the secret agent of search marketing. Cool, collected with an algorithm. Kit Latham is the most dynamic and creative SEO I’ve worked with. He has an deep understanding of search on many levels. His grasp of keyword research in the online market space is proof of a sixth sense.