Secret Agent Seo
Viral Marketing for a Web 3.0 World
“Robots.txt”
So there it is, the source of all we know about the growing AI that makes up Google. The only way to stop the avaricious engine of the Internet from indexing everything in it’s path is this simple phrase.
It is a fascinating thing to drill down into, the deeper workings on the spiders, bots and agents swarming the web in search of content. How do these things work, and what is really going on. I have read that Google has been building these vast server farms with independent power sources and resources at certain strategic places around the globe to ‘mirror’ the entire Internet.
Anyway, grand conspiracy theories aside, if you want to check the lexicon of search agents, check out robotstxt !