#robots.txt tells various robots where they can and cannot go. This allows robots everywhere. User-agent: * Disallow: