text robots.txt
# The asterisk (*) in the User-agent field is shorthand for "all robots" User-agent: * Disallow: /admin/ Allow: / Sitemap: http://www.example.com/sitemap.xml
Example robots.txt file - specifying all User Agents, allow, disallow and URL to the sitemap.
Updated: Tuesday 5th October 2010, 11:40pm
There are 0 comments
Comments are currently closed.