Tag: Robots.txt

The robots.txt file is a short text file at the root of your site that tells search engine crawlers which parts of the site they may fetch. WordPress generates a sensible default, and for most blogs the right robots.txt is only a few lines long. The risk runs the other way: one wrong line can…