Robots.txt GeneratorA few months agone I wrote a blurb nigh robots.txt - what it is together with how to role it. Google Webmaster Central only released their ain Robots.txt Generator. To access the tool , log-in to your Google Webmaster Tools employment concern human relationship , thus click on the Tools card selection on the left-hand side of the enshroud later you lot guide i of your verified sites. You'll come across a "Generate robots.txt" link amid the tool options. That's what you lot want.
The tool lets you lot arts and crafts specific rules for these item Google crawlers:
For to a greater extent than nigh Google's webmaster tools , live on certain to banking concern represent out the quick commencement guide they offer.