Robots.txt Generator

robots generator

robots generator

We’ve created a cool new robots.txt generator that is web based.

When wanting to hide a folder, file, or tell the search engines where our sitemap is without placing a direct link on the main page even we need help sometimes.

A robots.txt file is one of those things that you rarely edit, and therefore rarely re-use the syntax / commands. This tool will quickly generate all the data for you and even includes a link to a blank robots.txt file.

Features

  • Google, Yahoo, MSN, Google Image allow or deny feature
  • sitemap location feature
  • folder deny feature (up to 4 folders)

Check it out and bookmark it. 🙂
http://pagestat.com/robots-generator.php
[ad]

Robots.txt Disallow

foreign contaminant

foreign contaminant

The robots.txt file that sits in the root of your site is the place to add directions or permissions for robots. Search engine bots will check this file before indexing your pages. (at least they should)

If however you don’t want a certain bot to index your pages then you want to use the robots.txt disallow command in the file. The command is simple really.

Continue reading …