Search Engine Optimization (SEO) Tools #2 – Robots.txt Generator

Robots.txt is an essential but highly ignored element of web design and SEO. Robots.txt is a text file that gets placed in the root of your web site, one for each web domain (yoursite.com). All the search engine spiders (and other good bots) check it to see what directories of your site they are allowed to index and which ones they are restricted from visiting. The bad bots look in it to see what directories are off limits and they head straight to your off-limit files to see what’s hidden there. There are ways to get rid of the bad bots (through use of dynamic robots.txt files), but for now, here is a great tool that will let you generate a quick robots.txt file for your site. Read more on Search Engine Optimization (SEO) Tools #2 – Robots.txt Generator…

Read more »