Sorry it's been so long since I posted. Frankly, I've been a little burned out and needed a break. But I am back now and good to go...at least I think!
Anyway, here's a quick rundown of an important file to have on your web server 
right on the root. It's called robots.txt. Uploading the file to the root means that it is not located in any subfolder. After you upload this file to your server, you should be able to access the file by typing this into your web browser: http://www.mydomain.com/robots.txt
Robots.txt helps the visiting search engine spiders navigate to the pages and directories on your website. You can also set your robots.txt to tell the spiders to block certain directories on your site. One last item is that you can add the location of your site map so the spiders can find it.
Here is a sample robots.txt:
User-agent: *
Disallow: /cgi-bin/
Disallow: /private/
Sitemap: http://www.mydomain.com/sitemap.xml
The "User-agent: *" means this bit of code applies to all search engine robots.
The "Disallow: tells the robots that they should not visit any pages within those directories.If you are okay with the robots visiting all areas of your site then simply don't put anything after it like this: Disallow:
The line of code about the sitemap tells the spider where your sitemap is located. A sitemap is basically a list of all individual webpages on your site saved in a format called XML. I can't get into all the details here, but I will say that it is VERY important you create a sitemap! There a number of free tools online that will help you do this. Google also has some sitemap resources as well. I'll give more sitemap explanation in another post.
Remember that robots.txt can't force the spiders to do anything. If that were the case anyone could easily manipulate search engine results. However, having a robots.txt helps guide the spiders to the areas they need to go.
You can also use robots.txt to block visiting spiders, but again, some robots can ignore your robots.txt. Don't try to use your robots.txt for website security!
To create the file, open up notepad. Type in the code you want. Then save the file. Be sure to save it all lower case - robots.txt. Then upload the file to your webserver root.
To learn more advanced aspects of robots.txt, you can visit http://www.robotstxt.org.
Comments