A Simple and Powerful Library to Deal with Web Robots Control Strategy.

This is a simple library to parse robots.txt and robots meta tag. The library fully respects the RFC 1808 and the RFC 1945

Code project article : http://www.codeproject.com/Tips/740073/A-Simple-and-Powerful-Library-to-Deal-with-Web-Rob


Last edited Mar 7, 2014 at 10:56 PM by vodevil, version 27