This project is read-only.

A Simple and Powerful Library to Deal with Web Robots Control Strategy.

This is a simple library to parse robots.txt and robots meta tag. The library fully respects the RFC 1808 and the RFC 1945

Code project article :

Last edited Mar 7, 2014 at 10:56 PM by vodevil, version 27