meta tag for robots

Mar 22, 2009 at 1:47 AM
great job...it would also be nice to have something that can also parse the tag <meta name="robots"> ( http://www.robotstxt.org/meta.html ) on a page so that crawlers can obey page crawling rules along with the site's robot.txt.    
May 12, 2009 at 3:00 PM

This features will be avalaible for the next release.

May 16, 2009 at 10:57 PM

This feature is now avalaible