meta tag for robots

Mar 22, 2009 at 12:47 AM
great job...it would also be nice to have something that can also parse the tag <meta name="robots"> ( http://www.robotstxt.org/meta.html ) on a page so that crawlers can obey page crawling rules along with the site's robot.txt.    
Coordinator
May 12, 2009 at 2:00 PM

This features will be avalaible for the next release.

Coordinator
May 16, 2009 at 9:57 PM

This feature is now avalaible