1
Vote

Policy enhancement

description

Replace
bool[] policy = RobotRules.CheckHtmlRobotMetaTag("Name of your crawler", Html);
 
By something link :
 
Policy = RobotRules.CheckHtmlRobotMetaTag("Name of your crawler", Html);
 
Policy
{
CanFollow,
CanIndex
}

comments