<!-- Configuration -->
<?xml version="1.0" encoding="utf-8" ?>
    <add key="RobotRulesUseCache" value="True"/>
    <add key="RobotRulesCacheLibrary" value="RobotRules.Cache.MemoryCache, RobotRules"/>
    <add key="RobotRulesCacheTimeout" value="00:01:00" />

Use the the parser

using RobotRules; 

var robotParser = new RobotsFileParser() 
                               LocalUserAgent = @"Mozilla/5.0 (compatible; Bluebot/1.0; +"

robotParser.Parse(new Uri(""));
if (robotParser.IsAllowed("Bluebot", new Uri ("")))
   // your code

Embedded robots control

If you want to deal with html embedded robots control (meta tag robots) you can use the library like this :
var strategy = robotParser.CheckRobotControlStrategy("Bluebot", "HTML CONTENT");

if (strategy.CanFollow)
    // your code
if (strategy.CanIndex)
    // your code

Deal with the cache

If you set RobotRulesUseCache to True you can clear the cache


The Dispose() method of RobotsFileParser always call the Dispose() method of the cache.

Last edited Jun 10, 2014 at 8:49 PM by teddyalbina, version 11


teddyalbina Jun 14, 2014 at 10:29 AM 
Version :
Fix cache initialization