| Class | Description |
|---|---|
| AccessControlChecker | |
| FixedRobotsDirectives | |
| HRobotExclusionFilter | |
| RobotExclusionFilter |
CaptureSearchResult Filter that uses a LiveWebCache to retrieve robots.txt
documents from the live web, and filters SearchResults based on the rules
therein.
|
| RobotExclusionFilterFactory | |
| RobotRules |
Class which parses a robots.txt file, storing the rules contained therein,
and then allows for testing if path/userAgent tuples are blocked by those
rules.
|
| RobotsDirectiveAggregation |
Class which acts as an aggregation of RobotsDirectives.
|
| RobotsDirectives |
Represents the directives that apply to a user-agent (or set of
user-agents)
|
| Robotstxt |
Utility class for parsing and representing 'robots.txt' format
directives, into a list of named user-agents and map from user-agents
to RobotsDirectives.
|
Copyright © 2005–2015 IIPC. All rights reserved.