Package | Description |
---|---|
org.archive.wayback.accesscontrol.robotstxt |
Modifier and Type | Class and Description |
---|---|
class |
FixedRobotsDirectives |
Modifier and Type | Method and Description |
---|---|
RobotsDirectives |
Robotstxt.getDirectivesFor(String userAgent)
Return directives to use for the given User-Agent, resorting to wildcard
rules or the default no-directives if necessary.
|
RobotsDirectives |
Robotstxt.getDirectivesFor(String ua,
boolean useFallbacks)
Return the RobotsDirectives, if any, appropriate for the given User-Agent
string.
|
Modifier and Type | Method and Description |
---|---|
void |
RobotsDirectiveAggregation.addDirectives(String url,
RobotsDirectives directives) |
Copyright © 2005–2015 IIPC. All rights reserved.