org.archive.wayback.accesscontrol.robotstxt

Class Robotstxt

java.lang.Object

org.archive.wayback.accesscontrol.robotstxt.Robotstxt

All Implemented Interfaces:

Serializable

public class Robotstxt
extends Object
implements Serializable

Utility class for parsing and representing 'robots.txt' format directives, into a list of named user-agents and map from user-agents to RobotsDirectives.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static Robotstxt	NO_ROBOTS empty, reusable instance for all sites providing no rules

Constructor Summary

Constructors

Constructor and Description
Robotstxt()
Robotstxt(BufferedReader reader)
Robotstxt(org.archive.io.ReadSource customRobots)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	allowsAll() Does this policy effectively allow everything?
RobotsDirectives	getDirectivesFor(String userAgent) Return directives to use for the given User-Agent, resorting to wildcard rules or the default no-directives if necessary.
RobotsDirectives	getDirectivesFor(String ua, boolean useFallbacks) Return the RobotsDirectives, if any, appropriate for the given User-Agent string.
List<String>	getNamedUserAgents()
protected void	initializeFromReader(BufferedReader reader)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NO_ROBOTS

public static Robotstxt NO_ROBOTS

empty, reusable instance for all sites providing no rules

Constructor Detail

Robotstxt

public Robotstxt()

Robotstxt

public Robotstxt(BufferedReader reader)
          throws IOException

Throws:

IOException

Robotstxt

public Robotstxt(org.archive.io.ReadSource customRobots)

Method Detail

initializeFromReader

protected void initializeFromReader(BufferedReader reader)
                             throws IOException

Throws:

IOException

allowsAll

public boolean allowsAll()

Does this policy effectively allow everything? (No disallows or timing (crawl-delay) directives?)

Returns:

true if the map containing user-agents to directives is empty, false otherwise.

getNamedUserAgents

public List<String> getNamedUserAgents()

getDirectivesFor

public RobotsDirectives getDirectivesFor(String ua,
                                boolean useFallbacks)

Return the RobotsDirectives, if any, appropriate for the given User-Agent string. If useFallbacks is true, a wildcard ('*') directives or the default of NO_DIRECTIVES will be returned, as appropriate, if there is no better match. If useFallbacks is false, a null will be returned if no declared directives targeted the given User-Agent.

Parameters:

ua - String User-Agent to lookup

useFallbacks - if true, fall-back to wildcard directives or default allow as needed

Returns:

directives to use, or null if useFallbacks is false and no non-wildcard directives match the supplied User-Agent

getDirectivesFor

public RobotsDirectives getDirectivesFor(String userAgent)

Return directives to use for the given User-Agent, resorting to wildcard rules or the default no-directives if necessary.

Parameters:

userAgent - String User-Agent to lookup

Returns:

directives to use