org.archive.wayback.accesscontrol.robotstxt

Class RobotsDirectives

java.lang.Object

org.archive.wayback.accesscontrol.robotstxt.RobotsDirectives

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

FixedRobotsDirectives

public class RobotsDirectives
extends Object
implements Serializable

Represents the directives that apply to a user-agent (or set of user-agents)

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
RobotsDirectives()

Method Summary

Methods

Modifier and Type	Method and Description
void	addAllow(String path)
void	addDisallow(String path)
boolean	allows(String path)
float	getCrawlDelay()
protected int	longestPrefixLength(ConcurrentSkipListSet<String> prefixSet, String str)
void	setCrawlDelay(float i)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RobotsDirectives

public RobotsDirectives()

Method Detail

allows

public boolean allows(String path)

longestPrefixLength

protected int longestPrefixLength(ConcurrentSkipListSet<String> prefixSet,
                      String str)

Parameters:

prefixSet -

str -

Returns:

length of longest entry in prefixSet that prefixes str, or zero if no entry prefixes str

addDisallow

public void addDisallow(String path)

addAllow

public void addAllow(String path)

setCrawlDelay

public void setCrawlDelay(float i)

getCrawlDelay

public float getCrawlDelay()