org.archive.wayback.resourceindex.distributed

Class AlphaPartitionedIndex

java.lang.Object

org.archive.wayback.resourceindex.distributed.AlphaPartitionedIndex

All Implemented Interfaces:

ResourceIndex

public class AlphaPartitionedIndex
extends Object
implements ResourceIndex

ResourceIndex which delegates queries to multiple partitioned remote indexes.

The remote index can be duplicated for failover and load balancing. Configuration is done with a map file which contains the range for each remote index partition and the urls to all instances of the partition.

Format of mapfile:

 <partition name> <range start> <range end> <url to index partition> [<url to alternative index partition> ...]
 

Example:

 part_a  apple.com/ http://part_a1.foo.com http://part_a2.foo.com
 part_b apple.com/ banana.com/ http://part_b1.foo.com http://part_b2.foo.com
 part_c banana.com/ cups.com/ http://part_c1.foo.com http://part_c2.foo.com
 part_d cups.com/ zorro.com/ http://part_d1.foo.com http://part_d2.foo.com
 

Author:

brad

See Also:

RemoteResourceIndex

Field Summary

Fields

Modifier and Type	Field and Description
static String	RANGE_CHECK_INTERVAL config name for interval, in seconds, to check if the map file changed
static String	RANGE_MAP_PATH config name for path where map file is found

Constructor Summary

Constructors

Constructor and Description
AlphaPartitionedIndex()

Method Summary

Methods

Modifier and Type	Method and Description
String	canonicalize(String url)
UrlCanonicalizer	getCanonicalizer()
long	getCheckInterval()
String	getMapPath()
protected RangeGroup	getRangeGroupForRequest(WaybackRequest wbRequest)
SearchResults	query(WaybackRequest wbRequest) Transform a WaybackRequest into a ResourceResults.
void	setCanonicalizer(UrlCanonicalizer canonicalizer)
void	setCheckInterval(long checkInterval)
void	setMapPath(String mapPath)
void	shutdown() Release any resources used by this ResourceIndex cleanly

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

RANGE_MAP_PATH

public static String RANGE_MAP_PATH

config name for path where map file is found

RANGE_CHECK_INTERVAL

public static String RANGE_CHECK_INTERVAL

config name for interval, in seconds, to check if the map file changed

Constructor Detail

AlphaPartitionedIndex

public AlphaPartitionedIndex()

Method Detail

getRangeGroupForRequest

protected RangeGroup getRangeGroupForRequest(WaybackRequest wbRequest)
                                      throws BadQueryException,
                                             ResourceIndexNotAvailableException

Throws:

BadQueryException

ResourceIndexNotAvailableException

query

public SearchResults query(WaybackRequest wbRequest)
                    throws ResourceIndexNotAvailableException,
                           ResourceNotInArchiveException,
                           BadQueryException,
                           AccessControlException

Description copied from interface: ResourceIndex

Transform a WaybackRequest into a ResourceResults.

Specified by:

query in interface ResourceIndex

Parameters:

wbRequest - WaybackRequest object from RequestParser

Returns:

SearchResults containing SearchResult objects matching the WaybackRequest

Throws:

ResourceIndexNotAvailableException - if the ResourceIndex is not available (remote host down, local files missing, etc)

ResourceNotInArchiveException - if the ResourceIndex could be contacted, but no SearchResult objects matched the request

BadQueryException - if the WaybackRequest is lacking information required to make a reasonable search of this ResourceIndex

AccessControlException - if SearchResult objects actually matched, but could not be returned due to AccessControl restrictions (robots.txt documents, Administrative URL blocks, etc)

canonicalize

public String canonicalize(String url)
                    throws org.apache.commons.httpclient.URIException

Parameters:

url -

Returns:

canonicalized key version of url argument

Throws:

org.apache.commons.httpclient.URIException

getCheckInterval

public long getCheckInterval()

Returns:

the checkInterval

setCheckInterval

public void setCheckInterval(long checkInterval)

Parameters:

checkInterval - the checkInterval to set

getMapPath

public String getMapPath()

Returns:

the mapPath

setMapPath

public void setMapPath(String mapPath)

Parameters:

mapPath - the mapPath to set

getCanonicalizer

public UrlCanonicalizer getCanonicalizer()

setCanonicalizer

public void setCanonicalizer(UrlCanonicalizer canonicalizer)

shutdown

public void shutdown()
              throws IOException

Description copied from interface: ResourceIndex

Release any resources used by this ResourceIndex cleanly

Specified by:

shutdown in interface ResourceIndex

Throws:

IOException - for usual causes