org.archive.wayback.resourceindex.cdxserver

Class EmbeddedCDXServerIndex

java.lang.Object

org.archive.wayback.util.webapp.AbstractRequestHandler

org.archive.wayback.resourceindex.cdxserver.EmbeddedCDXServerIndex

All Implemented Interfaces:

MementoHandler, ResourceIndex, RequestHandler, org.springframework.beans.factory.BeanNameAware

public class EmbeddedCDXServerIndex
extends AbstractRequestHandler
implements MementoHandler, ResourceIndex

ResourceIndex on top of CDXServer. Also a RequestHandler for CDX server queries.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	baseStatusFilter
protected String	baseStatusRegexp
protected UrlCanonicalizer	canonicalizer
protected CDXServer	cdxServer
protected org.archive.format.cdx.CDXInputSource	extraSource
protected List<String>	ignoreRobotPaths
protected int	limit
protected String	preferContains
protected String	remoteCdxPath
static String	REQUEST_REVISIT_LOOKUP WaybackRequest parameter name for telling EmbeddedCDXServerIndex that it's looking up a specific single capture needed for replaying URL-agnostic revisit.
protected SelfRedirectFilter	selfRedirFilter
protected int	timestampDedupLength
protected boolean	tryFuzzyMatch

Constructor Summary

Constructors

Constructor and Description
EmbeddedCDXServerIndex()

Method Summary

Methods

Modifier and Type	Method and Description
void	addTimegateHeaders(javax.servlet.http.HttpServletResponse response, CaptureSearchResults results, WaybackRequest wbRequest, boolean includeOriginal)
protected static String	buildStatusFilter(String regexp)
protected AuthToken	createAuthToken(WaybackRequest wbRequest, String urlkey) robots.txt may be ignored for embedded resources (CSS, images, javascripts) robots.txt may be ignored if urlkey starts with any of ignoreRobotPaths
protected CDXQuery	createQuery(WaybackRequest wbRequest, boolean isFuzzy) Create CDXQuery that is sent to CDXServer.
protected org.archive.util.iterator.CloseableIterator<String>	createRemoteIter(String urlkey, org.archive.util.binsearch.impl.HTTPSeekableLineReader reader)
SearchResults	doQuery(WaybackRequest wbRequest)
String	getBaseStatusRegexp()
UrlCanonicalizer	getCanonicalizer()
protected CDXToCaptureSearchResultsWriter	getCaptureSearchWriter(WaybackRequest wbRequest, AuthToken waybackAuthToken, boolean isFuzzy) create CDXWriter for writing capture search result.
CDXServer	getCdxServer()
org.archive.format.cdx.CDXInputSource	getExtraSource()
List<String>	getIgnoreRobotPaths()
int	getLimit()
String	getPreferContains()
String	getRemoteAuthCookie()
String	getRemoteAuthCookieIgnoreRobots()
org.archive.util.binsearch.impl.HTTPSeekableLineReaderFactory	getRemoteCdxHttp()
String	getRemoteCdxPath()
SelfRedirectFilter	getSelfRedirFilter()
int	getTimestampDedupLength()
protected CDXToSearchResultWriter	getUrlSearchWriter(WaybackRequest wbRequest)
boolean	handleRequest(javax.servlet.http.HttpServletRequest httpRequest, javax.servlet.http.HttpServletResponse httpResponse) Possibly handle an incoming HttpServletRequest, much like a normal HttpServlet, but includes a return value.
boolean	isTryFuzzyMatch()
protected void	loadWaybackCdx(String urlkey, WaybackRequest wbRequest, CDXQuery query, AuthToken waybackAuthToken, CDXToSearchResultWriter resultWriter, boolean fuzzy)
SearchResults	query(WaybackRequest wbRequest) Transform a WaybackRequest into a ResourceResults.
protected void	remoteCdxServerQuery(String urlkey, CDXQuery query, AuthToken authToken, CDXToSearchResultWriter resultWriter)
boolean	renderMementoTimemap(WaybackRequest wbRequest, javax.servlet.http.HttpServletRequest request, javax.servlet.http.HttpServletResponse response)
void	setBaseStatusRegexp(String baseStatusRegexp) filter on statuscode field applied by default for interactive CDX lookup (i.e. from Wayback UI, not via CDX Server API).
void	setCanonicalizer(UrlCanonicalizer canonicalizer)
void	setCdxServer(CDXServer cdxServer)
void	setExtraSource(org.archive.format.cdx.CDXInputSource extraSource)
void	setIgnoreRobotPaths(List<String> ignoreRobotPaths)
void	setLimit(int limit)
void	setPreferContains(String preferContains) substring of filename field identifying preferred archive among multiple copies of the same capture.
void	setRemoteAuthCookie(String remoteAuthCookie)
void	setRemoteAuthCookieIgnoreRobots(String remoteAuthCookieIgnoreRobots)
void	setRemoteCdxHttp(org.archive.util.binsearch.impl.HTTPSeekableLineReaderFactory remoteCdxHttp)
void	setRemoteCdxPath(String remoteCdxPath)
void	setSelfRedirFilter(SelfRedirectFilter selfRedirFilter)
void	setTimestampDedupLength(int timestampDedupLength) The number of digits of timestamp used for culling (deduplicating) captures in CDX query result.
void	setTryFuzzyMatch(boolean tryFuzzyMatch)
void	shutdown() Release any resources used by this ResourceIndex cleanly

Methods inherited from class org.archive.wayback.util.webapp.AbstractRequestHandler

getAccessPointPath, getBeanName, getInternalPort, getMapParam, getMapParamOrEmpty, getRequiredMapParam, getServletContext, registerPortListener, setAccessPointPath, setBeanName, setInternalPort, setServletContext, translateRequestPath, translateRequestPathQuery

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

cdxServer

protected CDXServer cdxServer

timestampDedupLength

protected int timestampDedupLength

limit

protected int limit

canonicalizer

protected UrlCanonicalizer canonicalizer

selfRedirFilter

protected SelfRedirectFilter selfRedirFilter

remoteCdxPath

protected String remoteCdxPath

extraSource

protected org.archive.format.cdx.CDXInputSource extraSource

preferContains

protected String preferContains

tryFuzzyMatch

protected boolean tryFuzzyMatch

ignoreRobotPaths

protected List<String> ignoreRobotPaths

baseStatusRegexp

protected String baseStatusRegexp

baseStatusFilter

protected String baseStatusFilter

REQUEST_REVISIT_LOOKUP

public static final String REQUEST_REVISIT_LOOKUP

WaybackRequest parameter name for telling EmbeddedCDXServerIndex that it's looking up a specific single capture needed for replaying URL-agnostic revisit.

Defined here, without setter/getter, because this is an experimental parameter supporting soft-block feature. It's very likely to change.

See Also:

Constant Field Values

Constructor Detail

EmbeddedCDXServerIndex

public EmbeddedCDXServerIndex()

Method Detail

query

public SearchResults query(WaybackRequest wbRequest)
                    throws ResourceIndexNotAvailableException,
                           ResourceNotInArchiveException,
                           BadQueryException,
                           AccessControlException

Description copied from interface: ResourceIndex

Transform a WaybackRequest into a ResourceResults.

Specified by:

query in interface ResourceIndex

Parameters:

wbRequest - WaybackRequest object from RequestParser

Returns:

SearchResults containing SearchResult objects matching the WaybackRequest

Throws:

ResourceIndexNotAvailableException - if the ResourceIndex is not available (remote host down, local files missing, etc)

ResourceNotInArchiveException - if the ResourceIndex could be contacted, but no SearchResult objects matched the request

BadQueryException - if the WaybackRequest is lacking information required to make a reasonable search of this ResourceIndex

AccessControlException - if SearchResult objects actually matched, but could not be returned due to AccessControl restrictions (robots.txt documents, Administrative URL blocks, etc)

createAuthToken

protected AuthToken createAuthToken(WaybackRequest wbRequest,
                        String urlkey)

• robots.txt may be ignored for embedded resources (CSS, images, javascripts)
• robots.txt may be ignored if urlkey starts with any of ignoreRobotPaths

Parameters:

wbRequest -

urlkey -

Returns:

AuthToken representing user's privileges on urlkey.

doQuery

public SearchResults doQuery(WaybackRequest wbRequest)
                      throws ResourceIndexNotAvailableException,
                             ResourceNotInArchiveException,
                             BadQueryException,
                             AccessControlException

Throws:

ResourceIndexNotAvailableException

ResourceNotInArchiveException

BadQueryException

AccessControlException

loadWaybackCdx

protected void loadWaybackCdx(String urlkey,
                  WaybackRequest wbRequest,
                  CDXQuery query,
                  AuthToken waybackAuthToken,
                  CDXToSearchResultWriter resultWriter,
                  boolean fuzzy)
                       throws IOException,
                              AccessControlException

Throws:

IOException

AccessControlException

createQuery

protected CDXQuery createQuery(WaybackRequest wbRequest,
                   boolean isFuzzy)

Create CDXQuery that is sent to CDXServer. The query specifies standard CDX server params described at: https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server Note: this method adds extra filters meant for interactive (Wayback UI) use. CDXServer web API should not use this method. this method is used for replay and capture-search requests only. TODO: move this to CDXQuery as static method.

Parameters:

wbRequest - WaybackRequest either replay or capture-query

isFuzzy - unused (?)

Returns:

A CDXQuery object for the supplied request.

remoteCdxServerQuery

protected void remoteCdxServerQuery(String urlkey,
                        CDXQuery query,
                        AuthToken authToken,
                        CDXToSearchResultWriter resultWriter)
                             throws IOException,
                                    AccessControlException

Throws:

IOException

AccessControlException

createRemoteIter

protected org.archive.util.iterator.CloseableIterator<String> createRemoteIter(String urlkey,
                                                                   org.archive.util.binsearch.impl.HTTPSeekableLineReader reader)
                                                                        throws IOException

Throws:

IOException

getCaptureSearchWriter

protected CDXToCaptureSearchResultsWriter getCaptureSearchWriter(WaybackRequest wbRequest,
                                                     AuthToken waybackAuthToken,
                                                     boolean isFuzzy)

create CDXWriter for writing capture search result.

possible future changes:

• drop unused argument waybackAuthToken
• change return type to super class (as far up as appropriate)

Parameters:

wbRequest - WaybackRequest for configuring CDXQuery

waybackAuthToken - unused

isFuzzy - true to enable fuzzy query

Returns:

CDXCaptureSearchResultWriter

getUrlSearchWriter

protected CDXToSearchResultWriter getUrlSearchWriter(WaybackRequest wbRequest)

renderMementoTimemap

public boolean renderMementoTimemap(WaybackRequest wbRequest,
                           javax.servlet.http.HttpServletRequest request,
                           javax.servlet.http.HttpServletResponse response)
                             throws WaybackException,
                                    IOException

Specified by:

renderMementoTimemap in interface MementoHandler

Throws:

WaybackException

IOException

handleRequest

public boolean handleRequest(javax.servlet.http.HttpServletRequest httpRequest,
                    javax.servlet.http.HttpServletResponse httpResponse)
                      throws javax.servlet.ServletException,
                             IOException

Description copied from interface: RequestHandler

Possibly handle an incoming HttpServletRequest, much like a normal HttpServlet, but includes a return value.

Specified by:

handleRequest in interface RequestHandler

Parameters:

httpRequest - the incoming HttpServletRequest

httpResponse - the HttpServletResponse to return data to the client.

Returns:

true if the RequestHandler returned a response to the client, false otherwise

Throws:

javax.servlet.ServletException - for usual reasons.

IOException - for usual reasons.

shutdown

public void shutdown()
              throws IOException

Description copied from interface: ResourceIndex

Release any resources used by this ResourceIndex cleanly

Specified by:

shutdown in interface ResourceIndex

Throws:

IOException - for usual causes

getCdxServer

public CDXServer getCdxServer()

setCdxServer

public void setCdxServer(CDXServer cdxServer)

getTimestampDedupLength

public int getTimestampDedupLength()

setTimestampDedupLength

public void setTimestampDedupLength(int timestampDedupLength)

The number of digits of timestamp used for culling (deduplicating) captures in CDX query result.

For example, with this property set to 11, {#query} will return at most only one captures within each 10 minutes span.

Non-positive value or 14 disables deduplication.

Note: deduplication is done by CDXServer. ZipNumIndex also implements timestamp-deduplication, which can be turned on by setting positive value to defaultParams.timestampDedupLength. It is recommended to leave this off and use this parameter only, for several reasons:

• ZipNumIndex's collapsing works at ZipNum block level. It works only when captures of single URL spans multiple ZipNum blocks, and can produce unexpected result if ever worked.
• ZipNumIndex's collapsing is closed to single index cluster. it can produce confusing result when multiple index clusters are combined.
• There's no point running timestamp-collapsing more than once (although the intent of ZipNumIndex's collapsing may be to quickly cut down on the number of results from URL with many many captures.)

Note now it is possible to pass collapseTime parameter to EmbeddedCDXServerIndex#query, and this timestampDedupLength parameter serves as a default, used only when collapseTime is unspecified. See WaybackRequest.setCollapseTime(int).

Parameters:

timestampDedupLength - the number of digits of timestamp used for deduplication.

See Also:

ZipNumParams.setTimestampDedupLength(int), CDXServer.writeCdxResponse(org.archive.cdxserver.writer.CDXWriter, org.archive.util.iterator.CloseableIterator<java.lang.String>, int, org.archive.cdxserver.CDXQuery, org.archive.cdxserver.auth.AuthToken, org.archive.cdxserver.filter.CDXAccessFilter), WaybackRequest.setCollapseTime(int)

getSelfRedirFilter

public SelfRedirectFilter getSelfRedirFilter()

setSelfRedirFilter

public void setSelfRedirFilter(SelfRedirectFilter selfRedirFilter)

getCanonicalizer

public UrlCanonicalizer getCanonicalizer()

setCanonicalizer

public void setCanonicalizer(UrlCanonicalizer canonicalizer)

getLimit

public int getLimit()

setLimit

public void setLimit(int limit)

addTimegateHeaders

public void addTimegateHeaders(javax.servlet.http.HttpServletResponse response,
                      CaptureSearchResults results,
                      WaybackRequest wbRequest,
                      boolean includeOriginal)

Specified by:

addTimegateHeaders in interface MementoHandler

getRemoteCdxPath

public String getRemoteCdxPath()

setRemoteCdxPath

public void setRemoteCdxPath(String remoteCdxPath)

getRemoteAuthCookie

public String getRemoteAuthCookie()

setRemoteAuthCookie

public void setRemoteAuthCookie(String remoteAuthCookie)

getRemoteAuthCookieIgnoreRobots

public String getRemoteAuthCookieIgnoreRobots()

setRemoteAuthCookieIgnoreRobots

public void setRemoteAuthCookieIgnoreRobots(String remoteAuthCookieIgnoreRobots)

getRemoteCdxHttp

public org.archive.util.binsearch.impl.HTTPSeekableLineReaderFactory getRemoteCdxHttp()

setRemoteCdxHttp

public void setRemoteCdxHttp(org.archive.util.binsearch.impl.HTTPSeekableLineReaderFactory remoteCdxHttp)

getExtraSource

public org.archive.format.cdx.CDXInputSource getExtraSource()

setExtraSource

public void setExtraSource(org.archive.format.cdx.CDXInputSource extraSource)

getPreferContains

public String getPreferContains()

setPreferContains

public void setPreferContains(String preferContains)

substring of filename field identifying preferred archive among multiple copies of the same capture.

Parameters:

preferContains -

See Also:

CDXToCaptureSearchResultsWriter

getIgnoreRobotPaths

public List<String> getIgnoreRobotPaths()

setIgnoreRobotPaths

public void setIgnoreRobotPaths(List<String> ignoreRobotPaths)

isTryFuzzyMatch

public boolean isTryFuzzyMatch()

setTryFuzzyMatch

public void setTryFuzzyMatch(boolean tryFuzzyMatch)

getBaseStatusRegexp

public String getBaseStatusRegexp()

setBaseStatusRegexp

public void setBaseStatusRegexp(String baseStatusRegexp)

filter on statuscode field applied by default for interactive CDX lookup (i.e. from Wayback UI, not via CDX Server API).

Value is a regular expression for status code field. Only those CDXes with matching statuscode field will be returned. Leading/Trailing spaces are stripped off. If value starts with "!", only CDXes with unmatching statuscode field will be returned (exception: value "!" is treated as empty string, i.e. no filtering).

Value will be ignored if WybackRequest.isBestLatestReplayRequest is set, for which hard-coded value "[23].." is used.

Default value is "!(500|502|504)".

NOTE: this is a quick hack to allow for customizing replay/listing of 5xx captures. it may be replaced by different customization method, or moved to other class in the future.

Parameters:

baseStatusRegexp - regular expression for status code.

buildStatusFilter

protected static String buildStatusFilter(String regexp)