org.archive.wayback.resourceindex.cdxserver

Class CDXToCaptureSearchResultsWriter

java.lang.Object

org.archive.cdxserver.writer.CDXWriter

org.archive.wayback.resourceindex.cdxserver.CDXToSearchResultWriter

org.archive.wayback.resourceindex.cdxserver.CDXToCaptureSearchResultsWriter

All Implemented Interfaces:

BaseProcessor

public class CDXToCaptureSearchResultsWriter
extends CDXToSearchResultWriter

CDXToSearchResultWriter for producing CaptureSearchResults.

Also resolves revisits and sets closest.

Field Summary

Fields

Modifier and Type	Field and Description
protected CaptureSearchResult	closest
protected HashMap<String,CaptureSearchResult>	digestToOriginal
protected HashMap<String,LinkedList<CaptureSearchResult>>	digestToRevisits
protected boolean	done
protected ExclusionFilter	exclusionFilter
protected int	flip
protected boolean	includeBlockedCaptures
protected boolean	isReverse
protected String	preferContains
protected org.archive.format.cdx.CDXLine	prevLine
protected CaptureSearchResult	prevResult
protected boolean	resolveRevisits
protected CaptureSearchResults	results
static String	REVISIT_VALUE
protected boolean	seekSingleCapture
protected SelfRedirectFilter	selfRedirFilter
protected String	targetTimestamp

Fields inherited from class org.archive.wayback.resourceindex.cdxserver.CDXToSearchResultWriter

msg, query

Constructor Summary

Constructors

Constructor and Description
CDXToCaptureSearchResultsWriter(CDXQuery query, boolean resolveRevisits, boolean seekSingleCapture, String preferContains) Initialize with CDXQuery and other options.

Method Summary

Modifier and Type	Method and Description
void	begin() This method will be called just before looping over the sequence of CDX lines.
protected CaptureSearchResult	determineClosest(CaptureSearchResult nextResult)
void	end() Called at the end.
CaptureSearchResult	getClosest()
ExclusionFilter	getExclusionFilter() Deprecated.
protected CaptureSearchResult	getLastAdded()
CaptureSearchResults	getSearchResults()
SelfRedirectFilter	getSelfRedirFilter()
boolean	isAborted()
boolean	isIncludeBlockedCaptures()
void	setExclusionFilter(ExclusionFilter exclusionFilter) Deprecated. 2014-11-10 Use new implementation AccessPoint.setExclusionFactory(org.archive.wayback.accesscontrol.ExclusionFilterFactory)
void	setIncludeBlockedCaptures(boolean includeBlockedCaptures) set to true if blocked captures are to be included in the result.
void	setSelfRedirFilter(SelfRedirectFilter selfRedirFilter)
void	setTargetTimestamp(String timestamp)
int	writeLine(org.archive.format.cdx.CDXLine line) Process line.

Methods inherited from class org.archive.wayback.resourceindex.cdxserver.CDXToSearchResultWriter

getErrorMsg, getQuery, modifyOutputFormat, printError, writeResumeKey

Methods inherited from class org.archive.cdxserver.writer.CDXWriter

close, printNumPages, serverError, setContentType, setMaxLines, trackLine, writeMiscLine

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

REVISIT_VALUE

public static final String REVISIT_VALUE

See Also:

Constant Field Values

results

protected CaptureSearchResults results

targetTimestamp

protected String targetTimestamp

flip

protected int flip

done

protected boolean done

closest

protected CaptureSearchResult closest

selfRedirFilter

protected SelfRedirectFilter selfRedirFilter

exclusionFilter

protected ExclusionFilter exclusionFilter

prevResult

protected CaptureSearchResult prevResult

prevLine

protected org.archive.format.cdx.CDXLine prevLine

digestToOriginal

protected HashMap<String,CaptureSearchResult> digestToOriginal

digestToRevisits

protected HashMap<String,LinkedList<CaptureSearchResult>> digestToRevisits

resolveRevisits

protected boolean resolveRevisits

seekSingleCapture

protected boolean seekSingleCapture

isReverse

protected boolean isReverse

preferContains

protected String preferContains

includeBlockedCaptures

protected boolean includeBlockedCaptures

Constructor Detail

CDXToCaptureSearchResultsWriter

public CDXToCaptureSearchResultsWriter(CDXQuery query,
                                       boolean resolveRevisits,
                                       boolean seekSingleCapture,
                                       String preferContains)

Initialize with CDXQuery and other options.

This class generates CaptureSearchResult in chronological order, even when CDXQuery.isReverse() is true.

Note: preferContains parameter is specifically intended for choosing one out of two copies of the identical capture record in different storage locations. For example, If WARCs in staging area are made available for replay through secondary index, there may be a period where one capture is indexed in both main and secondary index, with different filename field. If preferContains is set, CDX line that has preferContains as substring in filename will be picked over others that does not. It can be used, for example, to put higher preference on the archive in primary storage area.

Parameters:

query - CDXQuery

resolveRevisits - Whether to resolve revisit captures

seekSingleCapture - Whether just one capture is wanted. (Only effective when resolveRevisits is also true.)

preferContains - Preferred archive filename substring. If non-null, It picks capture in the archive with a given substring in its filename, out of multiple captures of the same timestamp, original URL, length and offset (if any).

Method Detail

setTargetTimestamp

public void setTargetTimestamp(String timestamp)

begin

public void begin()

Description copied from interface: BaseProcessor

This method will be called just before looping over the sequence of CDX lines. Intermediary processor must call begin() on nested processor.

Specified by:

begin in interface BaseProcessor

Overrides:

begin in class CDXToSearchResultWriter

writeLine

public int writeLine(org.archive.format.cdx.CDXLine line)

Description copied from interface: BaseProcessor

Process line.

Parameters:

line - CDXLine

Returns:

1 if line is sent to output, 0 otherwise.

isAborted

public boolean isAborted()

Overrides:

isAborted in class CDXWriter

determineClosest

protected CaptureSearchResult determineClosest(CaptureSearchResult nextResult)

end

public void end()

Description copied from interface: BaseProcessor

Called at the end. Perform any clean ups / finalizations here. Intermediaries processor should call end() on nested processor.

Specified by:

end in interface BaseProcessor

Overrides:

end in class CDXToSearchResultWriter

getClosest

public CaptureSearchResult getClosest()

getLastAdded

protected CaptureSearchResult getLastAdded()

getSearchResults

public CaptureSearchResults getSearchResults()

Specified by:

getSearchResults in class CDXToSearchResultWriter

getSelfRedirFilter

public SelfRedirectFilter getSelfRedirFilter()

setSelfRedirFilter

public void setSelfRedirFilter(SelfRedirectFilter selfRedirFilter)

getExclusionFilter

@Deprecated
public ExclusionFilter getExclusionFilter()

Deprecated.

setExclusionFilter

public void setExclusionFilter(ExclusionFilter exclusionFilter)

Deprecated. 2014-11-10 Use new implementation AccessPoint.setExclusionFactory(org.archive.wayback.accesscontrol.ExclusionFilterFactory)

If non-null, the filter will be applied before revisit resolution.

Note: there is no class using this property in baseline Wayback. You need to write a custom class to utilize this property. See CDXServer and LocalResourceIndex for other ways of configuring exclusion filters.

This method is deprecated because this can run exclusion after timestamp deduplication, which results in undesirable capture search results. Exclusion should happen in regular CDXServer pipeline. This method was necessary to implement collection sensitive exclusion filter. New exclusion filter factory addresses such needs in ordinary CDX filtering pipeline.

Parameters:

exclusionFilter -

See Also:

CDXServer, LocalResourceIndex, AuthChecker.createAccessFilter(org.archive.cdxserver.auth.AuthToken), ExclusionCaptureFilterGroup.ExclusionCaptureFilterGroup(org.archive.wayback.core.WaybackRequest, org.archive.wayback.UrlCanonicalizer)

isIncludeBlockedCaptures

public boolean isIncludeBlockedCaptures()

setIncludeBlockedCaptures

public void setIncludeBlockedCaptures(boolean includeBlockedCaptures)

set to true if blocked captures are to be included in the result.

This is a tentative property and specifically intended for looking up revisit original for URL-agnostic revisits. May change in the future.

Parameters:

includeBlockedCaptures -