org.archive.cdxserver.processor

Interface BaseProcessor

All Known Implementing Classes:

CDXListWriter, CDXToCaptureSearchResultsWriter, CDXToSearchResultWriter, CDXToUrlSearchResultWriter, CDXWriter, ClosestTimestampSorted, DupeCountProcessor, DupeTimestampBestStatusFilter, DupeTimestampLastBestStatusFilter, ForwardRevisitResolver, GroupCountProcessor, HttpCDXWriter, JsonWriter, LastNLineProcessor, MementoLinkWriter, PlainTextWriter, ReverseRevisitResolver, RevisitResolver, WrappedProcessor

public interface BaseProcessor

BaseProcessor is an interface for a receiver of CDXLines.

Implementation may be final output formatter (CDXWriter subclasses for example), or an intermediary processor that performs transformation and/or filtering on the sequence of CDXLines.

CDXServer starts from CDXWriter and builds up nested pipeline of BaseProcessors, and then calls following methods in sequence on the BaseProcessor at the top:

1. modifyOutputFormat(FieldSplitFormat)
2. begin()
3. for each CDXLine:
1. trackLine(CDXLine)
2. writeLine(CDXLine)
4. writeResumeKey(String) (if showResumeKey)
5. end()

Method Summary

Methods

Modifier and Type	Method and Description
void	begin() This method will be called just before looping over the sequence of CDX lines.
void	end() Called at the end.
org.archive.format.cdx.FieldSplitFormat	modifyOutputFormat(org.archive.format.cdx.FieldSplitFormat format) Return output format (list of fields), given input format format.
void	trackLine(org.archive.format.cdx.CDXLine line) Called on each CDX line, just before timestamp range filtering (to and from parameters), regexp filtering, and collapser processing.
int	writeLine(org.archive.format.cdx.CDXLine line) Process line.
void	writeResumeKey(String resumeKey) Write resumption key.

Method Detail

begin

void begin()

This method will be called just before looping over the sequence of CDX lines. Intermediary processor must call begin() on nested processor.

trackLine

void trackLine(org.archive.format.cdx.CDXLine line)

Called on each CDX line, just before timestamp range filtering (to and from parameters), regexp filtering, and collapser processing. Typically used for counting the number of CDX lines collapsed / grouped.

Parameters:

line - CDX line

writeLine

int writeLine(org.archive.format.cdx.CDXLine line)

Process line.

Parameters:

line - CDXLine

Returns:

1 if line is sent to output, 0 otherwise.

writeResumeKey

void writeResumeKey(String resumeKey)

Write resumption key. Only the final CDXWriter should do actual work. All intermediaries shall simply call writeResumeKey(resumeKey) on nested processor.

Parameters:

resumeKey -

end

void end()

Called at the end. Perform any clean ups / finalizations here. Intermediaries processor should call end() on nested processor.

modifyOutputFormat

org.archive.format.cdx.FieldSplitFormat modifyOutputFormat(org.archive.format.cdx.FieldSplitFormat format)

Return output format (list of fields), given input format format. Intermediaries should call modifyOutputFormat(format) on nested processor first, then make appropriate changes to it if they add/remove fields.

Parameters:

format - input format

Returns:

output format

See Also:

CDXFieldConstants