org.archive.wayback.resourceindex.ziplines

Class ZiplinesSearchResultSource

java.lang.Object

org.archive.wayback.resourceindex.ziplines.ZiplinesSearchResultSource

All Implemented Interfaces:

SearchResultSource

Deprecated.

Note, this implementation is now superceded by the one in archive-commons

public class ZiplinesSearchResultSource
extends Object
implements SearchResultSource

A set of Ziplines files, which are CDX files specially compressed into a series of GZipMembers such that: 1) each member is exactly 128K, padded using a GZip comment header 2) each member contains complete lines: no line spans two GZip members If the data put into these files is sorted, then the data within the files can be uncompressed when needed, minimizing the total data to be uncompressed This SearchResultSource assumes a set of alphabetically partitioned Ziplined CDX files, so that each file is sorted, and no regions overlap. This class takes 2 files as input: 1) a specially constructed map of the first N bytes of data from each GZip member, and the filename and offset of that GZip member. 2) a mapping of filenames to URLs Data from #1 is actually stored in a serialized

Author:

brad

See Also:

That implementation provides support for stream loading of blocks, as well as summary files from http and hdfs as well as local filesystem

Field Summary

Fields

Modifier and Type	Field and Description
protected int	timestampDedupLength Deprecated.

Constructor Summary

Constructors

Constructor and Description
ZiplinesSearchResultSource() Deprecated.
ZiplinesSearchResultSource(CDXFormat format) Deprecated.

Method Summary

Modifier and Type	Method and Description
protected org.archive.util.iterator.CloseableIterator<CaptureSearchResult>	adaptIterator(Iterator<String> itr) Deprecated.
void	cleanup(org.archive.util.iterator.CloseableIterator<CaptureSearchResult> c) Deprecated.
BlockLoader	getBlockLoader() Deprecated.
String	getChunkIndexPath() Deprecated.
String	getChunkMapPath() Deprecated.
CDXFormat	getFormat() Deprecated.
int	getMaxBlocks() Deprecated.
org.archive.util.iterator.CloseableIterator<CaptureSearchResult>	getPrefixIterator(String prefix) Deprecated.
org.archive.util.iterator.CloseableIterator<CaptureSearchResult>	getPrefixReverseIterator(String prefix) Deprecated.
Iterator<String>	getStringPrefixIterator(String prefix) Deprecated.
int	getTimestampDedupLength() Deprecated.
Iterator<String>	getZiplinesChunkIterator(String prefix, String urlkey) Deprecated.
void	init() Deprecated.
static void	main(String[] args) Deprecated.
void	setBlockLoader(BlockLoader blockLoader) Deprecated.
void	setChunkIndexPath(String chunkIndexPath) Deprecated.
void	setChunkMapPath(String chunkMapPath) Deprecated.
void	setFormat(CDXFormat format) Deprecated.
void	setMaxBlocks(int maxBlocks) Deprecated.
void	setTimestampDedupLength(int timestampDedupLength) Deprecated.
void	shutdown() Deprecated.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

timestampDedupLength

protected int timestampDedupLength

Deprecated.

Constructor Detail

ZiplinesSearchResultSource

public ZiplinesSearchResultSource()

Deprecated.

ZiplinesSearchResultSource

public ZiplinesSearchResultSource(CDXFormat format)

Deprecated.

Method Detail

getTimestampDedupLength

public int getTimestampDedupLength()

Deprecated.

setTimestampDedupLength

public void setTimestampDedupLength(int timestampDedupLength)

Deprecated.

init

public void init()
          throws IOException

Deprecated.

Throws:

IOException

adaptIterator

protected org.archive.util.iterator.CloseableIterator<CaptureSearchResult> adaptIterator(Iterator<String> itr)
                                                                                  throws IOException

Deprecated.

Throws:

IOException

cleanup

public void cleanup(org.archive.util.iterator.CloseableIterator<CaptureSearchResult> c)
             throws IOException

Deprecated.

Specified by:

cleanup in interface SearchResultSource

Throws:

IOException

getPrefixIterator

public org.archive.util.iterator.CloseableIterator<CaptureSearchResult> getPrefixIterator(String prefix)
                                                                                   throws ResourceIndexNotAvailableException

Deprecated.

Specified by:

getPrefixIterator in interface SearchResultSource

Returns:

CleanableIterator that will return SearchResults beginning with prefix argument, with subsequent next() calls returning subsequent results.

Throws:

ResourceIndexNotAvailableException

getZiplinesChunkIterator

public Iterator<String> getZiplinesChunkIterator(String prefix,
                                                 String urlkey)
                                          throws ResourceIndexNotAvailableException,
                                                 IOException

Deprecated.

Throws:

ResourceIndexNotAvailableException

IOException

getStringPrefixIterator

public Iterator<String> getStringPrefixIterator(String prefix)
                                         throws ResourceIndexNotAvailableException,
                                                IOException

Deprecated.

Throws:

ResourceIndexNotAvailableException

IOException

getPrefixReverseIterator

public org.archive.util.iterator.CloseableIterator<CaptureSearchResult> getPrefixReverseIterator(String prefix)
                                                                                          throws ResourceIndexNotAvailableException

Deprecated.

Specified by:

getPrefixReverseIterator in interface SearchResultSource

Returns:

CleanableIterator that will return SearchResults starting *before* prefix argument, with subsequent next() calls returning previous results.

Throws:

ResourceIndexNotAvailableException

shutdown

public void shutdown()
              throws IOException

Deprecated.

Specified by:

shutdown in interface SearchResultSource

Throws:

IOException

getFormat

public CDXFormat getFormat()

Deprecated.

Returns:

the format

setFormat

public void setFormat(CDXFormat format)

Deprecated.

Parameters:

format - the format to set

getChunkIndexPath

public String getChunkIndexPath()

Deprecated.

Returns:

the chunkIndexPath

setChunkIndexPath

public void setChunkIndexPath(String chunkIndexPath)

Deprecated.

Parameters:

chunkIndexPath - the chunkIndexPath to set

getChunkMapPath

public String getChunkMapPath()

Deprecated.

Returns:

the chunkMapPath

setChunkMapPath

public void setChunkMapPath(String chunkMapPath)

Deprecated.

Parameters:

chunkMapPath - the chunkMapPath to set

getMaxBlocks

public int getMaxBlocks()

Deprecated.

Returns:

the maxBlocks

setMaxBlocks

public void setMaxBlocks(int maxBlocks)

Deprecated.

Parameters:

maxBlocks - the maxBlocks to set

getBlockLoader

public BlockLoader getBlockLoader()

Deprecated.

Returns:

the blockLoader

setBlockLoader

public void setBlockLoader(BlockLoader blockLoader)

Deprecated.

Parameters:

blockLoader - the blockLoader to set

main

public static void main(String[] args)

Deprecated.

Parameters:

args -