org.archive.io.warc

Class TestWARCReader

java.lang.Object

org.archive.io.ArchiveReader

org.archive.io.warc.TestWARCReader

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<org.archive.io.ArchiveRecord>, org.archive.format.ArchiveFileConstants

public class TestWARCReader
extends org.archive.io.ArchiveReader

Fixture WARCReader.

It works as ArchiveReader reading from WARC file with just one WARC record at offset 0 (there's no "warcinfo" record).

Content of the record is customized through WARCRecordInfo. (TestWARCRecordInfo offers commonly-used default values and convenient factory methods.

Typical test code would be:

 String payload = "hogehogehogehogehoge";
 WARCRecordInfo recinfo = TestWARCRecordInfo.createHttpResponse(payload);
 TestWARCReader ar = new TestWARCReader(recinfo);
 WARCRecord rec = (WARCRecord)ar.get(0);
 

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.io.ArchiveReader

org.archive.io.ArchiveReader.ArchiveRecordIterator

Field Summary

Fields

Modifier and Type	Field and Description
static String	CRLF

Fields inherited from class org.archive.io.ArchiveReader

in, MAX_ALLOWED_RECOVERABLES

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DOT_COMPRESSED_FILE_EXTENSION, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Constructor and Description
TestWARCReader(InputStream is)
TestWARCReader(org.archive.io.warc.WARCRecordInfo recinfo)

Method Summary

Methods

Modifier and Type	Method and Description
static InputStream	buildRecordContent(org.archive.io.warc.WARCRecordInfo recinfo) build minimal WARC record byte stream.
protected org.archive.io.warc.WARCRecord	createArchiveRecord(InputStream is, long offset)
void	dump(boolean compress)
org.archive.io.warc.WARCRecord	get(long offset)
org.archive.io.ArchiveReader	getDeleteFileOnCloseReader(File f)
String	getDotFileExtension()
String	getFileExtension()
protected void	gotoEOR(org.archive.io.ArchiveRecord record)

Methods inherited from class org.archive.io.ArchiveReader

cdxOutput, cleanupCurrentRecord, close, currentRecord, get, getCurrentRecord, getFileName, getIn, getInputStream, getLogger, getOptions, getReaderIdentifier, getStrippedFileName, getStrippedFileName, getTrueOrFalse, getVersion, initialize, isCompressed, isDigest, isStrict, isValid, iterator, logStdErr, output, outputRecord, outputRecord, positionForRecord, setCompressed, setDigest, setIn, setReaderIdentifier, setStrict, setVersion, stripExtension, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CRLF

public static final String CRLF

See Also:

Constant Field Values

Constructor Detail

TestWARCReader

public TestWARCReader(InputStream is)

TestWARCReader

public TestWARCReader(org.archive.io.warc.WARCRecordInfo recinfo)
               throws IOException

Throws:

IOException

Method Detail

get

public org.archive.io.warc.WARCRecord get(long offset)
                                   throws IOException

Overrides:

get in class org.archive.io.ArchiveReader

Throws:

IOException

createArchiveRecord

protected org.archive.io.warc.WARCRecord createArchiveRecord(InputStream is,
                                                 long offset)
                                                      throws IOException

Specified by:

createArchiveRecord in class org.archive.io.ArchiveReader

Throws:

IOException

gotoEOR

protected void gotoEOR(org.archive.io.ArchiveRecord record)
                throws IOException

Specified by:

gotoEOR in class org.archive.io.ArchiveReader

Throws:

IOException

getFileExtension

public String getFileExtension()

Specified by:

getFileExtension in class org.archive.io.ArchiveReader

getDotFileExtension

public String getDotFileExtension()

Specified by:

getDotFileExtension in class org.archive.io.ArchiveReader

dump

public void dump(boolean compress)
          throws IOException,
                 ParseException

Specified by:

dump in class org.archive.io.ArchiveReader

Throws:

IOException

ParseException

getDeleteFileOnCloseReader

public org.archive.io.ArchiveReader getDeleteFileOnCloseReader(File f)

Specified by:

getDeleteFileOnCloseReader in class org.archive.io.ArchiveReader

buildRecordContent

public static InputStream buildRecordContent(org.archive.io.warc.WARCRecordInfo recinfo)
                                      throws IOException

build minimal WARC record byte stream.

Parameters:

recinfo - WARCRecordInfo with record metadata and content

Returns:

InputStream reading from created record bits

Throws:

IOException