org.archive.io.warc

Class TestWARCRecordInfo

java.lang.Object

org.archive.io.warc.WARCRecordInfo

org.archive.io.warc.TestWARCRecordInfo

All Implemented Interfaces:

org.archive.format.ArchiveFileConstants, org.archive.format.warc.WARCConstants

public class TestWARCRecordInfo
extends org.archive.io.warc.WARCRecordInfo
implements org.archive.format.warc.WARCConstants, org.archive.format.ArchiveFileConstants

WARCRecordInfo with default values and convenience factory methods. TODO: use existing well-tested HTTP library for generating HTTP content-block.

See Also:

TestWARCReader

Nested Class Summary

Nested classes/interfaces inherited from interface org.archive.format.warc.WARCConstants

org.archive.format.warc.WARCConstants.WARCRecordType

Field Summary

Fields

Modifier and Type	Field and Description
static byte[]	PAYLOAD_GIF
static String	REVISIT_WARC_PROFILE

Fields inherited from class org.archive.io.warc.WARCRecordInfo

contentLength, contentStream, create14DigitDate, enforceLength, extraHeaders, mimetype, recordId, type, url, warcFilename, warcFileOffset

Fields inherited from interface org.archive.format.warc.WARCConstants

COLON_SPACE, COMPRESSED_WARC_FILE_EXTENSION, CONTENT_DESCRIPTION, CONTENT_LENGTH, CONTENT_TYPE, DEFAULT_ENCODING, DEFAULT_MAX_WARC_FILE_SIZE, DOT_COMPRESSED_WARC_FILE_EXTENSION, DOT_WARC_FILE_EXTENSION, FTP_CONTROL_CONVERSATION_MIMETYPE, HEADER_FIELD_SEPARATOR, HEADER_KEY_BLOCK_DIGEST, HEADER_KEY_CONCURRENT_TO, HEADER_KEY_DATE, HEADER_KEY_ETAG, HEADER_KEY_FILENAME, HEADER_KEY_ID, HEADER_KEY_IP, HEADER_KEY_LAST_MODIFIED, HEADER_KEY_PAYLOAD_DIGEST, HEADER_KEY_PROFILE, HEADER_KEY_REFERS_TO, HEADER_KEY_REFERS_TO_DATE, HEADER_KEY_REFERS_TO_FILE_OFFSET, HEADER_KEY_REFERS_TO_FILENAME, HEADER_KEY_REFERS_TO_TARGET_URI, HEADER_KEY_TRUNCATED, HEADER_KEY_TYPE, HEADER_KEY_URI, HEADER_LINE_ENCODING, HTTP_REQUEST_MIMETYPE, HTTP_RESPONSE_MIMETYPE, MAX_LINE_LENGTH, MAX_WARC_HEADER_LINE_LENGTH, NAMED_FIELD_CHECKSUM_LABEL, NAMED_FIELD_DESCRIPTION, NAMED_FIELD_FILEDESC, NAMED_FIELD_IP_LABEL, NAMED_FIELD_RELATED_LABEL, NAMED_FIELD_TRUNCATED, NAMED_FIELD_TRUNCATED_VALUE_HEAD, NAMED_FIELD_TRUNCATED_VALUE_LENGTH, NAMED_FIELD_TRUNCATED_VALUE_TIME, NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED, NAMED_FIELD_WARCFILENAME, PLACEHOLDER_RECORD_LENGTH_STRING, PROFILE_REVISIT_IDENTICAL_DIGEST, PROFILE_REVISIT_NOT_MODIFIED, PROFILE_REVISIT_URI_AGNOSTIC_IDENTICAL_DIGEST, TRUNCATED_VALUE_UNSPECIFIED, TYPE, WARC_FIELDS_TYPE, WARC_FILE_EXTENSION, WARC_HEADER_ENCODING, WARC_ID, WARC_MAGIC, WARC_VERSION, WSP

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DOT_COMPRESSED_FILE_EXTENSION, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Constructor and Description
TestWARCRecordInfo(byte[] content)

Method Summary

Methods

Modifier and Type	Method and Description
static byte[]	buildCompressedHttpResponseBlock(String ctype, byte[] payloadBytes)
static byte[]	buildCompressedHttpResponseBlock(String ctype, byte[] payloadBytes, boolean chunked)
static byte[]	buildHttpRedirectResponseBlock(String location)
static byte[]	buildHttpRedirectResponseBlock(String statusline, String location)
static byte[]	buildHttpResponseBlock(String payload)
static byte[]	buildHttpResponseBlock(String ctype, byte[] payloadBytes) short cut for generating "200 OK" HTTP response content-block.
static byte[]	buildHttpResponseBlock(String status, String ctype, byte[] payloadBytes)
static byte[]	buildHttpResponseBlock(String status, String ctype, byte[] payloadBytes, boolean chunked) return content-block bytes for HTTP response.
static byte[]	buildRevisitHttpResponseBlock(String ctype, int len, boolean withHeader, boolean gzipContent) generates WARC content for new revisit record.
static TestWARCRecordInfo	createCompressedHttpResponse(String ctype, byte[] payloadBytes)
static TestWARCRecordInfo	createHttpResponse(String payload) return TestWARCRecordInfo for HTTP Response with entity payload.
static TestWARCRecordInfo	createHttpResponse(String ctype, byte[] payloadBytes) return TestWARCRecordInfo for HTTP Response with entity payload.
static TestWARCRecordInfo	createHttpResponse(String status, String ctype, byte[] payloadBytes) return TestWARCRecordInfo for HTTP Response with response status line status, entity payload of content-type ctype.
static TestWARCRecordInfo	createRevisitHttpResponse(String ctype, int len)
static TestWARCRecordInfo	createRevisitHttpResponse(String ctype, int len, boolean withHeader)
static TestWARCRecordInfo	createRevisitHttpResponse(String ctype, int len, boolean withHeader, boolean gzipContent)
static TestWARCRecordInfo	createUrlAgnosticRevisitHttpResponse(String ctype, int len) creates TestWARCRecordInfo with URL-Agnostic Revisit WARC record content.
static String	dt14ToISOZ(String dt14) translates DT14 (YYYYmmddHHMMSS) to ISOZ format used in WARC-Date header.
void	setCreate14DigitDateFromDT14(String dt14) utility method for updating create14DigitDate from DT14.

Methods inherited from class org.archive.io.warc.WARCRecordInfo

addExtraHeader, getContentLength, getContentStream, getCreate14DigitDate, getEnforceLength, getExtraHeaders, getMimetype, getRecordId, getType, getUrl, getWARCFilename, getWARCFileOffset, isEnforceLength, setContentLength, setContentStream, setCreate14DigitDate, setEnforceLength, setExtraHeaders, setMimetype, setRecordId, setType, setUrl, setWARCFilename, setWARCFileOffset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

REVISIT_WARC_PROFILE

public static final String REVISIT_WARC_PROFILE

See Also:

Constant Field Values

PAYLOAD_GIF

public static final byte[] PAYLOAD_GIF

Constructor Detail

TestWARCRecordInfo

public TestWARCRecordInfo(byte[] content)

Method Detail

dt14ToISOZ

public static String dt14ToISOZ(String dt14)
                         throws IOException

translates DT14 (YYYYmmddHHMMSS) to ISOZ format used in WARC-Date header.

Parameters:

dt14 -

Returns:

ISOZ (YYYY-mm-ddTHH:MM:SSZ)

Throws:

IOException - dt14 is in bad format (wrapping ParseException to simply error handling)

setCreate14DigitDateFromDT14

public void setCreate14DigitDateFromDT14(String dt14)
                                  throws IOException

utility method for updating create14DigitDate from DT14.

Parameters:

dt14 - DT14 (YYYYmmddHHMMSS)

Throws:

IOException - dt14 is in bad format.

createHttpResponse

public static TestWARCRecordInfo createHttpResponse(String payload)
                                             throws IOException

return TestWARCRecordInfo for HTTP Response with entity payload. Content-Type is text/plain, and payload is encoded in UTF-8.

Parameters:

payload -

Returns:

Throws:

IOException

createHttpResponse

public static TestWARCRecordInfo createHttpResponse(String ctype,
                                    byte[] payloadBytes)
                                             throws IOException

return TestWARCRecordInfo for HTTP Response with entity payload.

Parameters:

ctype - Content-Type value

payloadBytes - payload bytes

Returns:

WARCRecordInfo with default values set to key properties.

Throws:

IOException

createHttpResponse

public static TestWARCRecordInfo createHttpResponse(String status,
                                    String ctype,
                                    byte[] payloadBytes)
                                             throws IOException

return TestWARCRecordInfo for HTTP Response with response status line status, entity payload of content-type ctype.

Parameters:

status - status line, such as "200 OK"

ctype - content-type

payloadBytes - payload bytes

Returns:

TestWARCRecordInfo

Throws:

IOException

createCompressedHttpResponse

public static TestWARCRecordInfo createCompressedHttpResponse(String ctype,
                                              byte[] payloadBytes)
                                                       throws IOException

Throws:

IOException

createRevisitHttpResponse

public static TestWARCRecordInfo createRevisitHttpResponse(String ctype,
                                           int len,
                                           boolean withHeader)
                                                    throws IOException

Throws:

IOException

createRevisitHttpResponse

public static TestWARCRecordInfo createRevisitHttpResponse(String ctype,
                                           int len,
                                           boolean withHeader,
                                           boolean gzipContent)
                                                    throws IOException

Throws:

IOException

createRevisitHttpResponse

public static TestWARCRecordInfo createRevisitHttpResponse(String ctype,
                                           int len)
                                                    throws IOException

Throws:

IOException

createUrlAgnosticRevisitHttpResponse

public static TestWARCRecordInfo createUrlAgnosticRevisitHttpResponse(String ctype,
                                                      int len)
                                                               throws IOException

creates TestWARCRecordInfo with URL-Agnostic Revisit WARC record content.

• WARC-Refers-To-Target-URI = http://example.com/
• WARC-Refers-To-Date = 2014-01-01T10:10:10Z

Parameters:

ctype - Content-Type

len - Content-Length (arbitrary)

Returns:

TestWARCRecordInfo

Throws:

IOException

buildHttpResponseBlock

public static byte[] buildHttpResponseBlock(String payload)
                                     throws IOException

Throws:

IOException

buildHttpResponseBlock

public static byte[] buildHttpResponseBlock(String ctype,
                            byte[] payloadBytes)
                                     throws IOException

short cut for generating "200 OK" HTTP response content-block.

Parameters:

ctype - HTTP Content-Type, such as "text/plain", "image/gif"

payloadBytes - payload bytes

Returns:

content-block bytes with HTTP status line, HTTP headers and payload.

Throws:

IOException

buildHttpResponseBlock

public static byte[] buildHttpResponseBlock(String status,
                            String ctype,
                            byte[] payloadBytes,
                            boolean chunked)
                                     throws IOException

return content-block bytes for HTTP response.

Parameters:

status - HTTP status code and status text separated by a space. ex. "200 OK".

ctype - HTTP Content-Type

payloadBytes - payload bytes

chunked - if true, use chunked transfer-encoding

Returns:

content-block bytes with HTTP status line, HTTP headers and payload.

Throws:

IOException

buildHttpResponseBlock

public static byte[] buildHttpResponseBlock(String status,
                            String ctype,
                            byte[] payloadBytes)
                                     throws IOException

Throws:

IOException

buildHttpRedirectResponseBlock

public static byte[] buildHttpRedirectResponseBlock(String location)
                                             throws IOException

Throws:

IOException

buildHttpRedirectResponseBlock

public static byte[] buildHttpRedirectResponseBlock(String statusline,
                                    String location)
                                             throws IOException

Throws:

IOException

buildCompressedHttpResponseBlock

public static byte[] buildCompressedHttpResponseBlock(String ctype,
                                      byte[] payloadBytes,
                                      boolean chunked)
                                               throws IOException

Throws:

IOException

buildCompressedHttpResponseBlock

public static byte[] buildCompressedHttpResponseBlock(String ctype,
                                      byte[] payloadBytes)
                                               throws IOException

Throws:

IOException

buildRevisitHttpResponseBlock

public static byte[] buildRevisitHttpResponseBlock(String ctype,
                                   int len,
                                   boolean withHeader,
                                   boolean gzipContent)
                                            throws IOException

generates WARC content for new revisit record.

Parameters:

ctype - value for Content-Type

len - value for Content-Length

withHeader - include HTTP status line and headers. passing false generates old-style revisit content block.

gzipContent - if true, block will have "Content-Encoding: gzip" header. (this shall match the compress-ness of previous capture).

Returns:

record content as byte array

Throws:

IOException