Third Party Packages

Please see the System Requirements .

Wayback Software

Please see the Software Downloads page .

Installing Tomcat

Please refer to the README file included with your Tomcat distribution.

Installing Wayback

Once you have downloaded the .tar.gz file from sourceforge, you will need to unpack the file to access the webapp file, wayback-webapp-1.6.0.war.

Installation and configuration of this software involves the following steps:

1. Placing .war file in appropriate location.
2. Waiting for Tomcat to unpack the .war file.
3. Customizing base wayback.xml and possibly other XML configuration files.
4. Restarting tomcat.

AccessPoint configuration options

An AccessPoint's configuration must specify the following implementations:

• collection the specific WaybackCollection being exposed via this AccessPoint.
• query responsible for generating user visible content(HTML, XML, etc) in response to user Queries.
• replay responsible for determining the appropriate ReplayRenderer implementation based on the users request and the particular document to be Replayed.
• uriConverter responsible for constructing Replay URLs from records matching users queries. See Replay Modes below.
• parser - responsible for translating incoming requests into WaybackRequests. See Replay Modes below.

An AccessPoint's configuration may optionally specify the following, but must specify at least one of replayPrefix, queryPrefix, or staticPrefix:

• exception - an implementation responsible for generating error pages to users
• configs - a Properties associating arbitrary key-value pairs which are accessible to .jsp files responsible for generating the UI
• exclusionFactory - an implementation specifying what documents should be accessible within this AccessPoint
• authentication - an implementation specifying who is allowed to connect to this AccessPoint
• replayPrefix - a String URL prefix indicating the host, port, and path to the correct Replay AccessPoint. If unspecified, defaults to queryPrefix, then staticPrefix.
• queryPrefix - a String URL prefix indicating the host, port, and path to the correct Query AccessPoint. If unspecified, defaults to staticPrefix, then replayPrefix.
• staticPrefix - a String URL prefix indicating the host, port, and path to static content used within the UI. If unspecified, defaults to queryPrefix, then replayPrefix.
• livewebPrefix - a String URL prefix indicating the host, port, and path to an AccessPoint configured with Live Web fetching.
• locale - A specific Locale to use for all requests within this AccessPoint, overriding the users preferred Locale as specified by their web browser.
• exactHostMatch - true or false, if true, only returns results exactly matching a given request hostname (case insensitive). Default is false.
• exactSchemeMatch - true of false, if true, only returns results exactly matching a given request scheme. Default is true.

AccessPoints can be used to provide different levels and types of access to the same collection for different users. For example, you can provide both Proxy and Archival URL mode access to a single collection by defining 2 AccessPoints with different Replay User Interfaces but the same WaybackCollection. Using AccessPoints, you can also provide different levels of access to a collection. For example, users within a particular subnet may be able to access all documents within a collection via one AccessPoint, but users outside that subnet may be restricted to viewing documents allowed by a web sites current robots.txt file.

Please refer to wayback.xml within the wayback .war file for detailed example AccessPoint configurations.

WaybackCollection Configuration

A WaybackCollection's configuration must specify the following implementations:

• resourceStore the specific implementation used to specific set of documents within this collection, and how to access them for Replay requests.
• resourceIndex the specific implementation responsible for locating documents within the collection.

A WaybackCollection's configuration may optionally specify the following:

• shutdownables - an List of one or more beans implementing org.archive.wayback.Shutdownable needed to maintain this WaybackCollection, typically Daemon Threads which perform automatic indexing operations on the resourceStore and the resourceIndex.

For more information on WaybackCollection configuration options and automatic indexing, please refer to the following documentation pages and to the example Spring .xml configuration files within the wayback .war:

• ResourceStore configuration and automatic indexing
• ResourceIndex configuration
• BDBCollection.xml
• CDXCollection.xml
• RemoteCollection.xml

Archival URL Replay Mode

Archival URL Replay mode uses a modified URL to designate documents stored in ARC/WARC files. The general form of an Archival URL is:

• HOSTNAME is the host where the Wayback software is running.
• PORT is the port where Tomcat is listening for incoming HTTP requests, which also refers to part of the name of the Access Point. See below for example CONTEXT mappings.
• CONTEXT is an optional context where the Wayback webapp has been deployed, plus an optional name of the Access Point within the webapp. See below for example CONTEXT mappings.
• TIMESTAMP is 0 to 14 digits of a date, possibly followed by an asterisk ('*'), or one or more tags providing further specifics for the request. The format of a TIMESTAMP is:
  YYYYMMDDHHmmss
  where
  • YYYY represents a 4-digit year
  • MM represents a 2-digit, 1-based month (Jan = 1 - Dec = 12)
  • DD represents a 2-digit day of the month (01-31)
  • HH represents a 2-digit hour (01-24)
  • mm represents a 2-digit minute (00-59)
  • ss represents a 2-digit second (00-59)
  The following are example dates expressed as 14-digit Timestamps:
  
  
  Jan 13, 1999 03:34:35 (am UTC) - 19990113033435
  
  
  
  Dec 31, 2004 23:01:00 (pm UTC) - 20041231230100
  
  
  

  Following the date portion of a timestamp, the following flags can be appended:

  • id_ Identity - perform no alterations of the original resource, return it as it was archived.
  • js_ Javascript - return document marked up as javascript.
  • cs_ CSS - return document marked up as CSS.
  • im_ Image - return document as an image.
• URL represents the actual URL that should be replayed.

The properties parser and uriConverter for Archival URL Access Points must be set to the following implementations:

    <property name="parser">
      <bean class="org.archive.wayback.archivalurl.ArchivalUrlRequestParser"
        init-method="init">
        <property name="maxRecords" value="1000" />
        <property name="earliestTimestamp" value="1996" />
      </bean>
    </property>

    <property name="uriConverter">
      <bean class="org.archive.wayback.archivalurl.ArchivalUrlResultURIConverter">
        <property name="replayURIPrefix" value="http://wayback.example.org:8080/collection/" />
      </bean>
    </property>

          

For additional configuration examples and information about ArchivalUrl Replay mode, please see the file ArchivalUrlReplay.xml

Proxy Replay Mode

Wayback can be configured to act as an HTTP proxy server. To utilize this mode, the wayback webapp must be deployed as the ROOT context, no other AccessPoints can use the port dedicated to the Proxy AccessPoint, and client browsers must be configured to proxy all HTTP requests through the Wayback Machine application. Instead of retrieving documents from the live web, the Wayback Machine will retrieve documents from the configured WaybackCollection.

Proxy Replay mode does not suffer from the shortcomings of the inserted Javascript that the Archival URL mode uses, all URLs function as they did originally, but there can be another drawback to using this feature: no date information is sent with each request. Wayback attempts to address this problem by associating the date clicked on query pages when a Replay session is begun, with the users IP address. This can fail to work properly in situations where multiple users are behind a NAT system which causes them to appear to have the same IP address.

Additionally, there is an experimental Firefox-specific plugin developed by Oskar Grenholm, which provides a novel interface to navigate between different captured versions of a page within Proxy mode, and also sends a special HTTP header which allows Wayback to uniquely associate the correct date with browsers, even those behind a NAT system. You can find out more about this plugin and download it here .

Thanks Oskar!

The following is an example Proxy Replay Access Point definition. It assumes to be running on a host wayback.somehost.org, that a Tomcat Connector has been added for port 8090, that the Wayback webapp has been deployed at the ROOT context, and that another Archival URL Access Point named "8080:wayback" has been configured.

<bean name="8090" parent="8080:wayback">
  <property name="queryPrefix" value="http://wayback.somehost.org/" />
  <property name="replay"> ref="proxyreplay" />
  <property name="uriconverter">
    <bean class="org.archive.wayback.proxy.RedirectResultURIConverter">
      <property name="redirectURI" value="http://wayback.somehost.org/jsp/Redirect.jsp" />
    </bean>
  </property>
  <property name="parser">
    <bean class="org.archive.wayback.proxy.ProxyRequestParser" >
      <property name="localhostNames">
        <list>
          <value>wayback.somehost.org</value>
        </list>
      </property>
      <property name="maxRecords" value="1000" />
    </bean>
  </property>
</bean>

          

redirectURI is required, and must be set to the name of the host where the Wayback application is running. If this is not the primary name of the machine running the Wayback application, then you may need to also specify the hostname used for the Wayback application in the localhostNames configuration list.

For additional configuration examples and information about Proxy Replay mode, please see the file ProxyReplay.xml

DomainPrefix Replay Mode

Wayback includes an additional, experimental Replay mode which is similar to Archival URL mode, in that any document can be refernced as a global URL, without any browser configuration requirements. This mode requires deploying the Wayback webapp in ROOT context, and a special DNS wildcard aliasing, so that all hostnames with a common suffix will be directed to your host running Wayback.

The general form of a DomainPrefix URL is:

Here is an example DomainPrefix URL, on an assumed host wayback.somehost.org, with a wayback webapp deployed as ROOT, via the Access Point named 8081 (which indicates the port Wayback requests will be recieved on) for the page http://www.yahoo.com/foo.gif on Dec 31, 1999 at 12:00:00 UTC.

This mode performs all URL rewriting on the server side, so needs no client-side Javascript to execute, and also does not suffer from some of the request leakage problems present in Archival URL mode. It presently is somewhat naive about rewriting links within returned documents, and will also rewrite URLs in the text of pages (not desired), as well as URLs referenced within the page (desired).

For additional configuration examples and information about Domain Prefix Replay mode, please see the files wayback.xml and DomainPrefixReplay.xml .

Query UI

All content returned by Wayback in response to Query requests is generated by .jsp files, which are executed and provided access to the results found within the ResourceIndex. Wayback is distributed with several sample implementations.

To alter the default behavior, you may either provide your own .jsp files, and configure the Renderer to use them instead of the default .jsp files, or the default .jsp files may be modified directly.

• captureJsp - used when the request indicates that a listing of all dates available for a single URL should be returned. Default is /WEB-INF/query/HTMLCaptureResults.jsp. An alternate implementation, /WEB-INF/query/CalendarResults.jsp will generate HTML output similar to the global Wayback Machine service.
• urlJsp - used when the request indicates that a summary of captures available for a number of URLs should be returned. Default is /WEB-INF/query/HTMLUrlResults.jsp
• xmlCaptureJsp - used when the request indicates that a listing of all dates available for a single URL should be returned in XML format. Default is /WEB-INF/query/XMLCaptureResults.jsp.
• xmlUrlJsp - used when the request indicates that a summary of captures available for a number of URLs should be returned in XML format. Default is /WEB-INF/query/XMLUrlResults.jsp

Replay Inserts

Wayback allows for embedding additional content within replayed HTML pages in all Replay modes. This is accomplished by executing one or more .jsp files with access to context information about the request, the results, and the actual Resource being returned. The output of each .jsp file is included within the returned page.

Wayback is distributed with several example .jsp insert files that can be used as is, modified to suit installation requirements, or used as examples for more elaborate customizations:

• /WEB-INF/replay/ArchiveComment.jsp inserts an HTML comment indicating when the document was captured and retrieved.
• /WEB-INF/replay/ClientSideJSInsert.jsp inserts some Javascript into the returned HTML page that updates links, images, and other embedded content, attempting to make all URL references within the page point back into the Wayback service.
• /WEB-INF/replay/DebugBanner.jsp Not intended for production use, but a slightly more complex jsp insert example that demonstrates how to access various request context data, and is sometimes useful for debugging.
• /WEB-INF/replay/Disclaimer.jsp Inserts a small banner at the top of replayed HTML pages, alerting users that they are viewing an archived page, and providing some information about the particular capture.
• /WEB-INF/replay/JSLessTimeline.jsp Inserts a banner in the top of replayed documents which allows users to navigate directly between other captures of the current page they are viewing. This version does not use Javascript to place the banner, so it will appear in all HTML pages within a frameset.
• /WEB-INF/replay/Timeline.jsp Inserts a banner in the top of replayed documents which allows users to navigate directly between other captures of the current page they are viewing. This version uses Javascript to place the banner, attempting to only place the banner in the largest frame within a frameset.
• /WEB-INF/replay/Toolbar.jsp Inserts a fancier banner in the top of replayed documents which includes a graphic representaion of the number of captures over time and allows users to navigate directly between other captures of the current page they are viewing. This version uses Javascript to place the banner, attempting to only place the banner in the largest frame within a frameset.

Exception Rendering

Wayback is distributed with a default ExceptionRenderer that allows customization of several types of anticipated exceptions that can occur through normal operations. The BaseExceptionRenderer allows installations to provide alternate .jsp files which are executed, and the output of these .jsp files are returned to end users. To alter the default behavior, you may either provide your own .jsp files, and configure the BaseExceptionRenderer to use them instead of the default .jsp files, or the default .jsp files may be modified directly.

• xmlErrorJsp - used when the request indicates that XML data should be returned. Default is /WEB-INF/exception/XMLError.jsp
• errorJsp - used for HTML Replay exceptions, and for all Query exceptions. Default is /WEB-INF/exception/HTMLError.jsp
• imageErrorJsp - used when the request appears to be an embedded Replay request that expects an image to be returned. Default is /WEB-INF/exception/HTMLError.jsp which produced HTML output. This may be desirable over returning an actual image, since web browsers will usually show any HTML alternate text associated with the image in place of the image when image data is not returned. Wayback also includes a 1x1 pixel gif, error_image.gif, which can be used to display a gray box in place of images requests that result in an exception.
• javascriptErrorJsp - used when the request appears to be an embedded Replay request that expects Javascript content to be returned. Default is /WEB-INF/exception/JavaScriptError.jsp
• cssErrorJsp - used when the request appears to be an embedded Replay request that expects CSS content to be returned. Default is /WEB-INF/exception/CSSError.jsp

Localization .properties files.

Wayback is packaged with a set of reference implementation .jsp files for generating Query, Replay, and Exception user interface pages. References to actual user visible text is abstracted within these .jsp files so the specific text to display in various pages are read from a .properties file. Wayback will automatically search for a Locale-specific .properties file from which these text values should be loaded, allowing the language presented to users to be changed.

By default, Wayback will use the language preference indicated by the users web browser to find an appropriate .properties files, defaulting to the standard English text if the users preferred language is not available. Particular AccessPoints can be forced to a particular Locale using the AccessPoint.locale property.

Several language customization .property files have already been contributed by users in the community and are now included with the standard Wayback distribution. We plan for a completely new and improved UI implementation for version 1.6, and plan a more active outreach program to create customizations in as many languages as possible once this new UI is completed, and the required text elements are determined.

Excluding Documents with live Robots.txt

<property name="exclusionFactory" ref="excluder-factory-robot" />

        

Excluding Documents with an Administrative List

<bean id="static-exclusion" class="org.archive.wayback.accesscontrol.staticmap.StaticMapExclusionFilterFactory" init-method="init">
  <property name="file" value="/tmp/exclude.txt" />
  <property name="checkInterval" value="600" />
</bean>

        

<property name="exclusionFactory" ref="static-exclusion">

        

Limiting Access based on IP Addresses

<property name="authentication">
  <bean class="org.archive.wayback.authenticationcontrol.IPMatchesBooleanOperator">
    <property name="allowedRanges">
      <list>
        <value>192.168.1.16/24</value>
      </list>
    </property>
  </bean>
</property>

        

Limiting Access based on HTTP BASIC Authentication

<security-role>
  <description>Secured-Wayback</description>
  <role-name>wayback</role-name>
</security-role>

<security-constraint>
  <web-resource-collection>
    <web-resource-name>Secured-Wayback</web-resource-name>
    <url-pattern>/usersecure/*</url-pattern>
  </web-resource-collection>
  <auth-constraint>
    <role-name>wayback</role-name>
  </auth-constraint>
</security-constraint>

<login-config>
  <auth-method>BASIC</auth-method>
  <realm-name>Secured-Wayback</realm-name>
</login-config>

        

<role rolename="wayback"/>
<user password="changeM3" roles="wayback" name="brad"/>

        

bdb-client

This tool allows several maintenance operations to be performed on BDB files. There are two primary modes, read and write.

1. bin/bdb-client -r BDB_DIR BDB_NAME [PREFIX]
   Output records from a BDB database on STDOUT.
   where:

   • BDB_DIR Open BDB in this directory.
   • BDB_NAME Open BDB with this name.
   • PREFIX (optional) if present, only output records whose KEY begins with PREFIX. If this option is omitted, all records will be output from the BDB. Records are always output in sorted order.
2. bin/bdb-client -w BDB_DIR BDB_NAME
   Read CDX format lines from STDIN, and insert into a BDB, creating the BDB if needed.
   where:

   • BDB_DIR Open BDB in this directory.
   • BDB_NAME Open BDB with this name.

bin-search

This tool allows binary searching against large sorted text files. It will output lines prefixed with a particular key on STDOUT.

bin/bin-search KEY FILE [FILE2 ...]

• KEY String prefix for lines that should be output.
• FILE [FILE2 ...] Search through all files specified, outputting the lines prefixed with KEY from each file in a single, sorted stream. This assumes that all FILE arguments are sorted.

cdx-indexer

These tools create a CDX format index for the ARC/WARC file at PATH, either on STDOUT, or at the path specified by CDX_PATH. The resulting file can be sorted and merged with other CDX format index files to generate CDX format ResourceIndex.

            bin/cdx-indexer [-identity] PATH [CDX_PATH]
          

Note that when manually constructing CDX files using these tools, you must set the environment variable LC_ALL=C when using the standard UNIX sort command line tool.

The -identity option causes the tools to skip canonicalization of URLs. When using this option, you will need to pass the CDX records through the url-client tool before sorting them into a production CDX index. See the documentation for the url-client tool, and the URL Canonicalization section for more information.

location-client

If you have already populated your ResourceIndex, and just need to inform the ArcProxy LocationDB of where ARC files are located. This script will allow you to synchronize the ArcProxy LocationDB with the directories holding your ARC files.

Execute the script once for each directory containing ARC files (on each machine containing ARC files.) Again, this script will not index the content of the ARC files, but will only populate the ArcProxy LocationDB with the locations of ARC files.

bin/location-client sync LOCATION_URL ARC_DIR ARC_URL_PREFIX

where:

• LOCATION_URL is the absolute URL where the FileProxy can be accessed. ex. http://wayback-webapp.your-archive.org:8080/locationdb/locationDB
• ARC_DIR is the absolute path to the directory on the local machine which holds ARC files ex. /2/arc-collection-1
• ARC_URL_PREFIX is the absolute URL where the directory ARC_DIR can be accessed. ex. http://arc-storage-node-1.your-archive.org/2/arc-collection-1/

url-client

URLs stored in BDB and CDX format ResourceIndexes are canonicalized to a more generic form. Before performing a lookup operation on the ResourceIndex, the same canonicalization function is applied to requested URLs. This tool will read space(" ") delimited lines from STDIN, and output the same lines on STDOUT, but with one column altered. The column that is changed is assumed to be an URL, and the version output is the canonicalized form of the input URL.

This tool is required when using the cdx-indexer tool with the -identity option. Typical usage involves generating an identity CDX index, then passing the lines in that index through this tool to canonicalize the record URL key for queries. If the identity CDX files are kept, then canonicalization schemes can be swapped without reindexing the original ARC/WARC content. This tool can also be useful for debugging the canonicalization function. See the section URL Canonicalization for more information.

bin/url-client [-cdx] [-d DELIMITER] [-f FIELD] [-f FIELD2] ...

• -cdx Pass thru lines prefixed with " CDX " unchanged.
• -d DELIMITER Use DELIMITER as to separate fields instead of default Space(" ").
• -f FIELD alter column FIELD of each line, instead of the default column 1. If specified multiple times, then each column will be canonicalized in transformed lines.