org.archive.wayback.accesscontrol.robotstxt

Class RobotRulesTest

java.lang.Object

junit.framework.Assert

junit.framework.TestCase

org.archive.wayback.accesscontrol.robotstxt.RobotRulesTest

All Implemented Interfaces:

junit.framework.Test

public class RobotRulesTest
extends junit.framework.TestCase

Tests for RobotRules

References:

• https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt - referred as "Google/Yahoo/Bing/Ask (GYBA) convention".
• http://www.robotstxt.org/orig.html
• http://www.robotstxt.org/norobots-rfc.txt - referred as "RFC".

Note: GYBA convention document and RFC differs in terminology. What GYBA calls "group" is called "record" in RFC.

Author:

brad

Field Summary

Fields

Modifier and Type	Field and Description
static String	WB_UA

Constructor Summary

Constructors

Constructor and Description
RobotRulesTest()

Method Summary

Methods

Modifier and Type	Method and Description
protected RobotRules	load(String txt)
void	setUp()
void	testBlankLineInGroup() this is a syntax error per RFC, but ok with Google/Yahoo/Bing/Ask convention.
void	testComments()
void	testDirectivesAreCaseInsensitive() basic test
void	testEmptyDisallowHasNoEffect() Disallow: with empty path has no effect.
void	testEndOfPath() Google/Bing/Yahoo/Ask extension: $ matches the end of path.
void	testEOLs() LF, CRLF, CR are recognized as end-of-line.
void	testExtraSpace2() white spaces are allowed at the beginning of the line, too.
void	testLessSpaceExtraSpace() optional white spaces before/after ":", before EOL.
void	testMostSpecificPathPrevails() By GYBA convention, if multiple disallow and allow directives matches the URL, the most specific rule based on the length of the path will win over less specific (shorter) ones.
void	testMultiUA() multiple User-agent: for a record.
void	testMultiUAWithOtherLinesLine() there's sitemap (non-allow/disallow) directive after User-agent.
void	testMultiUAWithOtherLinesLine2() similarly to previous test case, Crawl-delay: line shall end the group.
void	testNonBlocksPathForUA() multiple records for different User-agent's.
void	testPercentEncodedPath() character may be %-escaped, but %2f (/) is special.
void	testSubpath() path matching basics. substring-based, / is no special, and case-sensitive.
void	testUserAgentIsCaseInsensitive() user-agent name comparisons are case-insensitive.
void	testWildcardMatch() wildcard in path, matches any chars including /.

Methods inherited from class junit.framework.TestCase

countTestCases, createResult, getName, run, run, runBare, runTest, setName, tearDown, toString

Methods inherited from class junit.framework.Assert

assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertFalse, assertFalse, assertNotNull, assertNotNull, assertNotSame, assertNotSame, assertNull, assertNull, assertSame, assertSame, assertTrue, assertTrue, fail, fail

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

WB_UA

public static final String WB_UA

See Also:

Constant Field Values

Constructor Detail

RobotRulesTest

public RobotRulesTest()

Method Detail

setUp

public void setUp()

Overrides:

setUp in class junit.framework.TestCase

load

protected RobotRules load(String txt)
                   throws IOException

Throws:

IOException

testDirectivesAreCaseInsensitive

public void testDirectivesAreCaseInsensitive()
                                      throws Exception

basic test

Throws:

Exception

testEmptyDisallowHasNoEffect

public void testEmptyDisallowHasNoEffect()
                                  throws Exception

Disallow: with empty path has no effect.

Throws:

Exception

testLessSpaceExtraSpace

public void testLessSpaceExtraSpace()
                             throws Exception

optional white spaces before/after ":", before EOL.

Throws:

Exception

testExtraSpace2

public void testExtraSpace2()
                     throws Exception

white spaces are allowed at the beginning of the line, too.

Throws:

Exception

testComments

public void testComments()
                  throws Exception

Throws:

Exception

testEOLs

public void testEOLs()
              throws Exception

LF, CRLF, CR are recognized as end-of-line.

Throws:

Exception

testUserAgentIsCaseInsensitive

public void testUserAgentIsCaseInsensitive()
                                    throws Exception

user-agent name comparisons are case-insensitive.

Throws:

Exception

testNonBlocksPathForUA

public void testNonBlocksPathForUA()
                            throws Exception

multiple records for different User-agent's.

while RFC states "the format logically consists of a non-empty set or records, separated by blank lines", Google's documentation has no mention to blank lines as group separator - instead, it recognizes a sequence of User-agent: as the start of "group". So this sample is syntax error per RFC, but okay according to Google/Yahoo/Bing/Ask convention.

Throws:

Exception

testMultiUAWithOtherLinesLine

public void testMultiUAWithOtherLinesLine()
                                   throws Exception

there's sitemap (non-allow/disallow) directive after User-agent. next user-agent should start a new group. ia_archiver ALLOWED as disallow rule only applies to B.

Throws:

Exception

testMultiUAWithOtherLinesLine2

public void testMultiUAWithOtherLinesLine2()
                                    throws Exception

similarly to previous test case, Crawl-delay: line shall end the group.

Throws:

Exception

testBlankLineInGroup

public void testBlankLineInGroup()
                          throws Exception

this is a syntax error per RFC, but ok with Google/Yahoo/Bing/Ask convention. blank lines are permitted within a group to improve readability. they are simply ignored, don't end group.

Throws:

Exception

testMultiUA

public void testMultiUA()
                 throws Exception

multiple User-agent: for a record. ia_archiver BLOCKED as disallow rule applies to both all and B

Throws:

Exception

testSubpath

public void testSubpath()
                 throws Exception

path matching basics. substring-based, / is no special, and case-sensitive.

Throws:

Exception

testPercentEncodedPath

public void testPercentEncodedPath()
                            throws Exception

character may be %-escaped, but %2f (/) is special. (TODO: additional tests: robots.txt is assumed to be UTF-8 encoded. non-7bit-ascii characters are allowed, and also can be %-escaped.)

Throws:

Exception

testMostSpecificPathPrevails

public void testMostSpecificPathPrevails()
                                  throws Exception

By GYBA convention, if multiple disallow and allow directives matches the URL, the most specific rule based on the length of the path will win over less specific (shorter) ones.

RFC says differently: "a robot must attempt to match the paths in Allow and Disallow lines against the URL, in the order they occur in the record. The first match found is used."

we follow GYBA convention here.

Throws:

Exception

testWildcardMatch

public void testWildcardMatch()
                       throws Exception

wildcard in path, matches any chars including /. /* is the same as /.

Throws:

Exception

testEndOfPath

public void testEndOfPath()
                   throws Exception

Google/Bing/Yahoo/Ask extension: $ matches the end of path.

Throws:

Exception