The AspectJ compiler test harness can compile and run AspectJ programs
as specified by the test definitions.
This document tells you how to run the harness.
It describes the options you can specify on the command-line to
control various components that form the harness, either to 
specify options that augment the test definitions or to
change how the harness works, e.g., selecting particular tests
or logging more information.
For information on how to write a test definition, see
readme-writing-compiler-tests.html in the testing module.
The harness drives compiler tests, using a chain of responsibility to map elements in the schema of a test definition to implementing classes.
| Test feature | Description | Implementing class | 
|---|---|---|
| (loading suites...) | general harness | Harness | 
| (logging...) | subclass feature harness | FeatureHarness | 
| <suite> | Test suite | AjcTest.Suite | 
| <ajc-test> | Test case | AjcTest | 
| <compile> | Initial (batch) compile run | CompilerRun | 
| <inc-compile> | Incremental re-compile | IncCompilerRun | 
| <run> | Run class | JavaRun | 
ajc
(optionally as wrapped by the Ant task or AJDE API's), but
in principle any compiler accepting similar options can be
used.
To run from the command-line, use 
Harness.main(String[]).
To run programmatically, use Harness.getHarness().
Harness.runMain(String[]) takes arguments that
each component in the chain may accept and interpret, so
you can modify how the tests run by specifying the following 
arguments on the harness command line:
| Component | Options | 
|---|---|
| Harness suite files, harness verbosity, temp files, option variants | |
| suite files: ajcTest-compliant .txt or .xml files are accepted. | |
| -verboseHarness,-quietHarness: 
      Log accepted options and skipped tests,
      or do not print even info messages. | |
| -keepTemp: Normally the harness saves temp files until
     the end of the run, and deletes them.  If you abort the run or specify-keepTemp, then temporary (sandbox) directories will remain for analysis.
     In either case, the file system accumulates all temporary directories
     and files used for a give harness run; for theajcTests.xmlsuite, this runs into thousands of files. | |
| -killTemp: The opposite of-keepTemp,
     this causes the harness to delete temporary (sandbox) directories at 
     the end of each test run. 
     In this case, the file system only accumulates files for 
     the current test. | |
| *- variants: Options with a trailing "-" cause two sets of option lists to be produced, one with and one without the corresponding option. E.g., "-emacssym-" will run the suite twice, once with and once without the "-emacssym" flag. That means if you use this on each of three options, you will get 8 variant sets (1 with no options, 1 with all 3 options, 3 with 2 options, and 3 with 1 option). | |
| FeatureHarness output and logging options | |
| tracing: -progressDotswill print "." for every passed
     test completed and "!" for every test completed but not passed.-traceTestswill print a one-line summary for each test
     of the time and space taken and whether the test passed.-traceTestsMinwill print only the test and whether it passed.-baselineis an alias for-traceTestsMin-hideStreamsand!eclipse, used to emit tests results in a form
     comparable byorg.aspectj.testing.util.TestDiffs. 
     or usable to select tests by title for options like-ajctestTitleList. | |
| output: -hide{Compiler|Run}Streamswill prevent output and
     error streams from being printed to System.err and System.out, 
     optionally only for run or compile steps. | |
| logging: 
     Log variants take the form -log{Min|Xml}[Fail|Pass|All].
     The suffix {All|Pass|Fail} selects all tests or only passing or failing tests.
     The infix {Min} means to log with minimal information, typically only any
     fail messages.
     The infix {Xml} means to log the XML form of the test definition, so that
     you can inspect the input or re-run arbitrary tests.  
     (You can also re-run a set of tests using keywords
     (e.g., "-ajctestsRequireKeywords=..." or using titles
     (e.g., "-ajctestsTitleFailList=ajcTestResults.txt".)
     Finally, the experimental option-XlogPublicTypewill 
      log the XML test definition for 
  	  any test run that emits any ERROR messages containing the text "public type". | |
| interaction of output streams and logging: Streams will be emitted in real-time, before the test is logged, unless streams are hidden. When logging in normal (non-Min or -XML) form, the log will emit the streams with the test report, so e.g., you can use -hideStreams -logFail to hide streams for passing tests but emit them for failing tests in the context of the log. | |
| AjcTest selection options for keywords, bugID (PR), or title (description) | |
| keywords: -ajctest[Require|Skip]Keywords=one{,two}will either require or skip a test that has one of the 
     specified keywords. | |
| Bugs: -ajctestPR=101{,102}will require that a test have one of the listed bug id's. | |
| title: "-ajctestTitleContains=one,two"will require that the title (description) of a test contain 
     one of the specified substrings, here either "one" or "two".
     Use this to select a few tests you know generally."-ajctestTitleList=first title\, in theory, second title"will require that the title (description) of a test be
     exactly "first title, in theory" or "second title".  
     The entire option must be one argument on the command line.
     Use this when working with just a few specific tests."-ajctestTitleList=../tests/ajcTestResults.txt"will require that the title (description) of a test be
     equal to one listed in../tests/ajcTestResults.txtas a line of the form "[PASS|FAIL] {title}(.."
     (This form is emitted by the-traceTestsMinoption).
     This option only differs from the prior in that the parameter
     is a valid file to read.
     Use this to re-run a large set of tests."-ajctestTitleFailList=../tests/ajcTestResults.txt"is the same as the-ajctestTitleList={file}variant,
     except that only results prefixed "FAIL" are included.
     Use this to re-run only the tests that failed from a large set. | |
| Combinations: all selectors are applied to each test,
     so all tests selected will comply with all constraints.
     Specifying lists within a particular constraints will match
     a union of tests for that constraint 
     (e.g., all tests with bug id's 101 or 102),     
     but there is no way to get a union of constraints 
     (e.g., all tests with bug id's 101 or 102 or 
     with keywords pure-java or knownLimitation).
     However, -ajctestSkipKeywords=...can return all
     tests without the specified keywords, so it can form unions like
     "all tests without the knownLimitation keyword, but with 
     bug id's 101 or 102".
     Title lists can work similarly.  E.g., to run the failed
     incremental tests from ajcTestResults.txt, specify-ajctestTitleFailList=../tests/ajcTestResults.txt-ajctestRequireKeywords=incremental-test. | |
| CompilerRun compiler options and side-effects | |
| supported options: 
     The command line passed to the compiler by CompilerRunis composed of entries derived from the<compile>attributes and recognized from the harness command line.<compile>has specific attributes likefiles,argfiles,classpathandsourcerootwhich translate directly to their counterparts.
     The output option-dis defined byCompilerRunand
     may not be specified (and-outjaris not supported).
     Most other compiler options are defined inCompilerRun.Spec.CRSOptionsand may be specified
     on the harness command-line
     or in theoptionsattribute of<compile>.  
     In theoptionsattribute, each argument is comma-delimited,
     so an option with an argument would look like<compile options="-source,1.4" ...>.  
     If options collide, duplicates
     can be resolved using option dominance (below). | |
| compiler selectors: 
  	 Use -ajcor-eclipseto select the old
     (ajc 1.0) or new (eajc 1.1) compilers.  
      Note that the old compiler is not
      available in the CVS source tree at eclipse.org.
      Use-ajdeCompilerto run a wrapper around the
      AJDE interface
      and-ajctaskCompilerto run a wrapper around the
      AjcTask (Ant task) interface. | |
| option dominance [-|!|^]: 
     Some tests require or prohibit certain options; 
     likewise, sometime you want to force all tests
     run with or without an option specified on the command-line,
     regardless of its setting in the<compile options=".." ...>attribute. 
     For this reason an option may be specified in the options attribute
     or on the harness command-line as-option,!option, or^option.
 | |
| auto-skip: After combining global and local options, there may be
     conflicting or impossible options, which cause the test to be skipped: 
 | |
| source searching: Given -seek:{literal},  
  	  as a side-effect,CompilerRunwill search source files for {literal},
      emitting for each instance an INFO message of the form: 
     found: {file}:{line}:{column}
      (Note that the harness does not display INFO messages unless -verboseHarness
      or -loud is used.) | |
| JavaRun Options and forking | options: options specified in the test are passed to the main method as they would be on the command-line. No options passed to the harness are passed through to the main class. | 
| forking: 
        Forking is useful to run in a different version of Java
        than can be supported by the harness (i.e., some 1.1 flavor);
        it's very time-consuming otherwise.
  		Currently forking is only controllable through system properties
  		of the invoking vm (defined in JavaRun.java): 
 | 
java {harness} -hideStreams {suiteFile}
 Use this to output only a 1-line summary of the test results
    (tests skipped, incomplete, failed, passed).
java {harness} -hideStreams -traceTestsMin {suiteFile} > results.txt
 This writes to result.txt one line [PASS|FAIL] per test, plus a
    1-line summary of the test results.
java {harness} -logFail {suiteFile} -ajctestTitleFailList=results.txt
 This re-runs any test that failed from the "results.txt" run,
    verbosely logging any fails.
java {harness} -hideStreams -logMinFail {suiteFile}
 Use this when running tests mainly to see if they pass or
    if the failure messages are typically enough information
    to indicate why the test is failing.  It produces only minimal
    output for failed tests.
java {harness} -hideStreams -verboseHarness -logFail {suiteFile}
 When it's not clear at first glance why a test is failing, before
    looking at the test code you can run it and print any harness or test
    setup failures and all the associated messages from the test components.
java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile}
 Because of the trailing '-' on two of the options,
    this would do four complete runs with the old (Ajc 1.0) compiler: one with
    no options, one with -lenient, one with -Xlint, and one with both.
java {harness} --ajctestPR=101,102 -Xlint- ^usejavac !eclipse {suiteFile}
 Run any tests associated with bugs 101 and 102, with and without -Xlint,
    forcing off -usejavac and forcing the use of the new eclipse-based compiler.
Harness.optionAliases.
modules/lib/test, defined in 
org.aspectj.testing.harness.bridge.Globals.  
Normally the harness finds these by relative path 
../lib/tests/*.jar, which works whenever the tests are
run from a peer module directory.  When running tests elsewhere,
define the environment variable harness.libdir - e.g., 
    $ cd aspectj/tests
    $ java -Dharness.libdir=../modules/lib/test ...
   java -Djavarun.java=d:\jdk1.1.8\bin\java.exe \
        -Djavarun.bootclasspath=d:\jdk1.1.8\lib\classes.zip \
        -Djavarun.java.home=d:\jdk1.1.8 \
        -Djavarun.fork=true \
        -jar ../aj-build/jars/testing-drivers-all.jar \
        ajcTests.xml -logFail
Here CompilerRun would add the bootclasspath as such when compiling.
JavaRun would fork using the 1.1 vm and prepend the bootclasspath
to the classpath, with an effect like these commands
(ignoring the line splitting in the classpath):
   set JAVA_HOME=d:\jdk1.1.8
   d:\jdk1.1.8\bin\java.exe \
     -classpath "d:\jdk1.1.8\lib\classes.zip;
                 d:\aspectj-src\lib\test\testing-client.jar;
                 d:\aspectj-src\lib\test\aspectjrt.jar;
                 c:\TEMP\sandbox7wers\classes"
     {mainClass} {option..}