The tests are located in two separate CVS modules. The basic feature and -regression tests are in the main "llvm" module under the directory -llvm/test. A more comprehensive test suite that includes whole -programs in C and C++ is in the llvm-test module. This module should -be checked out to the llvm/projects directory. When you -configure the llvm module, the llvm-test module -will be automatically configured. Alternatively, you can configure the - llvm-test module manually.
+The tests are located in two separate Subversion modules. The basic feature + and regression tests are in the main "llvm" module under the directory + llvm/test. A more comprehensive test suite that includes whole +programs in C and C++ is in the test-suite module. This module should +be checked out to the llvm/projects directory as llvm-test (for +historical purpose). When you configure the llvm module, +the llvm-test directory will be automatically configured. +Alternatively, you can configure the test-suite module manually.
To run all of the simple tests in LLVM using DejaGNU, use the master Makefile in the llvm/test directory:
+ +% gmake -C llvm/test-or
or
+ +% gmake check+
To run only a subdirectory of tests in llvm/test using DejaGNU (ie. Regression/Transforms), just set the TESTSUITE variable to the path of the subdirectory (relative to llvm/test):
+ +% gmake -C llvm/test TESTSUITE=Regression/Transforms+
Note: If you are running the tests with objdir != subdir, you must have run the complete testsuite before you can specify a @@ -138,13 +148,16 @@ subdirectory.
To run the comprehensive test suite (tests that compile and execute whole programs), run the llvm-test tests:
+% cd llvm/projects -% cvs co llvm-test -% cd llvm-test -% ./configure --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT +% svn co http://llvm.org/svn/llvm-project/test-suite/trunk llvm-test +% cd .. +% ./configure --with-llvmgccdir=$LLVM_GCC_DIR +% cd projects/llvm-test % gmake+
All "whole program" tests are located in the llvm-test CVS +
All "whole program" tests are located in the test-suite Subversion module.
@@ -220,7 +233,8 @@ subtrees of the test suite directory tree are as follows:This directory contains a large array of small tests that exercise various features of LLVM and to ensure that regressions do not occur. The directory is broken into several sub-directories, each focused on - a particular area of LLVM. A few of the important ones are:
Typically when a bug is found in LLVM, a regression test containing just enough code to reproduce the problem should be written and placed somewhere underneath this directory. In most cases, this will be a small piece of LLVM assembly language code, often distilled from an actual application or benchmark.
-The llvm-test CVS module contains programs that can be compiled +
The test-suite module contains programs that can be compiled with LLVM and executed. These programs are compiled using the native compiler and various LLVM backends. The output from the program compiled with the native compiler is assumed correct; the results from the other programs are @@ -324,11 +338,14 @@ location of these external programs is configured by the llvm-test
Below is an example of legal RUN lines in a .ll file:
-- ; RUN: llvm-as < %s | llvm-dis > %t1 - ; RUN: llvm-dis < %s.bc-13 > %t2 - ; RUN: diff %t1 %t2 -+ +
+; RUN: llvm-as < %s | llvm-dis > %t1 +; RUN: llvm-dis < %s.bc-13 > %t2 +; RUN: diff %t1 %t2 ++
As with a Unix shell, the RUN: lines permit pipelines and I/O redirection to be used. However, the usage is slightly different than for Bash. To check @@ -351,43 +368,66 @@ location of these external programs is configured by the llvm-test
There are some quoting rules that you must pay attention to when writing your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any ' or " so they will get passed to the invoked program. For example:
-- ... | grep 'find this string' -+ +
+... | grep 'find this string' ++
This will fail because the ' characters are passed to grep. This would instruction grep to look for 'find in the files this and string'. To avoid this use curly braces to tell Tcl that it should treat everything enclosed as one value. So our example would become:
-
- ... | grep {find this string}
-
+
+
+... | grep {find this string}
+
+Additionally, the characters [ and ] are treated specially by Tcl. They tell Tcl to interpret the content as a command to execute. Since these characters are often used in regular expressions this can have disastrous results and cause the entire test run in a directory to fail. For example, a common idiom is to look for some basicblock number:
-- ... | grep bb[2-8] -+ +
+... | grep bb[2-8] ++
This, however, will cause Tcl to fail because its going to try to execute a program named "2-8". Instead, what you want is this:
-
- ... | grep {bb\[2-8\]}
-
+
+
+... | grep {bb\[2-8\]}
+
+Finally, if you need to pass the \ character down to a program, then it must be doubled. This is another Tcl special character. So, suppose you had: -
- ... | grep 'i32\*' -+ +
+... | grep 'i32\*' ++
This will fail to match what you want (a pointer to i32). First, the ' do not get stripped off. Second, the \ gets stripped off by Tcl so what grep sees is: 'i32*'. That's not likely to match anything. To resolve this you must use \\ and the {}, like this:
-
- ... | grep {i32\\*}
-
+
+
+... | grep {i32\\*}
+
+Here are the available variable names. The alternate syntax is listed in parentheses.
+- ; XFAIL: darwin,sun,llvmgcc4 -+ +
+; XFAIL: darwin,sun,llvmgcc4 ++
To make the output more useful, the llvm_runtest function wil scan the lines of the test case for ones that contain a pattern that matches @@ -573,12 +638,14 @@ specify the following configuration options:
uses the default value /home/vadve/shared/benchmarks/speccpu2000/benchspec.+
+
The master Makefile in llvm/test is capable of running only the DejaGNU -driven tests. By default, it will run all of these tests.
+The master Makefile in llvm/test is capable of running only the +DejaGNU driven tests. By default, it will run all of these tests.
To run only the DejaGNU driven tests, run gmake at the command line in llvm/test. To run a specific directory of tests, use -the TESTSUITE variable. +the TESTSUITE variable.
For example, to run the Regression tests, type @@ -613,40 +680,54 @@ the TESTSUITE variable. llvm/test/Regression. You must use DejaGNU from the llvm/test directory to run them.
-To run the llvm-test suite, you need to use the following steps: -
+To run the llvm-test suite, you need to use the following steps:
+Check out the test-suite module with:
+ ++% svn co http://llvm.org/svn/llvm-project/test-suite/trunk llvm-test ++
This will get the test suite into llvm/projects/llvm-test
+ +Configure the test suite using llvm configure. This will automatically configure llvm-test. + You must do it from the top level otherwise llvm-gcc will not be set which is required to + run llvm-test:
++% cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure --with-llvmgccdir=$LLVM_GCC_DIR ++
Note that the second and third steps only need to be done once. After you have the suite checked out and configured, you don't need to do it again (unless -the test code or configure script changes).
+the test code or configure script changes). $LLVM_GCC_DIR is the path to the LLVM +C/C++ FrontEndTo make a specialized test (use one of the
-llvm-test/TEST.<type>.Makefiles), just run:
-gmake TEST=<type> test
For example, you could run the
-nightly tester tests using the following commands:
- % cd llvm/projects/llvm-test - % gmake TEST=nightly test +% gmake TEST=<type> test+
For example, you could run the nightly tester tests using the following +commands:
+ ++% cd llvm/projects/llvm-test +% gmake TEST=nightly test ++
Regardless of which test you're running, the results are printed on standard output and standard error. You can redirect these results to a file if you @@ -760,24 +841,15 @@ as keep track of LLVM's progress over time.
machine, take a look at the comments at the top of the utils/NewNightlyTest.pl file. If you decide to set up a nightly tester please choose a unique nickname and invoke utils/NewNightlyTest.pl -with the "-nickname [yournickname]" command line option. We usually run it -from a crontab entry that looks like this: - --5 3 * * * $HOME/llvm/utils/NewNightlyTest.pl -parallel -nickname Nickname \ - $CVSROOT $HOME/buildtest $HOME/cvs/testresults --
Or, you can create a shell script to encapsulate the running of the script. +
You can create a shell script to encapsulate the running of the script. The optimized x86 Linux nightly test is run from just such a script:
#!/bin/bash BASE=/proj/work/llvm/nightlytest -export CVSROOT=:pserver:anon@llvm.org:/var/cvs/llvm export BUILDDIR=$BASE/build export WEBDIR=$BASE/testresults export LLVMGCCDIR=/proj/work/llvm/cfrontend/install @@ -786,7 +858,7 @@ export LD_LIBRARY_PATH=/proj/install/lib cd $BASE cp /proj/work/llvm/llvm/utils/NewNightlyTest.pl . nice ./NewNightlyTest.pl -nice -release -verbose -parallel -enable-linscan \ - -nickname NightlyTester -noexternals 2>&1 > output.log + -nickname NightlyTester -noexternals > output.log 2>&1