To run all of the simple tests in LLVM using DejaGNU, use the master Makefile in the -llvm/test directory:

To run all of the simple tests in LLVM using DejaGNU, use the master Makefile + in the llvm/test directory:

@@ -174,7 +180,9 @@ determine correct behavior.

@@ -208,19 +216,27 @@ module.

subtrees of the test suite directory tree are as follows:

llvm/test/Features -
This directory contains sample codes that test various features of the -LLVM language. These pieces of sample code are run through various -assembler, disassembler, and optimizer passes.
-
llvm/test/Regression -
This directory contains regression tests for LLVM. When a bug is found -in LLVM, a regression test containing just enough code to reproduce the -problem should be written and placed somewhere underneath this directory. -In most cases, this will be a small piece of LLVM assembly language code, -often distilled from an actual application or benchmark.
-
llvm/test +
This directory contains a large array of small tests + that exercise various features of LLVM and to ensure that regressions do not + occur. The directory is broken into several sub-directories, each focused on + a particular area of LLVM. A few of the important ones are:
- Analysis: checks Analysis passes.
- Archive: checks the Archive library.
- Assembler: checks Assembly reader/writer functionality.
- Bytecode: checks Bytecode reader/writer functionality.
- CodeGen: checks code generation and each target.
- Features: checks various features of the LLVM language.
- Linker: tests bytecode linking.
- Transforms: tests each of the scalar, IPO, and utility + transforms to ensure they make the right transformations.
- Verifier: tests the IR verifier.
+
Typically when a bug is found in LLVM, a regression test containing + just enough code to reproduce the problem should be written and placed + somewhere underneath this directory. In most cases, this will be a small + piece of LLVM assembly language code, often distilled from an actual + application or benchmark.
llvm-test
The llvm-test CVS module contains programs that can be compiled @@ -259,79 +275,259 @@ location of these external programs is configured by the llvm-test
DejaGNU Structure
-
-
The LLVM test suite is partially driven by DejaGNU and partially -driven by GNU Make. Specifically, the Features and Regression tests -are all driven by DejaGNU. The llvm-test -module is currently driven by a set of Makefiles.
- -
The DejaGNU structure is very simple, but does require some -information to be set. This information is gathered via configure and -is written to a file, site.exp in llvm/test. The -llvm/test -Makefile does this work for you.
- -
In order for DejaGNU to work, each directory of tests must have a -dg.exp file. This file is a program written in tcl that calls -the llvm-runtests procedure on each test file. The -llvm-runtests procedure is defined in -llvm/test/lib/llvm-dg.exp. Any directory that contains only -directories does not need the dg.exp file.
- -
In order for a test to be run, it must contain information within -the test file on how to run the test. These are called RUN -lines. Run lines are specified in the comments of the test program -using the keyword RUN followed by a colon, and lastly the -commands to execute. These commands will be executed in a bash script, -so any bash syntax is acceptable. You can specify as many RUN lines as -necessary. Each RUN line translates to one line in the resulting bash -script. Below is an example of legal RUN lines in a .ll -file:
-
```
-; RUN: llvm-as < %s | llvm-dis > %t1
-; RUN: llvm-dis < %s.bc-13 > %t2
-; RUN: diff %t1 %t2
-
```
-
There are a couple patterns within a RUN line that the -llvm-runtest procedure looks for and replaces with the appropriate -syntax:
- -
-
%p
-
The path to the source directory. This is for locating -any supporting files that are not generated by the test, but used by -the test.
-
%s
-
The test file.
- -
%t
-
Temporary filename: testscript.test_filename.tmp, where -test_filename is the name of the test file. All temporary files are -placed in the Output directory within the directory the test is -located.
- -
%prcontext
-
Path to a script that performs grep -C. Use this since not all -platforms support grep -C.
- -
%llvmgcc

Full path to the llvm-gcc executable.
-
%llvmgxx

Full path to the llvm-g++ executable.
-
+
The LLVM test suite is partially driven by DejaGNU and partially driven by + GNU Make. Specifically, the Features and Regression tests are all driven by + DejaGNU. The llvm-test module is currently driven by a set of + Makefiles.
+ +
The DejaGNU structure is very simple, but does require some information to + be set. This information is gathered via configure and is written + to a file, site.exp in llvm/test. The llvm/test + Makefile does this work for you.
+ +
In order for DejaGNU to work, each directory of tests must have a + dg.exp file. DejaGNU looks for this file to determine how to run the + tests. This file is just a Tcl script and it can do anything you want, but + we've standardized it for the LLVM regression tests. It simply loads a Tcl + library (test/lib/llvm.exp) and calls the llvm_runtests + function defined in that library with a list of file names to run. The names + are obtained by using Tcl's glob command. Any directory that contains only + directories does not need the dg.exp file.
+ +
The llvm-runtests function lookas at each file that is passed to + it and gathers any lines together that match "RUN:". This are the "RUN" lines + that specify how the test is to be run. So, each test script must contain + RUN lines if it is to do anything. If there are no RUN lines, the + llvm-runtests function will issue an error and the test will + fail.
+ +
RUN lines are specified in the comments of the test program using the + keyword RUN followed by a colon, and lastly the command (pipeline) + to execute. Together, these lines form the "script" that + llvm-runtests executes to run the test case. The syntax of the + RUN lines is similar to a shell's syntax for pipelines including I/O + redirection and variable substitution. However, even though these lines + may look like a shell script, they are not. RUN lines are interpreted + directly by the Tcl exec command. They are never executed by a + shell. Consequently the syntax differs from normal shell script syntax in a + few ways. You can specify as many RUN lines as needed.
+ +
Each RUN line is executed on its own, distinct from other lines unless + its last character is \. This continuation character causes the RUN + line to be concatenated with the next one. In this way you can build up long + pipelines of commands without making huge line lengths. The lines ending in + \ are concatenated until a RUN line that doesn't end in \ is + found. This concatenated set or RUN lines then constitutes one execution. + Tcl will substitute variables and arrange for the pipeline to be executed. If + any process in the pipeline fails, the entire line (and test case) fails too. +
+ +
Below is an example of legal RUN lines in a .ll file:
+
```
+  ; RUN: llvm-as < %s | llvm-dis > %t1
+  ; RUN: llvm-dis < %s.bc-13 > %t2
+  ; RUN: diff %t1 %t2
+  
```
+ +
As with a Unix shell, the RUN: lines permit pipelines and I/O redirection + to be used. However, the usage is slightly different than for Bash. To check + what's legal, see the documentation for the + Tcl exec + command and the + tutorial. + The major differences are:
+
- You can't do 2>&1. That will cause Tcl to write to a + file named &1. Usually this is done to get stderr to go through + a pipe. You can do that in tcl with |& so replace this idiom: + ... 2>&1 | grep with ... |& grep
- You can only redirect to a file, not to another descriptor and not from + a here document.
- tcl supports redirecting to open files with the @ syntax but you + shouldn't use that here.
+ +
There are some quoting rules that you must pay attention to when writing + your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any + ' or " so they will get passed to the invoked program. For example:
+
```
+     ... | grep 'find this string'
+  
```
+
This will fail because the ' characters are passed to grep. This would + instruction grep to look for 'find in the files this and + string'. To avoid this use curly braces to tell Tcl that it should + treat everything enclosed as one value. So our example would become:
+
```
+     ... | grep {find this string}
+  
```
+
Additionally, the characters [ and ] are treated + specially by Tcl. They tell Tcl to interpret the content as a command to + execute. Since these characters are often used in regular expressions this can + have disastrous results and cause the entire test run in a directory to fail. + For example, a common idiom is to look for some basicblock number:
+
```
+     ... | grep bb[2-8]
+  
```
+
This, however, will cause Tcl to fail because its going to try to execute + a program named "2-8". Instead, what you want is this:
+
```
+     ... | grep {bb\[2-8\]}
+  
```
+
Finally, if you need to pass the \ character down to a program, + then it must be doubled. This is another Tcl special character. So, suppose + you had: +
```
+     ... | grep 'i32\*'
+  
```
+
This will fail to match what you want (a pointer to i32). First, the + ' do not get stripped off. Second, the \ gets stripped off + by Tcl so what grep sees is: 'i32*'. That's not likely to match + anything. To resolve this you must use \\ and the {}, like + this:
+
```
+     ... | grep {i32\\*}
+  
```
-
There are also several scripts in the llvm/test/Scripts directory -that you might find useful when writing RUN lines.
+
-
Lastly, you can easily mark a test that is expected to fail on a -specific platform by using the XFAIL keyword. Xfail lines are -specified in the comments of the test program using XFAIL, -followed by a colon, and one or more regular expressions (separated by -a comma) that will match against the target triplet for the -machine. You can use * to match all targets. Here is an example of an -XFAIL line:
-
```
-; XFAIL: darwin,sun
-
```
+ +
Vars And Substitutions
+
+
With a RUN line there are a number of substitutions that are permitted. In + general, any Tcl variable that is available in the substitute + function (in test/lib/llvm.exp) can be substituted into a RUN line. + To make a substitution just write the variable's name preceded by a $. + Additionally, for compatibility reasons with previous versions of the test + library, certain names can be accessed with an alternate syntax: a % prefix. + These alternates are deprecated and may go away in a future version. +
+ Here are the available variable names. The alternate syntax is listed in + parentheses.
+
+
$test (%s)
+
The full path to the test case's source. This is suitable for passing + on the command line as the input to an llvm tool.
+
$srcdir
+
The source directory from where the "make check" was run.
+
objdir
+
The object directory that corresponds to the $srcdir.
+
subdir
+
A partial path from the test directory that contains the + sub-directory that contains the test source being executed.
+
srcroot
+
The root directory of the LLVM src tree.
+
objroot
+
The root directory of the LLVM object tree. This could be the same + as the srcroot.
+
path
+
The path to the directory that contains the test case source. This is + for locating any supporting files that are not generated by the test, but + used by the test.
+
tmp
+
The path to a temporary file name that could be used for this test case. + The file name won't conflict with other test cases. You can append to it if + you need multiple temporaries. This is useful as the destination of some + redirected output.
+
llvmlibsdir (%llvmlibsdir)
+
The directory where the LLVM libraries are located.
+
target_triplet (%target_triplet)
+
The target triplet that corresponds to the current host machine (the one + running the test cases). This should probably be called "host".
+
prcontext (%prcontext)
+
Path to the prcontext tcl script that prints some context around a + line that matches a pattern. This isn't strictly necessary as the test suite + is run with its PATH altered to include the test/Scripts directory where + the prcontext script is located. Note that this script is similar to + grep -C but you should use the prcontext script because + not all platforms support grep -C.
+
llvmgcc (%llvmgcc)
+
The full path to the llvm-gcc executable as specified in the + configured LLVM environment
+
llvmgxx (%llvmgxx)
+
The full path to the llvm-gxx executable as specified in the + configured LLVM environment
+
llvmgcc_version (%llvmgcc_version)
+
The full version number of the llvm-gcc executable.
+
llvmgccmajvers (%llvmgccmajvers)
+
The major version number of the llvm-gcc executable.
+
gccpath
+
The full path to the C compiler used to build LLVM. Note that + this might not be gcc.
+
gxxpath
+
The full path to the C++ compiler used to build LLVM. Note that + this might not be g++.
+
compile_c (%compile_c)
+
The full command line used to compile LLVM C source code. This has all + the configured -I, -D and optimization options.
+
compile_cxx (%compile_cxx)
+
The full command used to compile LLVM C++ source code. This has + all the configured -I, -D and optimization options.
+
link (%link)
+
This full link command used to link LLVM executables. This has all the + configured -I, -L and -l options.
+
shlibext (%shlibext)
+
The suffix for the host platforms share library (dll) files. This + includes the period as the first character.
+
+
To add more variables, two things need to be changed. First, add a line in + the test/Makefile that creates the site.exp file. This will + "set" the variable as a global in the site.exp file. Second, in the + test/lib/llvm.exp file, in the substitute proc, add the variable name + to the list of "global" declarations at the beginning of the proc. That's it, + the variable can then be used in test scripts.
+
+ + +
Other Features
+
+
To make RUN line writing easier, there are several shell scripts located + in the llvm/test/Scripts directory. For example:
+
+
ignore
+
This script runs its arguments and then always returns 0. This is useful + in cases where the test needs to cause a tool to generate an error (e.g. to + check the error output). However, any program in a pipeline that returns a + non-zero result will cause the test to fail. This script overcomes that + issue and nicely documents that the test case is purposefully ignoring the + result code of the tool
+
not
+
This script runs its arguments and then inverts the result code from + it. Zero result codes become 1. Non-zero result codes become 0. This is + useful to invert the result of a grep. For example "not grep X" means + succeed only if you don't find X in the input.
+
+ +
Sometimes it is necessary to mark a test case as "expected fail" or XFAIL. + You can easily mark a test as XFAIL just by including XFAIL: on a + line near the top of the file. This signals that the test case should succeed + if the test fails. Such test cases are counted separately by DejaGnu. To + specify an expected fail, use the XFAIL keyword in the comments of the test + program followed by a colon and one or more regular expressions (separated by + a comma). The regular expressions allow you to XFAIL the test conditionally + by host platform. The regular expressions following the : are matched against + the target triplet or llvmgcc version number for the host machine. If there is + a match, the test is expected to fail. If not, the test is expected to + succeed. To XFAIL everywhere just specify XFAIL: *. When matching + the llvm-gcc version, you can specify the major (e.g. 3) or full version + (i.e. 3.4) number. Here is an example of an XFAIL line:
+
```
+   ; XFAIL: darwin,sun,llvmgcc4
+  
```
+ +
To make the output more useful, the llvm_runtest function wil + scan the lines of the test case for ones that contain a pattern that matches + PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that + is related to the test case. The numer after "PR" specifies the LLVM bugzilla + number. When a PR number is specified, it will be used in the pass/fail + reporting. This is useful to quickly get some context when a test fails.
+ +
Finally, any line that contains "END." will cause the special + interpretation of lines to terminate. This is generally done right after the + last RUN: line. This has two side effects: (a) it prevents special + interpretation of lines that are part of the test program, not the + instructions to the test case, and (b) it speeds things up for really big test + cases by avoiding interpretation of the remainder of the file.
@@ -402,8 +598,8 @@ specify the following configuration options:
are not executed inside of the LLVM source tree. This is because the test suite creates temporary files during execution.
-
The master Makefile in llvm/test is capable of running only the DejaGNU driven -tests. By default, it will run all of these tests.
+
The master Makefile in llvm/test is capable of running only the DejaGNU +driven tests. By default, it will run all of these tests.

To run only the DejaGNU driven tests, run gmake at the command line in llvm/test. To run a specific directory of tests, use @@ -432,7 +628,8 @@ directory to run them.
properly configured.
Use the configure script found in the llvm-test source directory:
- $LLVM_SRC_ROOT/projects/llvm-test/configure --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT + $LLVM_SRC_ROOT/projects/llvm-test/configure + --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT
gmake

+ +

Assuming you can run llvm-test, (e.g. "gmake TEST=nightly report" +should work), it is really easy to run optimizations or code generator +components against every program in the tree, collecting statistics or running +custom checks for correctness. At base, this is how the nightly tester works, +it's just one example of a general framework.

+ +

Lets say that you have an LLVM optimization pass, and you want to see how +many times it triggers. First thing you should do is add an LLVM +statistic to your pass, which +will tally counts of things you care about.

+ +

Following this, you can set up a test and a report that collects these and +formats them for easy viewing. This consists of two files, an +"llvm-test/TEST.XXX.Makefile" fragment (where XXX is the name of your +test) and an "llvm-test/TEST.XXX.report" file that indicates how to +format the output into a table. There are many example reports of various +levels of sophistication included with llvm-test, and the framework is very +general.

+ +

If you are interested in testing an optimization pass, check out the +"libcalls" test as an example. It can be run like this:

+ +

+% cd llvm/projects/llvm-test/MultiSource/Benchmarks  # or some other level
+% make TEST=libcalls report
+

+ +

This will do a bunch of stuff, then eventually print a table like this:

+ +

+Name                                  | total | #exit |
+...
+FreeBench/analyzer/analyzer           | 51    | 6     | 
+FreeBench/fourinarow/fourinarow       | 1     | 1     | 
+FreeBench/neural/neural               | 19    | 9     | 
+FreeBench/pifft/pifft                 | 5     | 3     | 
+MallocBench/cfrac/cfrac               | 1     | *     | 
+MallocBench/espresso/espresso         | 52    | 12    | 
+MallocBench/gs/gs                     | 4     | *     | 
+Prolangs-C/TimberWolfMC/timberwolfmc  | 302   | *     | 
+Prolangs-C/agrep/agrep                | 33    | 12    | 
+Prolangs-C/allroots/allroots          | *     | *     | 
+Prolangs-C/assembler/assembler        | 47    | *     | 
+Prolangs-C/bison/mybison              | 74    | *     | 
+...
+

+ +

This basically is grepping the -stats output and displaying it in a table. +You can also use the "TEST=libcalls report.html" target to get the table in HTML +form, similarly for report.csv and report.tex.

+ +

The source for this is in llvm-test/TEST.libcalls.*. The format is pretty +simple: the Makefile indicates how to run the test (in this case, +"opt -simplify-libcalls -stats"), and the report contains one line for +each column of the output. The first value is the header for the column and the +second is the regex to grep the output of the command for. There are lots of +example reports that can do fancy stuff.

+ +

-The LLVM Nightly Testers +The LLVM Nightly Testers automatically check out an LLVM tree, build it, run the "nightly" -program test (described above), run all of the feature and regression tests, -and then delete the checked out tree. This tester is designed to ensure that -programs don't break as well as keep track of LLVM's progress over time.

- -

If you'd like to set up an instance of the nightly tester to run on your -machine, take a look at the comments at the top of the -utils/NightlyTester.pl file. We usually run it from a crontab entry -that looks ilke this:

+program test (described above), run all of the feature and regression tests, +delete the checked out tree, and then submit the results to +http://llvm.org/nightlytest/. +After test results are submitted to +http://llvm.org/nightlytest/, +they are processed and displayed on the tests page. An email to + +llvm-testresults@cs.uiuc.edu summarizing the results is also generated. +This testing scheme is designed to ensure that programs don't break as well +as keep track of LLVM's progress over time.

+ +

If you'd like to set up an instance of the nightly tester to run on your +machine, take a look at the comments at the top of the +utils/NewNightlyTest.pl file. If you decide to set up a nightly tester +please choose a unique nickname and invoke utils/NewNightlyTest.pl +with the "-nickname [yournickname]" command line option. We usually run it +from a crontab entry that looks like this:

-5 3 * * *  $HOME/llvm/utils/NightlyTest.pl -parallel $CVSROOT $HOME/buildtest-X86 $HOME/cvs/testresults-X86
+5 3 * * *  $HOME/llvm/utils/NewNightlyTest.pl -parallel -nickname Nickname \
+           $CVSROOT $HOME/buildtest $HOME/cvs/testresults

@@ -505,15 +784,24 @@ export LLVMGCCDIR=/proj/work/llvm/cfrontend/install export PATH=/proj/install/bin:$LLVMGCCDIR/bin:$PATH export LD_LIBRARY_PATH=/proj/install/lib cd $BASE -cp /proj/work/llvm/llvm/utils/NightlyTest.pl . -nice ./NightlyTest.pl -nice -release -verbose -parallel -enable-linscan -noexternals 2>&1 > output.log -mail -s 'X86 nightly tester results' llvm-testresults@cs.uiuc.edu < output.log +cp /proj/work/llvm/llvm/utils/NewNightlyTest.pl . +nice ./NewNightlyTest.pl -nice -release -verbose -parallel -enable-linscan \ + -nickname NightlyTester -noexternals 2>&1 > output.log

Take a look at the NightlyTest.pl file to see what all of the flags -and strings do. If you start running the nightly tests, please let us know and -we'll link your page to the global tester page. Thanks!

It is also possible to specify the the location your nightly test results +are submitted. You can do this by passing the command line option +"-submit-server [server_address]" and "-submit-script [script_on_server]" to +utils/NewNightlyTest.pl. For example, to submit to the llvm.org +nightly test results page, you would invoke the nightly test script with +"-submit-server llvm.org -submit-script /nightlytest/NightlyTestAccept.cgi". +If these options are not specified, the nightly test script sends the results +to the llvm.org nightly test results page.

Take a look at the NewNightlyTest.pl file to see what all of the +flags and strings do. If you start running the nightly tests, please let us +know. Thanks!