"http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
- <title>LLVM Test Suite Guide</title>
+ <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
+ <title>LLVM Testing Infrastructure Guide</title>
<link rel="stylesheet" href="llvm.css" type="text/css">
</head>
<body>
-<div class="doc_title">
- LLVM Test Suite Guide
-</div>
+<h1>
+ LLVM Testing Infrastructure Guide
+</h1>
<ol>
<li><a href="#overview">Overview</a></li>
- <li><a href="#Requirements">Requirements</a></li>
- <li><a href="#quick">Quick Start</a></li>
- <li><a href="#org">LLVM Test Suite Organization</a>
+ <li><a href="#requirements">Requirements</a></li>
+ <li><a href="#org">LLVM testing infrastructure organization</a>
<ul>
- <li><a href="#codefragments">Code Fragments</a></li>
- <li><a href="#wholeprograms">Whole Programs</a></li>
+ <li><a href="#regressiontests">Regression tests</a></li>
+ <li><a href="#testsuite"><tt>test-suite</tt></a></li>
+ <li><a href="#debuginfotests">Debugging Information tests</a></li>
</ul>
</li>
- <li><a href="#tree">LLVM Test Suite Tree</a></li>
- <li><a href="#dgstructure">DejaGNU Structure</a></li>
- <li><a href="#progstructure"><tt>llvm-test</tt> Structure</a></li>
- <li><a href="#run">Running the LLVM Tests</a>
+ <li><a href="#quick">Quick start</a>
<ul>
- <li><a href="#customtest">Writing custom tests for llvm-test</a></li>
- </ul>
+ <li><a href="#quickregressiontests">Regression tests</a></li>
+ <li><a href="#quickdebuginfotests">Debugging Information tests</a></li>
+ </ul>
+ </li>
+ <li><a href="#rtstructure">Regression test structure</a>
+ <ul>
+ <li><a href="#rtcustom">Writing new regression tests</a></li>
+ <li><a href="#FileCheck">The FileCheck utility</a></li>
+ <li><a href="#rtvars">Variables and substitutions</a></li>
+ <li><a href="#rtfeatures">Other features</a></li>
+ </ul>
+ </li>
+ <li><a href="#testsuiteoverview"><tt>test-suite</tt> Overview</a>
+ <ul>
+ <li><a href="#testsuitequickstart"><tt>test-suite</tt> Quickstart</a></li>
+ <li><a href="#testsuitemakefiles"><tt>test-suite</tt> Makefiles</a></li>
+ </ul>
</li>
- <li><a href="#nightly">Running the nightly tester</a></li>
</ol>
<div class="doc_author">
- <p>Written by John T. Criswell, <a
- href="http://llvm.x10sys.com/rspencer">Reid Spencer</a>, and Tanya Lattner</p>
+ <p>Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner</p>
</div>
<!--=========================================================================-->
-<div class="doc_section"><a name="overview">Overview</a></div>
+<h2><a name="overview">Overview</a></h2>
<!--=========================================================================-->
-<div class="doc_text">
+<div>
-<p>This document is the reference manual for the LLVM test suite. It documents
-the structure of the LLVM test suite, the tools needed to use it, and how to add
-and run tests.</p>
+<p>This document is the reference manual for the LLVM testing infrastructure. It
+documents the structure of the LLVM testing infrastructure, the tools needed to
+use it, and how to add and run tests.</p>
</div>
<!--=========================================================================-->
-<div class="doc_section"><a name="Requirements">Requirements</a></div>
+<h2><a name="requirements">Requirements</a></h2>
<!--=========================================================================-->
-<div class="doc_text">
-
-<p>In order to use the LLVM test suite, you will need all of the software
-required to build LLVM, plus the following:</p>
-
-<dl>
-<dt><a href="http://www.gnu.org/software/dejagnu/">DejaGNU</a></dt>
-<dd>The Feature and Regressions tests are organized and run by DejaGNU.</dd>
-<dt><a href="http://expect.nist.gov/">Expect</a></dt>
-<dd>Expect is required by DejaGNU.</dd>
-<dt><a href="http://www.tcl.tk/software/tcltk/">tcl</a></dt>
-<dd>Tcl is required by DejaGNU. </dd>
-
-<dt><a href="http://www.netlib.org/f2c">F2C</a></dt>
-<dd>For now, LLVM does not have a Fortran front-end, but using F2C, we can run
-Fortran benchmarks. F2C support must be enabled via <tt>configure</tt> if not
-installed in a standard place. F2C requires three items: the <tt>f2c</tt>
-executable, <tt>f2c.h</tt> to compile the generated code, and <tt>libf2c.a</tt>
-to link generated code. By default, given an F2C directory <tt>$DIR</tt>, the
-configure script will search <tt>$DIR/bin</tt> for <tt>f2c</tt>,
-<tt>$DIR/include</tt> for <tt>f2c.h</tt>, and <tt>$DIR/lib</tt> for
-<tt>libf2c.a</tt>. The default <tt>$DIR</tt> values are: <tt>/usr</tt>,
-<tt>/usr/local</tt>, <tt>/sw</tt>, and <tt>/opt</tt>. If you installed F2C in a
-different location, you must tell <tt>configure</tt>:
+<div>
-<ul>
-<li><tt>./configure --with-f2c=$DIR</tt><br>
-This will specify a new <tt>$DIR</tt> for the above-described search
-process. This will only work if the binary, header, and library are in their
-respective subdirectories of <tt>$DIR</tt>.</li>
+<p>In order to use the LLVM testing infrastructure, you will need all of the
+software required to build LLVM, as well
+as <a href="http://python.org">Python</a> 2.4 or later.</p>
-<li><tt>./configure --with-f2c-bin=/binary/path --with-f2c-inc=/include/path
---with-f2c-lib=/lib/path</tt><br>
-This allows you to specify the F2C components separately. Note: if you choose
-this route, you MUST specify all three components, and you need to only specify
-<em>directories</em> where the files are located; do NOT include the
-filenames themselves on the <tt>configure</tt> line.</li>
-</ul></dd>
-</dl>
+</div>
-<p>Darwin (Mac OS X) developers can simplify the installation of Expect and tcl
-by using fink. <tt>fink install expect</tt> will install both. Alternatively,
-Darwinports users can use <tt>sudo port install expect</tt> to install Expect
-and tcl.</p>
+<!--=========================================================================-->
+<h2><a name="org">LLVM testing infrastructure organization</a></h2>
+<!--=========================================================================-->
+
+<div>
+
+<p>The LLVM testing infrastructure contains two major categories of tests:
+regression tests and whole programs. The regression tests are contained inside
+the LLVM repository itself under <tt>llvm/test</tt> and are expected to always
+pass -- they should be run before every commit.</p>
+
+<p>The whole programs tests are referred to as the "LLVM test suite" (or
+"test-suite") and are in the <tt>test-suite</tt> module in subversion. For
+historical reasons, these tests are also referred to as the "nightly tests" in
+places, which is less ambiguous than "test-suite" and remains in use although we
+run them much more often than nightly.</p>
+
+<!-- _______________________________________________________________________ -->
+<h3><a name="regressiontests">Regression tests</a></h3>
+<!-- _______________________________________________________________________ -->
+
+<div>
+
+<p>The regression tests are small pieces of code that test a specific feature of
+LLVM or trigger a specific bug in LLVM. They are usually written in LLVM
+assembly language, but can be written in other languages if the test targets a
+particular language front end (and the appropriate <tt>--with-llvmgcc</tt>
+options were used at <tt>configure</tt> time of the <tt>llvm</tt> module). These
+tests are driven by the 'lit' testing tool, which is part of LLVM.</p>
+
+<p>These code fragments are not complete programs. The code generated
+from them is never executed to determine correct behavior.</p>
+
+<p>These code fragment tests are located in the <tt>llvm/test</tt>
+directory.</p>
+
+<p>Typically when a bug is found in LLVM, a regression test containing
+just enough code to reproduce the problem should be written and placed
+somewhere underneath this directory. In most cases, this will be a small
+piece of LLVM assembly language code, often distilled from an actual
+application or benchmark.</p>
+
+</div>
+
+<!-- _______________________________________________________________________ -->
+<h3><a name="testsuite"><tt>test-suite</tt></a></h3>
+<!-- _______________________________________________________________________ -->
+
+<div>
+
+<p>The test suite contains whole programs, which are pieces of code which can be
+compiled and linked into a stand-alone program that can be executed. These
+programs are generally written in high level languages such as C or C++.</p>
+
+<p>These programs are compiled using a user specified compiler and set of flags,
+and then executed to capture the program output and timing information. The
+output of these programs is compared to a reference output to ensure that the
+program is being compiled correctly.</p>
+
+<p>In addition to compiling and executing programs, whole program tests serve as
+a way of benchmarking LLVM performance, both in terms of the efficiency of the
+programs generated as well as the speed with which LLVM compiles, optimizes, and
+generates code.</p>
+
+<p>The test-suite is located in the <tt>test-suite</tt> Subversion module.</p>
+
+</div>
+
+<!-- _______________________________________________________________________ -->
+<h3><a name="debuginfotests">Debugging Information tests</a></h3>
+<!-- _______________________________________________________________________ -->
+
+<div>
+
+<p>The test suite contains tests to check quality of debugging information.
+The test are written in C based languages or in LLVM assembly language. </p>
+
+<p>These tests are compiled and run under a debugger. The debugger output
+is checked to validate of debugging information. See README.txt in the
+test suite for more information . This test suite is located in the
+<tt>debuginfo-tests</tt> Subversion module. </p>
+
+</div>
</div>
<!--=========================================================================-->
-<div class="doc_section"><a name="quick">Quick Start</a></div>
+<h2><a name="quick">Quick start</a></h2>
<!--=========================================================================-->
-<div class="doc_text">
-
-<p>The tests are located in two separate CVS modules. The basic feature and
-regression tests are in the main "llvm" module under the directory
-<tt>llvm/test</tt>. A more comprehensive test suite that includes whole
-programs in C and C++ is in the <tt>llvm-test</tt> module. This module should
-be checked out to the <tt>llvm/projects</tt> directory. When you
-<tt>configure</tt> the <tt>llvm</tt> module, the <tt>llvm-test</tt> module
-will be automatically configured. Alternatively, you can configure the
- <tt>llvm-test</tt> module manually.</p>
-<p>To run all of the simple tests in LLVM using DejaGNU, use the master Makefile
- in the <tt>llvm/test</tt> directory:</p>
+<div>
+
+ <p>The tests are located in two separate Subversion modules. The regressions
+ tests are in the main "llvm" module under the directory
+ <tt>llvm/test</tt> (so you get these tests for free with the main llvm
+ tree). Use "make check-all" to run the regression tests after building
+ LLVM.</p>
+
+ <p>The more comprehensive test suite that includes whole programs in C and C++
+ is in the <tt>test-suite</tt>
+ module. See <a href="#testsuitequickstart"><tt>test-suite</tt> Quickstart</a>
+ for more information on running these tests.</p>
+
+<!-- _______________________________________________________________________ -->
+<h3><a name="quickregressiontests">Regression tests</a></h3>
+<div>
+<!-- _______________________________________________________________________ -->
+<p>To run all of the LLVM regression tests, use master Makefile in
+ the <tt>llvm/test</tt> directory:</p>
+
+<div class="doc_code">
<pre>
% gmake -C llvm/test
</pre>
-or<br>
-<pre>
-% gmake check
-</pre>
+</div>
+
+<p>or</p>
-<p>To run only a subdirectory of tests in llvm/test using DejaGNU (ie.
-Regression/Transforms), just set the TESTSUITE variable to the path of the
-subdirectory (relative to <tt>llvm/test</tt>):</p>
+<div class="doc_code">
<pre>
-% gmake -C llvm/test TESTSUITE=Regression/Transforms
+% gmake check
</pre>
+</div>
-<p><b>Note: If you are running the tests with <tt>objdir != subdir</tt>, you
-must have run the complete testsuite before you can specify a
-subdirectory.</b></p>
+<p>If you have <a href="http://clang.llvm.org/">Clang</a> checked out and built,
+you can run the LLVM and Clang tests simultaneously using:</p>
-<p>To run the comprehensive test suite (tests that compile and execute whole
-programs), run the <tt>llvm-test</tt> tests:</p>
+<p>or</p>
+<div class="doc_code">
<pre>
-% cd llvm/projects
-% cvs co llvm-test
-% cd llvm-test
-% ./configure --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT
-% gmake
+% gmake check-all
</pre>
-
</div>
-<!--=========================================================================-->
-<div class="doc_section"><a name="org">LLVM Test Suite Organization</a></div>
-<!--=========================================================================-->
-
-<div class="doc_text">
-
-<p>The LLVM test suite contains two major categories of tests: code
-fragments and whole programs. Code fragments are in the <tt>llvm</tt> module
-under the <tt>llvm/test</tt> directory. The whole programs
-test suite is in the <tt>llvm-test</tt> module under the main directory.</p>
+<p>To run the tests with Valgrind (Memcheck by default), just append
+<tt>VG=1</tt> to the commands above, e.g.:</p>
+<div class="doc_code">
+<pre>
+% gmake check VG=1
+</pre>
</div>
-<!-- _______________________________________________________________________ -->
-<div class="doc_subsection"><a name="codefragments">Code Fragments</a></div>
-<!-- _______________________________________________________________________ -->
+<p>To run individual tests or subsets of tests, you can use the 'llvm-lit'
+script which is built as part of LLVM. For example, to run the
+'Integer/BitCast.ll' test by itself you can run:</p>
-<div class="doc_text">
+<div class="doc_code">
+<pre>
+% llvm-lit ~/llvm/test/Integer/BitCast.ll
+</pre>
+</div>
-<p>Code fragments are small pieces of code that test a specific feature of LLVM
-or trigger a specific bug in LLVM. They are usually written in LLVM assembly
-language, but can be written in other languages if the test targets a particular
-language front end.</p>
+<p>or to run all of the ARM CodeGen tests:</p>
-<p>Code fragments are not complete programs, and they are never executed to
-determine correct behavior.</p>
+<div class="doc_code">
+<pre>
+% llvm-lit ~/llvm/test/CodeGen/ARM
+</pre>
+</div>
-<p>These code fragment tests are located in the <tt>llvm/test/Features</tt> and
-<tt>llvm/test/Regression</tt> directories.</p>
+<p>For more information on using the 'lit' tool, see 'llvm-lit --help' or the
+'lit' man page.</p>
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection"><a name="wholeprograms">Whole Programs</a></div>
+<h3><a name="quickdebuginfotests">Debugging Information tests</a></h3>
+<div>
<!-- _______________________________________________________________________ -->
+<div>
-<div class="doc_text">
+<p> To run debugging information tests simply checkout the tests inside
+clang/test directory. </p>
-<p>Whole Programs are pieces of code which can be compiled and linked into a
-stand-alone program that can be executed. These programs are generally written
-in high level languages such as C or C++, but sometimes they are written
-straight in LLVM assembly.</p>
+<div class="doc_code">
+<pre>
+%cd clang/test
+% svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
+</pre>
+</div>
-<p>These programs are compiled and then executed using several different
-methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation,
-etc). The output of these programs is compared to ensure that LLVM is compiling
-the program correctly.</p>
+<p> These tests are already set up to run as part of clang regression tests.</p>
-<p>In addition to compiling and executing programs, whole program tests serve as
-a way of benchmarking LLVM performance, both in terms of the efficiency of the
-programs generated as well as the speed with which LLVM compiles, optimizes, and
-generates code.</p>
+</div>
-<p>All "whole program" tests are located in the <tt>llvm-test</tt> CVS
-module.</p>
+</div>
</div>
<!--=========================================================================-->
-<div class="doc_section"><a name="tree">LLVM Test Suite Tree</a></div>
+<h2><a name="rtstructure">Regression test structure</a></h2>
<!--=========================================================================-->
+<div>
+ <p>The LLVM regression tests are driven by 'lit' and are located in
+ the <tt>llvm/test</tt> directory.
-<div class="doc_text">
-
-<p>Each type of test in the LLVM test suite has its own directory. The major
-subtrees of the test suite directory tree are as follows:</p>
-
-<ul>
- <li><tt>llvm/test</tt>
<p>This directory contains a large array of small tests
that exercise various features of LLVM and to ensure that regressions do not
occur. The directory is broken into several sub-directories, each focused on
- a particular area of LLVM. A few of the important ones are:<ul>
+ a particular area of LLVM. A few of the important ones are:</p>
+
+ <ul>
<li><tt>Analysis</tt>: checks Analysis passes.</li>
<li><tt>Archive</tt>: checks the Archive library.</li>
<li><tt>Assembler</tt>: checks Assembly reader/writer functionality.</li>
- <li><tt>Bytecode</tt>: checks Bytecode reader/writer functionality.</li>
+ <li><tt>Bitcode</tt>: checks Bitcode reader/writer functionality.</li>
<li><tt>CodeGen</tt>: checks code generation and each target.</li>
<li><tt>Features</tt>: checks various features of the LLVM language.</li>
- <li><tt>Linker</tt>: tests bytecode linking.</li>
+ <li><tt>Linker</tt>: tests bitcode linking.</li>
<li><tt>Transforms</tt>: tests each of the scalar, IPO, and utility
transforms to ensure they make the right transformations.</li>
<li><tt>Verifier</tt>: tests the IR verifier.</li>
- </ul></p>
- <p>Typically when a bug is found in LLVM, a regression test containing
- just enough code to reproduce the problem should be written and placed
- somewhere underneath this directory. In most cases, this will be a small
- piece of LLVM assembly language code, often distilled from an actual
- application or benchmark.</p></li>
-
-<li><tt>llvm-test</tt>
-<p>The <tt>llvm-test</tt> CVS module contains programs that can be compiled
-with LLVM and executed. These programs are compiled using the native compiler
-and various LLVM backends. The output from the program compiled with the
-native compiler is assumed correct; the results from the other programs are
-compared to the native program output and pass if they match.</p>
-
-<p>In addition for testing correctness, the <tt>llvm-test</tt> directory also
-performs timing tests of various LLVM optimizations. It also records
-compilation times for the compilers and the JIT. This information can be
-used to compare the effectiveness of LLVM's optimizations and code
-generation.</p></li>
-
-<li><tt>llvm-test/SingleSource</tt>
-<p>The SingleSource directory contains test programs that are only a single
-source file in size. These are usually small benchmark programs or small
-programs that calculate a particular value. Several such programs are grouped
-together in each directory.</p></li>
-
-<li><tt>llvm-test/MultiSource</tt>
-<p>The MultiSource directory contains subdirectories which contain entire
-programs with multiple source files. Large benchmarks and whole applications
-go here.</p></li>
+ </ul>
-<li><tt>llvm-test/External</tt>
-<p>The External directory contains Makefiles for building code that is external
-to (i.e., not distributed with) LLVM. The most prominent members of this
-directory are the SPEC 95 and SPEC 2000 benchmark suites. The presence and
-location of these external programs is configured by the llvm-test
-<tt>configure</tt> script.</p></li>
-
-</ul>
+<!-- _______________________________________________________________________ -->
+<h3><a name="rtcustom">Writing new regression tests</a></h3>
+<!-- _______________________________________________________________________ -->
+<div>
+ <p>The regression test structure is very simple, but does require some
+ information to be set. This information is gathered via <tt>configure</tt> and
+ is written to a file, <tt>lit.site.cfg</tt>
+ in <tt>llvm/test</tt>. The <tt>llvm/test</tt> Makefile does this work for
+ you.</p>
+
+ <p>In order for the regression tests to work, each directory of tests must
+ have a <tt>lit.local.cfg</tt> file. Lit looks for this file to determine how
+ to run the tests. This file is just Python code and thus is very flexible,
+ but we've standardized it for the LLVM regression tests. If you're adding a
+ directory of tests, just copy <tt>lit.local.cfg</tt> from another directory to
+ get running. The standard <tt>lit.local.cfg</tt> simply specifies which files
+ to look in for tests. Any directory that contains only directories does not
+ need the <tt>lit.local.cfg</tt> file. Read the
+ <a href="http://llvm.org/cmds/lit.html">Lit documentation</a> for more
+ information. </p>
+
+ <p>The <tt>llvm-runtests</tt> function looks at each file that is passed to
+ it and gathers any lines together that match "RUN:". These are the "RUN" lines
+ that specify how the test is to be run. So, each test script must contain
+ RUN lines if it is to do anything. If there are no RUN lines, the
+ <tt>llvm-runtests</tt> function will issue an error and the test will
+ fail.</p>
+
+ <p>RUN lines are specified in the comments of the test program using the
+ keyword <tt>RUN</tt> followed by a colon, and lastly the command (pipeline)
+ to execute. Together, these lines form the "script" that
+ <tt>llvm-runtests</tt> executes to run the test case. The syntax of the
+ RUN lines is similar to a shell's syntax for pipelines including I/O
+ redirection and variable substitution. However, even though these lines
+ may <i>look</i> like a shell script, they are not. RUN lines are interpreted
+ directly by the Tcl <tt>exec</tt> command. They are never executed by a
+ shell. Consequently the syntax differs from normal shell script syntax in a
+ few ways. You can specify as many RUN lines as needed.</p>
+
+ <p>lit performs substitution on each RUN line to replace LLVM tool
+ names with the full paths to the executable built for each tool (in
+ $(LLVM_OBJ_ROOT)/$(BuildMode)/bin). This ensures that lit does not
+ invoke any stray LLVM tools in the user's path during testing.</p>
+
+ <p>Each RUN line is executed on its own, distinct from other lines unless
+ its last character is <tt>\</tt>. This continuation character causes the RUN
+ line to be concatenated with the next one. In this way you can build up long
+ pipelines of commands without making huge line lengths. The lines ending in
+ <tt>\</tt> are concatenated until a RUN line that doesn't end in <tt>\</tt> is
+ found. This concatenated set of RUN lines then constitutes one execution.
+ Tcl will substitute variables and arrange for the pipeline to be executed. If
+ any process in the pipeline fails, the entire line (and test case) fails too.
+ </p>
+
+ <p> Below is an example of legal RUN lines in a <tt>.ll</tt> file:</p>
+<div class="doc_code">
+<pre>
+; RUN: llvm-as < %s | llvm-dis > %t1
+; RUN: llvm-dis < %s.bc-13 > %t2
+; RUN: diff %t1 %t2
+</pre>
</div>
-<!--=========================================================================-->
-<div class="doc_section"><a name="dgstructure">DejaGNU Structure</a></div>
-<!--=========================================================================-->
-<div class="doc_text">
-<p>The LLVM test suite is partially driven by DejaGNU and partially
-driven by GNU Make. Specifically, the Features and Regression tests
-are all driven by DejaGNU. The <tt>llvm-test</tt>
-module is currently driven by a set of Makefiles.</p>
-
-<p>The DejaGNU structure is very simple, but does require some
-information to be set. This information is gathered via <tt>configure</tt> and
-is written to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The
-<tt>llvm/test</tt>
-Makefile does this work for you.</p>
-
-<p>In order for DejaGNU to work, each directory of tests must have a
-<tt>dg.exp</tt> file. This file is a program written in tcl that calls
-the <tt>llvm-runtests</tt> procedure on each test file. The
-llvm-runtests procedure is defined in
-<tt>llvm/test/lib/llvm-dg.exp</tt>. Any directory that contains only
-directories does not need the <tt>dg.exp</tt> file.</p>
-
-<p>In order for a test to be run, it must contain information within
-the test file on how to run the test. These are called <tt>RUN</tt>
-lines. Run lines are specified in the comments of the test program
-using the keyword <tt>RUN</tt> followed by a colon, and lastly the
-commands to execute. These commands will be executed in a bash script,
-so any bash syntax is acceptable. You can specify as many RUN lines as
-necessary. Each RUN line translates to one line in the resulting bash
-script. Below is an example of legal RUN lines in a <tt>.ll</tt>
-file:</p>
+ <p>As with a Unix shell, the RUN: lines permit pipelines and I/O redirection
+ to be used. However, the usage is slightly different than for Bash. To check
+ what's legal, see the documentation for the
+ <a href="http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M2">Tcl exec</a>
+ command and the
+ <a href="http://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html">tutorial</a>.
+ The major differences are:</p>
+ <ul>
+ <li>You can't do <tt>2>&1</tt>. That will cause Tcl to write to a
+ file named <tt>&1</tt>. Usually this is done to get stderr to go through
+ a pipe. You can do that in tcl with <tt>|&</tt> so replace this idiom:
+ <tt>... 2>&1 | grep</tt> with <tt>... |& grep</tt></li>
+ <li>You can only redirect to a file, not to another descriptor and not from
+ a here document.</li>
+ <li>tcl supports redirecting to open files with the @ syntax but you
+ shouldn't use that here.</li>
+ </ul>
+
+ <p>There are some quoting rules that you must pay attention to when writing
+ your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any
+ quote characters so they will get passed to the invoked program. For
+ example:</p>
+
+<div class="doc_code">
<pre>
-; RUN: llvm-as < %s | llvm-dis > %t1
-; RUN: llvm-dis < %s.bc-13 > %t2
-; RUN: diff %t1 %t2
+... | grep 'find this string'
</pre>
-<p>There are a couple patterns within a <tt>RUN</tt> line that the
-llvm-runtest procedure looks for and replaces with the appropriate
-syntax:</p>
-
-<dl style="margin-left: 25px">
-<dt>%p</dt>
-<dd>The path to the source directory. This is for locating
-any supporting files that are not generated by the test, but used by
-the test.</dd>
-<dt>%s</dt>
-<dd>The test file.</dd>
-
-<dt>%t</dt>
-<dd>Temporary filename: testscript.test_filename.tmp, where
-test_filename is the name of the test file. All temporary files are
-placed in the Output directory within the directory the test is
-located.</dd>
-
-<dt>%prcontext</dt>
-<dd>Path to a script that performs grep -C. Use this since not all
-platforms support grep -C.</dd>
-
-<dt>%llvmgcc</dt> <dd>Full path to the llvm-gcc executable.</dd>
-<dt>%llvmgxx</dt> <dd>Full path to the llvm-g++ executable.</dd>
-</dl>
-
-<p>There are also several scripts in the llvm/test/Scripts directory
-that you might find useful when writing <tt>RUN</tt> lines.</p>
-
-<p>Lastly, you can easily mark a test that is expected to fail on a
-specific platform or with a specific version of llvmgcc by using the
- <tt>XFAIL</tt> keyword. Xfail lines are
-specified in the comments of the test program using <tt>XFAIL</tt>,
-followed by a colon, and one or more regular expressions (separated by
-a comma) that will match against the target triplet or llvmgcc version for the
-machine. You can use * to match all targets. You can specify the major or full
- version (i.e. 3.4) for llvmgcc. Here is an example of an
-<tt>XFAIL</tt> line:</p>
+</div>
+
+ <p>This will fail because the ' characters are passed to grep. This would
+ instruction grep to look for <tt>'find</tt> in the files <tt>this</tt> and
+ <tt>string'</tt>. To avoid this use curly braces to tell Tcl that it should
+ treat everything enclosed as one value. So our example would become:</p>
+
+<div class="doc_code">
<pre>
-; XFAIL: darwin,sun,llvmgcc4
+... | grep {find this string}
</pre>
+</div>
+
+ <p>Additionally, the characters <tt>[</tt> and <tt>]</tt> are treated
+ specially by Tcl. They tell Tcl to interpret the content as a command to
+ execute. Since these characters are often used in regular expressions this can
+ have disastrous results and cause the entire test run in a directory to fail.
+ For example, a common idiom is to look for some basicblock number:</p>
+<div class="doc_code">
+<pre>
+... | grep bb[2-8]
+</pre>
</div>
-<!--=========================================================================-->
-<div class="doc_section"><a name="progstructure"><tt>llvm-test</tt>
-Structure</a></div>
-<!--=========================================================================-->
+ <p>This, however, will cause Tcl to fail because its going to try to execute
+ a program named "2-8". Instead, what you want is this:</p>
-<div class="doc_text">
-
-<p>As mentioned previously, the <tt>llvm-test</tt> module provides three types
-of tests: MultiSource, SingleSource, and External. Each tree is then subdivided
-into several categories, including applications, benchmarks, regression tests,
-code that is strange grammatically, etc. These organizations should be
-relatively self explanatory.</p>
-
-<p>In addition to the regular "whole program" tests, the <tt>llvm-test</tt>
-module also provides a mechanism for compiling the programs in different ways.
-If the variable TEST is defined on the gmake command line, the test system will
-include a Makefile named <tt>TEST.<value of TEST variable>.Makefile</tt>.
-This Makefile can modify build rules to yield different results.</p>
-
-<p>For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt> to
-create the nightly test reports. To run the nightly tests, run <tt>gmake
-TEST=nightly</tt>.</p>
-
-<p>There are several TEST Makefiles available in the tree. Some of them are
-designed for internal LLVM research and will not work outside of the LLVM
-research group. They may still be valuable, however, as a guide to writing your
-own TEST Makefile for any optimization or analysis passes that you develop with
-LLVM.</p>
-
-<p>Note, when configuring the <tt>llvm-test</tt> module, you might want to
-specify the following configuration options:</p>
-<dl>
- <dt><i>--enable-spec2000</i>
- <dt><i>--enable-spec2000=<<tt>directory</tt>></i>
- <dd>
- Enable the use of SPEC2000 when testing LLVM. This is disabled by default
- (unless <tt>configure</tt> finds SPEC2000 installed). By specifying
- <tt>directory</tt>, you can tell configure where to find the SPEC2000
- benchmarks. If <tt>directory</tt> is left unspecified, <tt>configure</tt>
- uses the default value
- <tt>/home/vadve/shared/benchmarks/speccpu2000/benchspec</tt>.
- <p>
- <dt><i>--enable-spec95</i>
- <dt><i>--enable-spec95=<<tt>directory</tt>></i>
- <dd>
- Enable the use of SPEC95 when testing LLVM. It is similar to the
- <i>--enable-spec2000</i> option.
- <p>
- <dt><i>--enable-povray</i>
- <dt><i>--enable-povray=<<tt>directory</tt>></i>
- <dd>
- Enable the use of Povray as an external test. Versions of Povray written
- in C should work. This option is similar to the <i>--enable-spec2000</i>
- option.
-</dl>
+<div class="doc_code">
+<pre>
+... | grep {bb\[2-8\]}
+</pre>
</div>
-<!--=========================================================================-->
-<div class="doc_section"><a name="run">Running the LLVM Tests</a></div>
-<!--=========================================================================-->
+ <p>Finally, if you need to pass the <tt>\</tt> character down to a program,
+ then it must be doubled. This is another Tcl special character. So, suppose
+ you had:
-<div class="doc_text">
+<div class="doc_code">
+<pre>
+... | grep 'i32\*'
+</pre>
+</div>
-<p>First, all tests are executed within the LLVM object directory tree. They
-<i>are not</i> executed inside of the LLVM source tree. This is because the
-test suite creates temporary files during execution.</p>
+ <p>This will fail to match what you want (a pointer to i32). First, the
+ <tt>'</tt> do not get stripped off. Second, the <tt>\</tt> gets stripped off
+ by Tcl so what grep sees is: <tt>'i32*'</tt>. That's not likely to match
+ anything. To resolve this you must use <tt>\\</tt> and the <tt>{}</tt>, like
+ this:</p>
-<p>The master Makefile in llvm/test is capable of running only the DejaGNU
-driven tests. By default, it will run all of these tests.</p>
+<div class="doc_code">
+<pre>
+... | grep {i32\\*}
+</pre>
+</div>
-<p>To run only the DejaGNU driven tests, run <tt>gmake</tt> at the
-command line in <tt>llvm/test</tt>. To run a specific directory of tests, use
-the TESTSUITE variable.
-</p>
+<p>If your system includes GNU <tt>grep</tt>, make sure
+that <tt>GREP_OPTIONS</tt> is not set in your environment. Otherwise,
+you may get invalid results (both false positives and false
+negatives).</p>
-<p>For example, to run the Regression tests, type
-<tt>gmake TESTSUITE=Regression</tt> in <tt>llvm/tests</tt>.</p>
+</div>
-<p>Note that there are no Makefiles in <tt>llvm/test/Features</tt> and
-<tt>llvm/test/Regression</tt>. You must use DejaGNU from the <tt>llvm/test</tt>
-directory to run them.</p>
+<!-- _______________________________________________________________________ -->
+<h3><a name="FileCheck">The FileCheck utility</a></h3>
+<!-- _______________________________________________________________________ -->
-<p>To run the <tt>llvm-test</tt> suite, you need to use the following steps:
-</p>
-<ol>
- <li>cd into the llvm/projects directory</li>
- <li>check out the <tt>llvm-test</tt> module with:<br/>
- <tt>cvs -d :pserver:anon@llvm.org:/var/cvs/llvm co -PR llvm-test</tt><br>
- This will get the test suite into <tt>llvm/projects/llvm-test</tt></li>
- <li>configure the test suite. You can do this one of two ways:
- <ol>
- <li>Use the regular llvm configure:<br/>
- <tt>cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure</tt><br/>
- This will ensure that the <tt>projects/llvm-test</tt> directory is also
- properly configured.</li>
- <li>Use the <tt>configure</tt> script found in the <tt>llvm-test</tt> source
- directory:<br/>
- <tt>$LLVM_SRC_ROOT/projects/llvm-test/configure
- --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT</tt>
- </li>
- </ol>
- <li>gmake</li>
-</ol>
-<p>Note that the second and third steps only need to be done once. After you
-have the suite checked out and configured, you don't need to do it again (unless
-the test code or configure script changes).</p>
+<div>
+
+<p>A powerful feature of the RUN: lines is that it allows any arbitrary commands
+ to be executed as part of the test harness. While standard (portable) unix
+ tools like 'grep' work fine on run lines, as you see above, there are a lot
+ of caveats due to interaction with Tcl syntax, and we want to make sure the
+ run lines are portable to a wide range of systems. Another major problem is
+ that grep is not very good at checking to verify that the output of a tools
+ contains a series of different output in a specific order. The FileCheck
+ tool was designed to help with these problems.</p>
+
+<p>FileCheck (whose basic command line arguments are described in <a
+ href="http://llvm.org/cmds/FileCheck.html">the FileCheck man page</a> is
+ designed to read a file to check from standard input, and the set of things
+ to verify from a file specified as a command line argument. A simple example
+ of using FileCheck from a RUN line looks like this:</p>
+
+<div class="doc_code">
+<pre>
+; RUN: llvm-as < %s | llc -march=x86-64 | <b>FileCheck %s</b>
+</pre>
+</div>
-<p>To make a specialized test (use one of the
-<tt>llvm-test/TEST.<type>.Makefile</tt>s), just run:<br/>
-<tt>gmake TEST=<type> test</tt><br/>For example, you could run the
-nightly tester tests using the following commands:</p>
+<p>This syntax says to pipe the current file ("%s") into llvm-as, pipe that into
+llc, then pipe the output of llc into FileCheck. This means that FileCheck will
+be verifying its standard input (the llc output) against the filename argument
+specified (the original .ll file specified by "%s"). To see how this works,
+let's look at the rest of the .ll file (after the RUN line):</p>
+<div class="doc_code">
<pre>
- % cd llvm/projects/llvm-test
- % gmake TEST=nightly test
+define void @sub1(i32* %p, i32 %v) {
+entry:
+; <b>CHECK: sub1:</b>
+; <b>CHECK: subl</b>
+ %0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v)
+ ret void
+}
+
+define void @inc4(i64* %p) {
+entry:
+; <b>CHECK: inc4:</b>
+; <b>CHECK: incq</b>
+ %0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1)
+ ret void
+}
</pre>
+</div>
+
+<p>Here you can see some "CHECK:" lines specified in comments. Now you can see
+how the file is piped into llvm-as, then llc, and the machine code output is
+what we are verifying. FileCheck checks the machine code output to verify that
+it matches what the "CHECK:" lines specify.</p>
+
+<p>The syntax of the CHECK: lines is very simple: they are fixed strings that
+must occur in order. FileCheck defaults to ignoring horizontal whitespace
+differences (e.g. a space is allowed to match a tab) but otherwise, the contents
+of the CHECK: line is required to match some thing in the test file exactly.</p>
+
+<p>One nice thing about FileCheck (compared to grep) is that it allows merging
+test cases together into logical groups. For example, because the test above
+is checking for the "sub1:" and "inc4:" labels, it will not match unless there
+is a "subl" in between those labels. If it existed somewhere else in the file,
+that would not count: "grep subl" matches if subl exists anywhere in the
+file.</p>
+
+<!-- _______________________________________________________________________ -->
+<h4>
+ <a name="FileCheck-check-prefix">The FileCheck -check-prefix option</a>
+</h4>
-<p>Regardless of which test you're running, the results are printed on standard
-output and standard error. You can redirect these results to a file if you
-choose.</p>
+<div>
-<p>Some tests are known to fail. Some are bugs that we have not fixed yet;
-others are features that we haven't added yet (or may never add). In DejaGNU,
-the result for such tests will be XFAIL (eXpected FAILure). In this way, you
-can tell the difference between an expected and unexpected failure.</p>
+<p>The FileCheck -check-prefix option allows multiple test configurations to be
+driven from one .ll file. This is useful in many circumstances, for example,
+testing different architectural variants with llc. Here's a simple example:</p>
-<p>The tests in <tt>llvm-test</tt> have no such feature at this time. If the
-test passes, only warnings and other miscellaneous output will be generated. If
-a test fails, a large <program> FAILED message will be displayed. This
-will help you separate benign warnings from actual test failures.</p>
+<div class="doc_code">
+<pre>
+; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \
+; RUN: | <b>FileCheck %s -check-prefix=X32</b>
+; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \
+; RUN: | <b>FileCheck %s -check-prefix=X64</b>
+
+define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind {
+ %tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1
+ ret <4 x i32> %tmp1
+; <b>X32:</b> pinsrd_1:
+; <b>X32:</b> pinsrd $1, 4(%esp), %xmm0
+
+; <b>X64:</b> pinsrd_1:
+; <b>X64:</b> pinsrd $1, %edi, %xmm0
+}
+</pre>
+</div>
+
+<p>In this case, we're testing that we get the expected code generation with
+both 32-bit and 64-bit code generation.</p>
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection">
-<a name="customtest">Writing custom tests for llvm-test</a></div>
-<!-- _______________________________________________________________________ -->
+<h4>
+ <a name="FileCheck-CHECK-NEXT">The "CHECK-NEXT:" directive</a>
+</h4>
-<div class="doc_text">
+<div>
-<p>Assuming you can run llvm-test, (e.g. "<tt>gmake TEST=nightly report</tt>"
-should work), it is really easy to run optimizations or code generator
-components against every program in the tree, collecting statistics or running
-custom checks for correctness. At base, this is how the nightly tester works,
-it's just one example of a general framework.</p>
+<p>Sometimes you want to match lines and would like to verify that matches
+happen on exactly consecutive lines with no other lines in between them. In
+this case, you can use CHECK: and CHECK-NEXT: directives to specify this. If
+you specified a custom check prefix, just use "<PREFIX>-NEXT:". For
+example, something like this works as you'd expect:</p>
-<p>Lets say that you have an LLVM optimization pass, and you want to see how
-many times it triggers. First thing you should do is add an LLVM
-<a href="ProgrammersManual.html#Statistic">statistic</a> to your pass, which
-will tally counts of things you care about.</p>
+<div class="doc_code">
+<pre>
+define void @t2(<2 x double>* %r, <2 x double>* %A, double %B) {
+ %tmp3 = load <2 x double>* %A, align 16
+ %tmp7 = insertelement <2 x double> undef, double %B, i32 0
+ %tmp9 = shufflevector <2 x double> %tmp3,
+ <2 x double> %tmp7,
+ <2 x i32> < i32 0, i32 2 >
+ store <2 x double> %tmp9, <2 x double>* %r, align 16
+ ret void
+
+; <b>CHECK:</b> t2:
+; <b>CHECK:</b> movl 8(%esp), %eax
+; <b>CHECK-NEXT:</b> movapd (%eax), %xmm0
+; <b>CHECK-NEXT:</b> movhpd 12(%esp), %xmm0
+; <b>CHECK-NEXT:</b> movl 4(%esp), %eax
+; <b>CHECK-NEXT:</b> movapd %xmm0, (%eax)
+; <b>CHECK-NEXT:</b> ret
+}
+</pre>
+</div>
-<p>Following this, you can set up a test and a report that collects these and
-formats them for easy viewing. This consists of two files, an
-"<tt>llvm-test/TEST.XXX.Makefile</tt>" fragment (where XXX is the name of your
-test) and an "<tt>llvm-test/TEST.XXX.report</tt>" file that indicates how to
-format the output into a table. There are many example reports of various
-levels of sophistication included with llvm-test, and the framework is very
-general.</p>
+<p>CHECK-NEXT: directives reject the input unless there is exactly one newline
+between it an the previous directive. A CHECK-NEXT cannot be the first
+directive in a file.</p>
-<p>If you are interested in testing an optimization pass, check out the
-"libcalls" test as an example. It can be run like this:<p>
+</div>
+
+<!-- _______________________________________________________________________ -->
+<h4>
+ <a name="FileCheck-CHECK-NOT">The "CHECK-NOT:" directive</a>
+</h4>
+
+<div>
+
+<p>The CHECK-NOT: directive is used to verify that a string doesn't occur
+between two matches (or the first match and the beginning of the file). For
+example, to verify that a load is removed by a transformation, a test like this
+can be used:</p>
<div class="doc_code">
<pre>
-% cd llvm/projects/llvm-test/MultiSource/Benchmarks # or some other level
-% make TEST=libcalls report
+define i8 @coerce_offset0(i32 %V, i32* %P) {
+ store i32 %V, i32* %P
+
+ %P2 = bitcast i32* %P to i8*
+ %P3 = getelementptr i8* %P2, i32 2
+
+ %A = load i8* %P3
+ ret i8 %A
+; <b>CHECK:</b> @coerce_offset0
+; <b>CHECK-NOT:</b> load
+; <b>CHECK:</b> ret i8
+}
</pre>
</div>
-<p>This will do a bunch of stuff, then eventually print a table like this:</p>
+</div>
+
+<!-- _______________________________________________________________________ -->
+<h4>
+ <a name="FileCheck-Matching">FileCheck Pattern Matching Syntax</a>
+</h4>
+
+<div>
+
+<p>The CHECK: and CHECK-NOT: directives both take a pattern to match. For most
+uses of FileCheck, fixed string matching is perfectly sufficient. For some
+things, a more flexible form of matching is desired. To support this, FileCheck
+allows you to specify regular expressions in matching strings, surrounded by
+double braces: <b>{{yourregex}}</b>. Because we want to use fixed string
+matching for a majority of what we do, FileCheck has been designed to support
+mixing and matching fixed string matching with regular expressions. This allows
+you to write things like this:</p>
<div class="doc_code">
<pre>
-Name | total | #exit |
-...
-FreeBench/analyzer/analyzer | 51 | 6 |
-FreeBench/fourinarow/fourinarow | 1 | 1 |
-FreeBench/neural/neural | 19 | 9 |
-FreeBench/pifft/pifft | 5 | 3 |
-MallocBench/cfrac/cfrac | 1 | * |
-MallocBench/espresso/espresso | 52 | 12 |
-MallocBench/gs/gs | 4 | * |
-Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * |
-Prolangs-C/agrep/agrep | 33 | 12 |
-Prolangs-C/allroots/allroots | * | * |
-Prolangs-C/assembler/assembler | 47 | * |
-Prolangs-C/bison/mybison | 74 | * |
-...
+; CHECK: movhpd <b>{{[0-9]+}}</b>(%esp), <b>{{%xmm[0-7]}}</b>
</pre>
</div>
-<p>This basically is grepping the -stats output and displaying it in a table.
-You can also use the "TEST=libcalls report.html" target to get the table in HTML
-form, similarly for report.csv and report.tex.</p>
+<p>In this case, any offset from the ESP register will be allowed, and any xmm
+register will be allowed.</p>
-<p>The source for this is in llvm-test/TEST.libcalls.*. The format is pretty
-simple: the Makefile indicates how to run the test (in this case,
-"<tt>opt -simplify-libcalls -stats</tt>"), and the report contains one line for
-each column of the output. The first value is the header for the column and the
-second is the regex to grep the output of the command for. There are lots of
-example reports that can do fancy stuff.</p>
+<p>Because regular expressions are enclosed with double braces, they are
+visually distinct, and you don't need to use escape characters within the double
+braces like you would in C. In the rare case that you want to match double
+braces explicitly from the input, you can use something ugly like
+<b>{{[{][{]}}</b> as your pattern.</p>
</div>
+<!-- _______________________________________________________________________ -->
+<h4>
+ <a name="FileCheck-Variables">FileCheck Variables</a>
+</h4>
-<!--=========================================================================-->
-<div class="doc_section"><a name="nightly">Running the nightly tester</a></div>
-<!--=========================================================================-->
+<div>
-<div class="doc_text">
-
-<p>
-The <a href="http://llvm.org/nightlytest/">LLVM Nightly Testers</a>
-automatically check out an LLVM tree, build it, run the "nightly"
-program test (described above), run all of the feature and regression tests,
-delete the checked out tree, and then submit the results to
-<a href="http://llvm.org/nightlytest/">http://llvm.org/nightlytest/</a>.
-After test results are submitted to
-<a href="http://llvm.org/nightlytest/">http://llvm.org/nightlytest/</a>,
-they are processed and displayed on the tests page. An email to
-<a href="http://lists.cs.uiuc.edu/pipermail/llvm-testresults/">
-llvm-testresults@cs.uiuc.edu</a> summarizing the results is also generated.
-This testing scheme is designed to ensure that programs don't break as well
-as keep track of LLVM's progress over time.</p>
-
-<p>If you'd like to set up an instance of the nightly tester to run on your
-machine, take a look at the comments at the top of the
-<tt>utils/NewNightlyTest.pl</tt> file. If you decide to set up a nightly tester
-please choose a unique nickname and invoke <tt>utils/NewNightlyTest.pl</tt>
-with the "-nickname [yournickname]" command line option. We usually run it
-from a crontab entry that looks like this:</p>
+<p>It is often useful to match a pattern and then verify that it occurs again
+later in the file. For codegen tests, this can be useful to allow any register,
+but verify that that register is used consistently later. To do this, FileCheck
+allows named variables to be defined and substituted into patterns. Here is a
+simple example:</p>
<div class="doc_code">
<pre>
-5 3 * * * $HOME/llvm/utils/NewNightlyTest.pl -parallel -nickname Nickname \
- $CVSROOT $HOME/buildtest $HOME/cvs/testresults
+; CHECK: test5:
+; CHECK: notw <b>[[REGISTER:%[a-z]+]]</b>
+; CHECK: andw {{.*}}<b>[[REGISTER]]</b>
</pre>
</div>
-<p>Or, you can create a shell script to encapsulate the running of the script.
-The optimized x86 Linux nightly test is run from just such a script:</p>
+<p>The first check line matches a regex (<tt>%[a-z]+</tt>) and captures it into
+the variables "REGISTER". The second line verifies that whatever is in REGISTER
+occurs later in the file after an "andw". FileCheck variable references are
+always contained in <tt>[[ ]]</tt> pairs, are named, and their names can be
+formed with the regex "<tt>[a-zA-Z][a-zA-Z0-9]*</tt>". If a colon follows the
+name, then it is a definition of the variable, if not, it is a use.</p>
+
+<p>FileCheck variables can be defined multiple times, and uses always get the
+latest value. Note that variables are all read at the start of a "CHECK" line
+and are all defined at the end. This means that if you have something like
+"<tt>CHECK: [[XYZ:.*]]x[[XYZ]]</tt>" that the check line will read the previous
+value of the XYZ variable and define a new one after the match is performed. If
+you need to do something like this you can probably take advantage of the fact
+that FileCheck is not actually line-oriented when it matches, this allows you to
+define two separate CHECK lines that match on the same line.
+</p>
+
+</div>
+
+</div>
+
+<!-- _______________________________________________________________________ -->
+<h3><a name="rtvars">Variables and substitutions</a></h3>
+<!-- _______________________________________________________________________ -->
+<div>
+ <p>With a RUN line there are a number of substitutions that are permitted. In
+ general, any Tcl variable that is available in the <tt>substitute</tt>
+ function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line.
+ To make a substitution just write the variable's name preceded by a $.
+ Additionally, for compatibility reasons with previous versions of the test
+ library, certain names can be accessed with an alternate syntax: a % prefix.
+ These alternates are deprecated and may go away in a future version.
+ </p>
+ <p>Here are the available variable names. The alternate syntax is listed in
+ parentheses.</p>
+
+ <dl style="margin-left: 25px">
+ <dt><b>$test</b> (%s)</dt>
+ <dd>The full path to the test case's source. This is suitable for passing
+ on the command line as the input to an llvm tool.</dd>
+
+ <dt><b>$srcdir</b></dt>
+ <dd>The source directory from where the "<tt>make check</tt>" was run.</dd>
+
+ <dt><b>objdir</b></dt>
+ <dd>The object directory that corresponds to the <tt>$srcdir</tt>.</dd>
+
+ <dt><b>subdir</b></dt>
+ <dd>A partial path from the <tt>test</tt> directory that contains the
+ sub-directory that contains the test source being executed.</dd>
+
+ <dt><b>srcroot</b></dt>
+ <dd>The root directory of the LLVM src tree.</dd>
+
+ <dt><b>objroot</b></dt>
+ <dd>The root directory of the LLVM object tree. This could be the same
+ as the srcroot.</dd>
+
+ <dt><b>path</b><dt>
+ <dd>The path to the directory that contains the test case source. This is
+ for locating any supporting files that are not generated by the test, but
+ used by the test.</dd>
+
+ <dt><b>tmp</b></dt>
+ <dd>The path to a temporary file name that could be used for this test case.
+ The file name won't conflict with other test cases. You can append to it if
+ you need multiple temporaries. This is useful as the destination of some
+ redirected output.</dd>
+
+ <dt><b>target_triplet</b> (%target_triplet)</dt>
+ <dd>The target triplet that corresponds to the current host machine (the one
+ running the test cases). This should probably be called "host".<dd>
+
+ <dt><b>link</b> (%link)</dt>
+ <dd>This full link command used to link LLVM executables. This has all the
+ configured -I, -L and -l options.</dd>
+
+ <dt><b>shlibext</b> (%shlibext)</dt>
+ <dd>The suffix for the host platforms share library (dll) files. This
+ includes the period as the first character.</dd>
+ </dl>
+ <p>To add more variables, two things need to be changed. First, add a line in
+ the <tt>test/Makefile</tt> that creates the <tt>site.exp</tt> file. This will
+ "set" the variable as a global in the site.exp file. Second, in the
+ <tt>test/lib/llvm.exp</tt> file, in the substitute proc, add the variable name
+ to the list of "global" declarations at the beginning of the proc. That's it,
+ the variable can then be used in test scripts.</p>
+</div>
+
+<!-- _______________________________________________________________________ -->
+<h3><a name="rtfeatures">Other Features</a></h3>
+<!-- _______________________________________________________________________ -->
+<div>
+ <p>To make RUN line writing easier, there are several shell scripts located
+ in the <tt>llvm/test/Scripts</tt> directory. This directory is in the PATH
+ when running tests, so you can just call these scripts using their name. For
+ example:</p>
+ <dl>
+ <dt><b>ignore</b></dt>
+ <dd>This script runs its arguments and then always returns 0. This is useful
+ in cases where the test needs to cause a tool to generate an error (e.g. to
+ check the error output). However, any program in a pipeline that returns a
+ non-zero result will cause the test to fail. This script overcomes that
+ issue and nicely documents that the test case is purposefully ignoring the
+ result code of the tool</dd>
+
+ <dt><b>not</b></dt>
+ <dd>This script runs its arguments and then inverts the result code from
+ it. Zero result codes become 1. Non-zero result codes become 0. This is
+ useful to invert the result of a grep. For example "not grep X" means
+ succeed only if you don't find X in the input.</dd>
+ </dl>
+
+ <p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL.
+ You can easily mark a test as XFAIL just by including <tt>XFAIL: </tt> on a
+ line near the top of the file. This signals that the test case should succeed
+ if the test fails. Such test cases are counted separately by the testing tool. To
+ specify an expected fail, use the XFAIL keyword in the comments of the test
+ program followed by a colon and one or more regular expressions (separated by
+ a comma). The regular expressions allow you to XFAIL the test conditionally by
+ host platform. The regular expressions following the : are matched against the
+ target triplet for the host machine. If there is a match, the test is expected
+ to fail. If not, the test is expected to succeed. To XFAIL everywhere just
+ specify <tt>XFAIL: *</tt>. Here is an example of an <tt>XFAIL</tt> line:</p>
<div class="doc_code">
<pre>
-#!/bin/bash
-BASE=/proj/work/llvm/nightlytest
-export CVSROOT=:pserver:anon@llvm.org:/var/cvs/llvm
-export BUILDDIR=$BASE/build
-export WEBDIR=$BASE/testresults
-export LLVMGCCDIR=/proj/work/llvm/cfrontend/install
-export PATH=/proj/install/bin:$LLVMGCCDIR/bin:$PATH
-export LD_LIBRARY_PATH=/proj/install/lib
-cd $BASE
-cp /proj/work/llvm/llvm/utils/NewNightlyTest.pl .
-nice ./NewNightlyTest.pl -nice -release -verbose -parallel -enable-linscan \
- -nickname NightlyTester -noexternals 2>&1 > output.log
+; XFAIL: darwin,sun
</pre>
</div>
-<p>It is also possible to specify the the location your nightly test results
-are submitted. You can do this by passing the command line option
-"-submit-server [server_address]" and "-submit-script [script_on_server]" to
-<tt>utils/NewNightlyTest.pl</tt>. For example, to submit to the llvm.org
-nightly test results page, you would invoke the nightly test script with
-"-submit-server llvm.org -submit-script /nightlytest/NightlyTestAccept.cgi".
-If these options are not specified, the nightly test script sends the results
-to the llvm.org nightly test results page.</p>
+ <p>To make the output more useful, the <tt>llvm_runtest</tt> function wil
+ scan the lines of the test case for ones that contain a pattern that matches
+ PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that
+ is related to the test case. The number after "PR" specifies the LLVM bugzilla
+ number. When a PR number is specified, it will be used in the pass/fail
+ reporting. This is useful to quickly get some context when a test fails.</p>
+
+ <p>Finally, any line that contains "END." will cause the special
+ interpretation of lines to terminate. This is generally done right after the
+ last RUN: line. This has two side effects: (a) it prevents special
+ interpretation of lines that are part of the test program, not the
+ instructions to the test case, and (b) it speeds things up for really big test
+ cases by avoiding interpretation of the remainder of the file.</p>
+
+</div>
+
+</div>
+
+<!--=========================================================================-->
+<h2><a name="testsuiteoverview"><tt>test-suite</tt> Overview</a></h2>
+<!--=========================================================================-->
+
+<div>
+
+<p>The <tt>test-suite</tt> module contains a number of programs that can be
+compiled and executed. The <tt>test-suite</tt> includes reference outputs for
+all of the programs, so that the output of the executed program can be checked
+for correctness.</p>
+
+<p><tt>test-suite</tt> tests are divided into three types of tests: MultiSource,
+SingleSource, and External.</p>
+
+<ul>
+<li><tt>test-suite/SingleSource</tt>
+<p>The SingleSource directory contains test programs that are only a single
+source file in size. These are usually small benchmark programs or small
+programs that calculate a particular value. Several such programs are grouped
+together in each directory.</p></li>
+
+<li><tt>test-suite/MultiSource</tt>
+<p>The MultiSource directory contains subdirectories which contain entire
+programs with multiple source files. Large benchmarks and whole applications
+go here.</p></li>
+
+<li><tt>test-suite/External</tt>
+<p>The External directory contains Makefiles for building code that is external
+to (i.e., not distributed with) LLVM. The most prominent members of this
+directory are the SPEC 95 and SPEC 2000 benchmark suites. The <tt>External</tt>
+directory does not contain these actual tests, but only the Makefiles that know
+how to properly compile these programs from somewhere else. When
+using <tt>LNT</tt>, use the <tt>--test-externals</tt> option to include these
+tests in the results.</p></li>
+</ul>
+</div>
+
+<!--=========================================================================-->
+<h2><a name="testsuitequickstart"><tt>test-suite</tt> Quickstart</a></h2>
+<!--=========================================================================-->
+
+<div>
+<p>The modern way of running the <tt>test-suite</tt> is focused on testing and
+benchmarking complete compilers using
+the <a href="http://llvm.org/docs/lnt">LNT</a> testing infrastructure.</p>
+
+<p>For more information on using LNT to execute the <tt>test-suite</tt>, please
+see the <a href="http://llvm.org/docs/lnt/quickstart.html">LNT Quickstart</a>
+documentation.</p>
+</div>
+
+<!--=========================================================================-->
+<h2><a name="testsuitemakefiles"><tt>test-suite</tt> Makefiles</a></h2>
+<!--=========================================================================-->
-<p>Take a look at the <tt>NewNightlyTest.pl</tt> file to see what all of the
-flags and strings do. If you start running the nightly tests, please let us
-know. Thanks!</p>
+<div>
+<p>Historically, the <tt>test-suite</tt> was executed using a complicated setup
+of Makefiles. The LNT based approach above is recommended for most users, but
+there are some testing scenarios which are not supported by the LNT approach. In
+addition, LNT currently uses the Makefile setup under the covers and so
+developers who are interested in how LNT works under the hood may want to
+understand the Makefile based setup.</p>
+<p>For more information on the <tt>test-suite</tt> Makefile setup, please see
+the <a href="TestSuiteMakefileGuide.html">Test Suite Makefile Guide.</a></p>
</div>
<!-- *********************************************************************** -->
<hr>
<address>
<a href="http://jigsaw.w3.org/css-validator/check/referer"><img
- src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!"></a>
+ src="http://jigsaw.w3.org/css-validator/images/vcss-blue" alt="Valid CSS"></a>
<a href="http://validator.w3.org/check/referer"><img
- src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!" /></a>
+ src="http://www.w3.org/Icons/valid-html401-blue" alt="Valid HTML 4.01"></a>
- John T. Criswell, Reid Spencer, and Tanya Lattner<br>
- <a href="http://llvm.org">The LLVM Compiler Infrastructure</a><br/>
+ John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner<br>
+ <a href="http://llvm.org/">The LLVM Compiler Infrastructure</a><br>
Last modified: $Date$
</address>
</body>
projects / oota-llvm.git / blobdiff