"http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
+ <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>LLVM Testing Infrastructure Guide</title>
<link rel="stylesheet" href="llvm.css" type="text/css">
</head>
<body>
-<div class="doc_title">
+<h1>
LLVM Testing Infrastructure Guide
-</div>
+</h1>
<ol>
<li><a href="#overview">Overview</a></li>
<ul>
<li><a href="#regressiontests">Regression tests</a></li>
<li><a href="#testsuite">Test suite</a></li>
+ <li><a href="#debuginfotests">Debugging Information tests</a></li>
</ul>
</li>
<li><a href="#quick">Quick start</a>
<ul>
<li><a href="#quickregressiontests">Regression tests</a></li>
<li><a href="#quicktestsuite">Test suite</a></li>
+ <li><a href="#quickdebuginfotests">Debugging Information tests</a></li>
</ul>
</li>
<li><a href="#rtstructure">Regression test structure</a>
<li><a href="#testsuiteexternal">Configuring External Tests</a></li>
<li><a href="#testsuitetests">Running different tests</a></li>
<li><a href="#testsuiteoutput">Generating test output</a></li>
- <li><a href="#testsuitecustom">Writing custom tests for llvm-test</a></li>
+ <li><a href="#testsuitecustom">Writing custom tests for test-suite</a></li>
</ul>
</li>
</ol>
</div>
<!--=========================================================================-->
-<div class="doc_section"><a name="overview">Overview</a></div>
+<h2><a name="overview">Overview</a></h2>
<!--=========================================================================-->
-<div class="doc_text">
+<div>
<p>This document is the reference manual for the LLVM testing infrastructure. It
documents the structure of the LLVM testing infrastructure, the tools needed to
</div>
<!--=========================================================================-->
-<div class="doc_section"><a name="requirements">Requirements</a></div>
+<h2><a name="requirements">Requirements</a></h2>
<!--=========================================================================-->
-<div class="doc_text">
+<div>
<p>In order to use the LLVM testing infrastructure, you will need all of the
software required to build LLVM, as well
</div>
<!--=========================================================================-->
-<div class="doc_section"><a name="org">LLVM testing infrastructure organization</a></div>
+<h2><a name="org">LLVM testing infrastructure organization</a></h2>
<!--=========================================================================-->
-<div class="doc_text">
+<div>
<p>The LLVM testing infrastructure contains two major categories of tests:
regression tests and whole programs. The regression tests are contained inside
in subversion.
</p>
-</div>
-
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection"><a name="regressiontests">Regression tests</a></div>
+<h3><a name="regressiontests">Regression tests</a></h3>
<!-- _______________________________________________________________________ -->
-<div class="doc_text">
+<div>
<p>The regression tests are small pieces of code that test a specific feature of
LLVM or trigger a specific bug in LLVM. They are usually written in LLVM
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection"><a name="testsuite">Test suite</a></div>
+<h3><a name="testsuite">Test suite</a></h3>
<!-- _______________________________________________________________________ -->
-<div class="doc_text">
+<div>
<p>The test suite contains whole programs, which are pieces of
code which can be compiled and linked into a stand-alone program that can be
</div>
+<!-- _______________________________________________________________________ -->
+<h3><a name="debuginfotests">Debugging Information tests</a></h3>
+<!-- _______________________________________________________________________ -->
+
+<div>
+
+<p>The test suite contains tests to check quality of debugging information.
+The test are written in C based languages or in LLVM assembly language. </p>
+
+<p>These tests are compiled and run under a debugger. The debugger output
+is checked to validate of debugging information. See README.txt in the
+test suite for more information . This test suite is located in the
+<tt>debuginfo-tests</tt> Subversion module. </p>
+
+</div>
+
+</div>
+
<!--=========================================================================-->
-<div class="doc_section"><a name="quick">Quick start</a></div>
+<h2><a name="quick">Quick start</a></h2>
<!--=========================================================================-->
-<div class="doc_text">
+<div>
<p>The tests are located in two separate Subversion modules. The regressions
tests are in the main "llvm" module under the directory
The more comprehensive test suite that includes whole
programs in C and C++ is in the <tt>test-suite</tt> module. This module should
be checked out to the <tt>llvm/projects</tt> directory (don't use another name
-then the default "test-suite", for then the test suite will be run every time
+than the default "test-suite", for then the test suite will be run every time
you run <tt>make</tt> in the main <tt>llvm</tt> directory).
When you <tt>configure</tt> the <tt>llvm</tt> module,
the <tt>test-suite</tt> directory will be automatically configured.
Alternatively, you can configure the <tt>test-suite</tt> module manually.</p>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection"><a name="quickregressiontests">Regression tests</a></div>
+<h3><a name="quickregressiontests">Regression tests</a></h3>
+<div>
<!-- _______________________________________________________________________ -->
<p>To run all of the LLVM regression tests, use master Makefile in
the <tt>llvm/test</tt> directory:</p>
</pre>
</div>
-<p>If you have <a href="http://clang.llvm.org">Clang</a> checked out and built,
+<p>If you have <a href="http://clang.llvm.org/">Clang</a> checked out and built,
you can run the LLVM and Clang tests simultaneously using:</p>
<p>or</p>
<p>For more information on using the 'lit' tool, see 'llvm-lit --help' or the
'lit' man page.</p>
+</div>
+
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection"><a name="quicktestsuite">Test suite</a></div>
+<h3><a name="quicktestsuite">Test suite</a></h3>
<!-- _______________________________________________________________________ -->
+<div>
+
<p>To run the comprehensive test suite (tests that compile and execute whole
programs), first checkout and setup the <tt>test-suite</tt> module:</p>
<div class="doc_code">
<pre>
-% cd llvm/projects
+% cd ~/llvm/projects
% svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
% cd ..
-% ./configure --with-llvmgccdir=$LLVM_GCC_DIR
</pre>
</div>
-<p>where <tt>$LLVM_GCC_DIR</tt> is the directory where
-you <em>installed</em> llvm-gcc, not it's src or obj
-dir. The <tt>--with-llvmgccdir</tt> option assumes that
-the <tt>llvm-gcc-4.2</tt> module was configured with
-<tt>--program-prefix=llvm-</tt>, and therefore that the C and C++
-compiler drivers are called <tt>llvm-gcc</tt> and <tt>llvm-g++</tt>
-respectively. If this is not the case,
-use <tt>--with-llvmgcc</tt>/<tt>--with-llvmgxx</tt> to specify each
-executable's location.</p>
+<p>and then configure and build normally as you would from the
+<a href="http://llvm.org/docs/GettingStarted.html#quickstart">Getting Started
+Guide</a>. This will autodetect first the built clang if you are building
+clang, then <tt>clang</tt> in your path and finally look for <tt>llvm-gcc</tt>
+in your path.
<p>Then, run the entire test suite by running make in the <tt>test-suite</tt>
-directory:</p>
+subdirectory of your build directory:</p>
<div class="doc_code">
<pre>
-% cd projects/test-suite
+% cd <i>where-you-built-llvm</i>/projects/test-suite
% gmake
</pre>
</div>
-<p>Usually, running the "nightly" set of tests is a good idea, and you can also
+<p>Usually, running the "simple" set of tests is a good idea, and you can also
let it generate a report by running:</p>
<div class="doc_code">
<pre>
-% cd projects/test-suite
-% gmake TEST=nightly report report.html
+% cd <i>where-you-built-llvm</i>/projects/test-suite
+% gmake TEST=simple report report.html
</pre>
</div>
</div>
+<!-- _______________________________________________________________________ -->
+<h3><a name="quickdebuginfotests">Debugging Information tests</a></h3>
+<div>
+<!-- _______________________________________________________________________ -->
+<div>
+
+<p> To run debugging information tests simply checkout the tests inside
+clang/test directory. </p>
+
+<div class="doc_code">
+<pre>
+%cd clang/test
+% svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
+</pre>
+</div>
+
+<p> These tests are already set up to run as part of clang regression tests.</p>
+
+</div>
+
+</div>
+
+</div>
+
<!--=========================================================================-->
-<div class="doc_section"><a name="rtstructure">Regression test structure</a></div>
+<h2><a name="rtstructure">Regression test structure</a></h2>
<!--=========================================================================-->
-<div class="doc_text">
+<div>
<p>The LLVM regression tests are driven by 'lit' and are located in
the <tt>llvm/test</tt> directory.
<li><tt>Verifier</tt>: tests the IR verifier.</li>
</ul>
-</div>
-
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection"><a name="rtcustom">Writing new regression tests</a></div>
+<h3><a name="rtcustom">Writing new regression tests</a></h3>
<!-- _______________________________________________________________________ -->
-<div class="doc_text">
+<div>
<p>The regression test structure is very simple, but does require some
information to be set. This information is gathered via <tt>configure</tt> and
is written to a file, <tt>lit.site.cfg</tt>
obtained by using Tcl's glob command. Any directory that contains only
directories does not need the <tt>dg.exp</tt> file.</p>
- <p>The <tt>llvm-runtests</tt> function lookas at each file that is passed to
- it and gathers any lines together that match "RUN:". This are the "RUN" lines
+ <p>The <tt>llvm-runtests</tt> function looks at each file that is passed to
+ it and gathers any lines together that match "RUN:". These are the "RUN" lines
that specify how the test is to be run. So, each test script must contain
RUN lines if it is to do anything. If there are no RUN lines, the
<tt>llvm-runtests</tt> function will issue an error and the test will
shell. Consequently the syntax differs from normal shell script syntax in a
few ways. You can specify as many RUN lines as needed.</p>
+ <p>lit performs substitution on each RUN line to replace LLVM tool
+ names with the full paths to the executable built for each tool (in
+ $(LLVM_OBJ_ROOT)/$(BuildMode)/bin). This ensures that lit does not
+ invoke any stray LLVM tools in the user's path during testing.</p>
+
<p>Each RUN line is executed on its own, distinct from other lines unless
its last character is <tt>\</tt>. This continuation character causes the RUN
line to be concatenated with the next one. In this way you can build up long
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection"><a name="FileCheck">The FileCheck utility</a></div>
+<h3><a name="FileCheck">The FileCheck utility</a></h3>
<!-- _______________________________________________________________________ -->
-<div class="doc_text">
+<div>
<p>A powerful feature of the RUN: lines is that it allows any arbitrary commands
to be executed as part of the test harness. While standard (portable) unix
that would not count: "grep subl" matches if subl exists anywhere in the
file.</p>
-</div>
-
<!-- _______________________________________________________________________ -->
-<div class="doc_subsubsection"><a
-name="FileCheck-check-prefix">The FileCheck -check-prefix option</a></div>
+<h4>
+ <a name="FileCheck-check-prefix">The FileCheck -check-prefix option</a>
+</h4>
-<div class="doc_text">
+<div>
<p>The FileCheck -check-prefix option allows multiple test configurations to be
driven from one .ll file. This is useful in many circumstances, for example,
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsubsection"><a
-name="FileCheck-CHECK-NEXT">The "CHECK-NEXT:" directive</a></div>
+<h4>
+ <a name="FileCheck-CHECK-NEXT">The "CHECK-NEXT:" directive</a>
+</h4>
-<div class="doc_text">
+<div>
<p>Sometimes you want to match lines and would like to verify that matches
-happen on exactly consequtive lines with no other lines in between them. In
+happen on exactly consecutive lines with no other lines in between them. In
this case, you can use CHECK: and CHECK-NEXT: directives to specify this. If
you specified a custom check prefix, just use "<PREFIX>-NEXT:". For
example, something like this works as you'd expect:</p>
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsubsection"><a
-name="FileCheck-CHECK-NOT">The "CHECK-NOT:" directive</a></div>
+<h4>
+ <a name="FileCheck-CHECK-NOT">The "CHECK-NOT:" directive</a>
+</h4>
-<div class="doc_text">
+<div>
<p>The CHECK-NOT: directive is used to verify that a string doesn't occur
between two matches (or the first match and the beginning of the file). For
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsubsection"><a
-name="FileCheck-Matching">FileCheck Pattern Matching Syntax</a></div>
+<h4>
+ <a name="FileCheck-Matching">FileCheck Pattern Matching Syntax</a>
+</h4>
-<div class="doc_text">
+<div>
<p>The CHECK: and CHECK-NOT: directives both take a pattern to match. For most
uses of FileCheck, fixed string matching is perfectly sufficient. For some
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsubsection"><a
-name="FileCheck-Variables">FileCheck Variables</a></div>
+<h4>
+ <a name="FileCheck-Variables">FileCheck Variables</a>
+</h4>
-<div class="doc_text">
+<div>
<p>It is often useful to match a pattern and then verify that it occurs again
later in the file. For codegen tests, this can be useful to allow any register,
</div>
+</div>
+
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection"><a name="rtvars">Variables and
-substitutions</a></div>
+<h3><a name="rtvars">Variables and substitutions</a></h3>
<!-- _______________________________________________________________________ -->
-<div class="doc_text">
+<div>
<p>With a RUN line there are a number of substitutions that are permitted. In
general, any Tcl variable that is available in the <tt>substitute</tt>
function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line.
you need multiple temporaries. This is useful as the destination of some
redirected output.</dd>
- <dt><b>llvmlibsdir</b> (%llvmlibsdir)</dt>
- <dd>The directory where the LLVM libraries are located.</dd>
-
<dt><b>target_triplet</b> (%target_triplet)</dt>
<dd>The target triplet that corresponds to the current host machine (the one
running the test cases). This should probably be called "host".<dd>
- <dt><b>llvmgcc</b> (%llvmgcc)</dt>
- <dd>The full path to the <tt>llvm-gcc</tt> executable as specified in the
- configured LLVM environment</dd>
-
- <dt><b>llvmgxx</b> (%llvmgxx)</dt>
- <dd>The full path to the <tt>llvm-gxx</tt> executable as specified in the
- configured LLVM environment</dd>
-
- <dt><b>gccpath</b></dt>
- <dd>The full path to the C compiler used to <i>build </i> LLVM. Note that
- this might not be gcc.</dd>
-
- <dt><b>gxxpath</b></dt>
- <dd>The full path to the C++ compiler used to <i>build </i> LLVM. Note that
- this might not be g++.</dd>
-
- <dt><b>compile_c</b> (%compile_c)</dt>
- <dd>The full command line used to compile LLVM C source code. This has all
- the configured -I, -D and optimization options.</dd>
-
- <dt><b>compile_cxx</b> (%compile_cxx)</dt>
- <dd>The full command used to compile LLVM C++ source code. This has
- all the configured -I, -D and optimization options.</dd>
-
<dt><b>link</b> (%link)</dt>
<dd>This full link command used to link LLVM executables. This has all the
configured -I, -L and -l options.</dd>
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection"><a name="rtfeatures">Other Features</a></div>
+<h3><a name="rtfeatures">Other Features</a></h3>
<!-- _______________________________________________________________________ -->
-<div class="doc_text">
+<div>
<p>To make RUN line writing easier, there are several shell scripts located
in the <tt>llvm/test/Scripts</tt> directory. This directory is in the PATH
when running tests, so you can just call these scripts using their name. For
</div>
+</div>
+
<!--=========================================================================-->
-<div class="doc_section"><a name="testsuitestructure">Test suite
-Structure</a></div>
+<h2><a name="testsuitestructure">Test suite Structure</a></h2>
<!--=========================================================================-->
-<div class="doc_text">
+<div>
<p>The <tt>test-suite</tt> module contains a number of programs that can be compiled
with LLVM and executed. These programs are compiled using the native compiler
test using the <tt>TEST</tt> variable to change what tests or run on the
selected programs (see below for more info).</p>
-<p>In addition for testing correctness, the <tt>llvm-test</tt> directory also
+<p>In addition for testing correctness, the <tt>test-suite</tt> directory also
performs timing tests of various LLVM optimizations. It also records
compilation times for the compilers and the JIT. This information can be
used to compare the effectiveness of LLVM's optimizations and code
generation.</p>
-<p><tt>llvm-test</tt> tests are divided into three types of tests: MultiSource,
+<p><tt>test-suite</tt> tests are divided into three types of tests: MultiSource,
SingleSource, and External.</p>
<ul>
-<li><tt>llvm-test/SingleSource</tt>
+<li><tt>test-suite/SingleSource</tt>
<p>The SingleSource directory contains test programs that are only a single
source file in size. These are usually small benchmark programs or small
programs that calculate a particular value. Several such programs are grouped
together in each directory.</p></li>
-<li><tt>llvm-test/MultiSource</tt>
+<li><tt>test-suite/MultiSource</tt>
<p>The MultiSource directory contains subdirectories which contain entire
programs with multiple source files. Large benchmarks and whole applications
go here.</p></li>
-<li><tt>llvm-test/External</tt>
+<li><tt>test-suite/External</tt>
<p>The External directory contains Makefiles for building code that is external
to (i.e., not distributed with) LLVM. The most prominent members of this
directory are the SPEC 95 and SPEC 2000 benchmark suites. The <tt>External</tt>
directory does not contain these actual tests, but only the Makefiles that know
how to properly compile these programs from somewhere else. The presence and
-location of these external programs is configured by the llvm-test
+location of these external programs is configured by the test-suite
<tt>configure</tt> script.</p></li>
</ul>
</div>
<!--=========================================================================-->
-<div class="doc_section"><a name="testsuiterun">Running the test suite</a></div>
+<h2><a name="testsuiterun">Running the test suite</a></h2>
<!--=========================================================================-->
-<div class="doc_text">
+<div>
<p>First, all tests are executed within the LLVM object directory tree. They
<i>are not</i> executed inside of the LLVM source tree. This is because the
have the suite checked out and configured, you don't need to do it again (unless
the test code or configure script changes).</p>
-</div>
-
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection">
-<a name="testsuiteexternal">Configuring External Tests</a></div>
+<h3>
+ <a name="testsuiteexternal">Configuring External Tests</a>
+</h3>
<!-- _______________________________________________________________________ -->
-<div class="doc_text">
+<div>
<p>In order to run the External tests in the <tt>test-suite</tt>
module, you must specify <i>--with-externals</i>. This
must be done during the <em>re-configuration</em> step (see above),
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection">
-<a name="testsuitetests">Running different tests</a></div>
+<h3>
+ <a name="testsuitetests">Running different tests</a>
+</h3>
<!-- _______________________________________________________________________ -->
-<div class="doc_text">
+<div>
<p>In addition to the regular "whole program" tests, the <tt>test-suite</tt>
module also provides a mechanism for compiling the programs in different ways.
If the variable TEST is defined on the <tt>gmake</tt> command line, the test system will
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection">
-<a name="testsuiteoutput">Generating test output</a></div>
+<h3>
+ <a name="testsuiteoutput">Generating test output</a>
+</h3>
<!-- _______________________________________________________________________ -->
-<div class="doc_text">
+<div>
<p>There are a number of ways to run the tests and generate output. The most
simple one is simply running <tt>gmake</tt> with no arguments. This will
compile and run all programs in the tree using a number of different methods
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection">
-<a name="testsuitecustom">Writing custom tests for the test suite</a></div>
+<h3>
+ <a name="testsuitecustom">Writing custom tests for the test suite</a>
+</h3>
<!-- _______________________________________________________________________ -->
-<div class="doc_text">
+<div>
<p>Assuming you can run the test suite, (e.g. "<tt>gmake TEST=nightly report</tt>"
should work), it is really easy to run optimizations or code generator
will tally counts of things you care about.</p>
<p>Following this, you can set up a test and a report that collects these and
-formats them for easy viewing. This consists of two files, an
+formats them for easy viewing. This consists of two files, a
"<tt>test-suite/TEST.XXX.Makefile</tt>" fragment (where XXX is the name of your
-test) and an "<tt>llvm-test/TEST.XXX.report</tt>" file that indicates how to
+test) and a "<tt>test-suite/TEST.XXX.report</tt>" file that indicates how to
format the output into a table. There are many example reports of various
levels of sophistication included with the test suite, and the framework is very
general.</p>
</div>
+</div>
+
<!-- *********************************************************************** -->
<hr>
src="http://www.w3.org/Icons/valid-html401-blue" alt="Valid HTML 4.01"></a>
John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner<br>
- <a href="http://llvm.org">The LLVM Compiler Infrastructure</a><br>
+ <a href="http://llvm.org/">The LLVM Compiler Infrastructure</a><br>
Last modified: $Date$
</address>
</body>