<li><a href="#tree">LLVM Test Suite Tree</a></li>
<li><a href="#dgstructure">DejaGNU Structure</a></li>
<li><a href="#progstructure"><tt>llvm-test</tt> Structure</a></li>
- <li><a href="#run">Running the LLVM Tests</a></li>
+ <li><a href="#run">Running the LLVM Tests</a>
+ <ul>
+ <li><a href="#customtest">Writing custom tests for llvm-test</a></li>
+ </ul>
+ </li>
<li><a href="#nightly">Running the nightly tester</a></li>
</ol>
filenames themselves on the <tt>configure</tt> line.</li>
</ul></dd>
</dl>
+
+<p>Darwin (Mac OS X) developers can simplify the installation of Expect and tcl
+by using fink. <tt>fink install expect</tt> will install both. Alternatively,
+Darwinports users can use <tt>sudo port install expect</tt> to install Expect
+and tcl.</p>
+
</div>
<!--=========================================================================-->
programs in C and C++ is in the <tt>llvm-test</tt> module. This module should
be checked out to the <tt>llvm/projects</tt> directory. When you
<tt>configure</tt> the <tt>llvm</tt> module, the <tt>llvm-test</tt> module
-will be automatically configured. Alternatively, you can configure the <tt>llvm-test</tt> module manually.</p>
-<p>To run all of the simple tests in LLVM using DejaGNU, use the master Makefile in the
-<tt>llvm/test</tt> directory:</p>
+will be automatically configured. Alternatively, you can configure the
+ <tt>llvm-test</tt> module manually.</p>
+<p>To run all of the simple tests in LLVM using DejaGNU, use the master Makefile
+ in the <tt>llvm/test</tt> directory:</p>
<pre>
% gmake -C llvm/test
</pre>
<pre>
% gmake check
</pre>
-<p>To run only a subdirectory of tests in llvm/test using DejaGNU (ie. Regression/Transforms). Just substitute the path to the subdirectory:</p>
+
+<p>To run only a subdirectory of tests in llvm/test using DejaGNU (ie.
+Regression/Transforms), just set the TESTSUITE variable to the path of the
+subdirectory (relative to <tt>llvm/test</tt>):</p>
<pre>
% gmake -C llvm/test TESTSUITE=Regression/Transforms
</pre>
-<p><b>Note: If you are running the tests with <tt>objdir != subdir</tt> you must
-have run the complete testsuite before you can specify a subdirectory.</b></p>
+<p><b>Note: If you are running the tests with <tt>objdir != subdir</tt>, you
+must have run the complete testsuite before you can specify a
+subdirectory.</b></p>
<p>To run the comprehensive test suite (tests that compile and execute whole
programs), run the <tt>llvm-test</tt> tests:</p>
<p>The LLVM test suite contains two major categories of tests: code
fragments and whole programs. Code fragments are in the <tt>llvm</tt> module
-under the directory under the <tt>llvm/test</tt> directory. The whole programs
-test suite are in the <tt>llvm-test</tt> module under the main directory.</p>
+under the <tt>llvm/test</tt> directory. The whole programs
+test suite is in the <tt>llvm-test</tt> module under the main directory.</p>
</div>
-<div class="doc_subsection"><a name="codefragments">Code Fragments</a>
-</div>
+<!-- _______________________________________________________________________ -->
+<div class="doc_subsection"><a name="codefragments">Code Fragments</a></div>
+<!-- _______________________________________________________________________ -->
<div class="doc_text">
</div>
+<!-- _______________________________________________________________________ -->
<div class="doc_subsection"><a name="wholeprograms">Whole Programs</a></div>
+<!-- _______________________________________________________________________ -->
<div class="doc_text">
that you might find useful when writing <tt>RUN</tt> lines.</p>
<p>Lastly, you can easily mark a test that is expected to fail on a
-specific platform by using the <tt>XFAIL</tt> keyword. Xfail lines are
+specific platform or with a specific version of llvmgcc by using the
+ <tt>XFAIL</tt> keyword. Xfail lines are
specified in the comments of the test program using <tt>XFAIL</tt>,
followed by a colon, and one or more regular expressions (separated by
-a comma) that will match against the target triplet for the
-machine. You can use * to match all targets. Here is an example of an
+a comma) that will match against the target triplet or llvmgcc version for the
+machine. You can use * to match all targets. You can specify the major or full
+ version (i.e. 3.4) for llvmgcc. Here is an example of an
<tt>XFAIL</tt> line:</p>
<pre>
-; XFAIL: darwin,sun
+; XFAIL: darwin,sun,llvmgcc4
</pre>
</div>
<i>are not</i> executed inside of the LLVM source tree. This is because the
test suite creates temporary files during execution.</p>
-<p>The master Makefile in llvm/test is capable of running only the DejaGNU driven
-tests. By default, it will run all of these tests.</p>
+<p>The master Makefile in llvm/test is capable of running only the DejaGNU
+driven tests. By default, it will run all of these tests.</p>
<p>To run only the DejaGNU driven tests, run <tt>gmake</tt> at the
-command line in llvm/tests. To run a specific directory of tests, use the
-TESTSUITE variable.
+command line in <tt>llvm/test</tt>. To run a specific directory of tests, use
+the TESTSUITE variable.
</p>
<p>For example, to run the Regression tests, type
<ol>
<li>cd into the llvm/projects directory</li>
<li>check out the <tt>llvm-test</tt> module with:<br/>
- <tt>cvs -d :pserver:anon@llvm.cs.uiuc.edu:/var/cvs/llvm co -PR llvm-test</tt><br>
+ <tt>cvs -d :pserver:anon@llvm.org:/var/cvs/llvm co -PR llvm-test</tt><br>
This will get the test suite into <tt>llvm/projects/llvm-test</tt></li>
<li>configure the test suite. You can do this one of two ways:
<ol>
properly configured.</li>
<li>Use the <tt>configure</tt> script found in the <tt>llvm-test</tt> source
directory:<br/>
- <tt>$BUILD_SRC_DIR/configure --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT</tt>
+ <tt>$LLVM_SRC_ROOT/projects/llvm-test/configure
+ --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT</tt>
</li>
</ol>
<li>gmake</li>
the result for such tests will be XFAIL (eXpected FAILure). In this way, you
can tell the difference between an expected and unexpected failure.</p>
-<p>The tests in <tt>llvm-test</tt> have no such feature as of this time. If the
+<p>The tests in <tt>llvm-test</tt> have no such feature at this time. If the
test passes, only warnings and other miscellaneous output will be generated. If
a test fails, a large <program> FAILED message will be displayed. This
will help you separate benign warnings from actual test failures.</p>
</div>
+<!-- _______________________________________________________________________ -->
+<div class="doc_subsection">
+<a name="customtest">Writing custom tests for llvm-test</a></div>
+<!-- _______________________________________________________________________ -->
+
+<div class="doc_text">
+
+<p>Assuming you can run llvm-test, (e.g. "<tt>gmake TEST=nightly report</tt>"
+should work), it is really easy to run optimizations or code generator
+components against every program in the tree, collecting statistics or running
+custom checks for correctness. At base, this is how the nightly tester works,
+it's just one example of a general framework.</p>
+
+<p>Lets say that you have an LLVM optimization pass, and you want to see how
+many times it triggers. First thing you should do is add an LLVM
+<a href="ProgrammersManual.html#Statistic">statistic</a> to your pass, which
+will tally counts of things you care about.</p>
+
+<p>Following this, you can set up a test and a report that collects these and
+formats them for easy viewing. This consists of two files, an
+"<tt>llvm-test/TEST.XXX.Makefile</tt>" fragment (where XXX is the name of your
+test) and an "<tt>llvm-test/TEST.XXX.report</tt>" file that indicates how to
+format the output into a table. There are many example reports of various
+levels of sophistication included with llvm-test, and the framework is very
+general.</p>
+
+<p>If you are interested in testing an optimization pass, check out the
+"libcalls" test as an example. It can be run like this:<p>
+
+<div class="doc_code">
+<pre>
+% cd llvm/projects/llvm-test/MultiSource/Benchmarks # or some other level
+% make TEST=libcalls report
+</pre>
+</div>
+
+<p>This will do a bunch of stuff, then eventually print a table like this:</p>
+
+<div class="doc_code">
+<pre>
+Name | total | #exit |
+...
+FreeBench/analyzer/analyzer | 51 | 6 |
+FreeBench/fourinarow/fourinarow | 1 | 1 |
+FreeBench/neural/neural | 19 | 9 |
+FreeBench/pifft/pifft | 5 | 3 |
+MallocBench/cfrac/cfrac | 1 | * |
+MallocBench/espresso/espresso | 52 | 12 |
+MallocBench/gs/gs | 4 | * |
+Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * |
+Prolangs-C/agrep/agrep | 33 | 12 |
+Prolangs-C/allroots/allroots | * | * |
+Prolangs-C/assembler/assembler | 47 | * |
+Prolangs-C/bison/mybison | 74 | * |
+...
+</pre>
+</div>
+
+<p>This basically is grepping the -stats output and displaying it in a table.
+You can also use the "TEST=libcalls report.html" target to get the table in HTML
+form, similarly for report.csv and report.tex.</p>
+
+<p>The source for this is in llvm-test/TEST.libcalls.*. The format is pretty
+simple: the Makefile indicates how to run the test (in this case,
+"<tt>opt -simplify-libcalls -stats</tt>"), and the report contains one line for
+each column of the output. The first value is the header for the column and the
+second is the regex to grep the output of the command for. There are lots of
+example reports that can do fancy stuff.</p>
+
+</div>
+
+
<!--=========================================================================-->
<div class="doc_section"><a name="nightly">Running the nightly tester</a></div>
<!--=========================================================================-->
<div class="doc_text">
<p>
-The <a href="http://llvm.cs.uiuc.edu/testresults/">LLVM Nightly Testers</a>
+The <a href="http://llvm.org/nightlytest/">LLVM Nightly Testers</a>
automatically check out an LLVM tree, build it, run the "nightly"
-program test (described above), run all of the feature and regression tests,
-and then delete the checked out tree. This tester is designed to ensure that
-programs don't break as well as keep track of LLVM's progress over time.</p>
-
-<p>If you'd like to set up an instance of the nightly tester to run on your
-machine, take a look at the comments at the top of the
-<tt>utils/NightlyTester.pl</tt> file. We usually run it from a crontab entry
-that looks ilke this:</p>
+program test (described above), run all of the feature and regression tests,
+delete the checked out tree, and then submit the results to
+<a href="http://llvm.org/nightlytest/">http://llvm.org/nightlytest/</a>.
+After test results are submitted to
+<a href="http://llvm.org/nightlytest/">http://llvm.org/nightlytest/</a>,
+they are processed and displayed on the tests page. An email to
+<a href="http://lists.cs.uiuc.edu/pipermail/llvm-testresults/">
+llvm-testresults@cs.uiuc.edu</a> summarizing the results is also generated.
+This testing scheme is designed to ensure that programs don't break as well
+as keep track of LLVM's progress over time.</p>
+
+<p>If you'd like to set up an instance of the nightly tester to run on your
+machine, take a look at the comments at the top of the
+<tt>utils/NewNightlyTest.pl</tt> file. If you decide to set up a nightly tester
+please choose a unique nickname and invoke <tt>utils/NewNightlyTest.pl</tt>
+with the "-nickname [yournickname]" command line option. We usually run it
+from a crontab entry that looks like this:</p>
<div class="doc_code">
<pre>
-5 3 * * * $HOME/llvm/utils/NightlyTest.pl -parallel $CVSROOT $HOME/buildtest-X86 $HOME/cvs/testresults-X86
+5 3 * * * $HOME/llvm/utils/NewNightlyTest.pl -parallel -nickname Nickname \
+ $CVSROOT $HOME/buildtest $HOME/cvs/testresults
</pre>
</div>
<pre>
#!/bin/bash
BASE=/proj/work/llvm/nightlytest
-export CVSROOT=:pserver:anon@llvm.cs.uiuc.edu:/var/cvs/llvm
+export CVSROOT=:pserver:anon@llvm.org:/var/cvs/llvm
export BUILDDIR=$BASE/build
export WEBDIR=$BASE/testresults
export LLVMGCCDIR=/proj/work/llvm/cfrontend/install
export PATH=/proj/install/bin:$LLVMGCCDIR/bin:$PATH
export LD_LIBRARY_PATH=/proj/install/lib
cd $BASE
-cp /proj/work/llvm/llvm/utils/NightlyTest.pl .
-nice ./NightlyTest.pl -nice -release -verbose -parallel -enable-linscan -noexternals 2>&1 > output.log
-mail -s 'X86 nightly tester results' <a href="http://mail.cs.uiuc.edu/mailman/listinfo/llvm-testresults">llvm-testresults@cs.uiuc.edu</a> < output.log
+cp /proj/work/llvm/llvm/utils/NewNightlyTest.pl .
+nice ./NewNightlyTest.pl -nice -release -verbose -parallel -enable-linscan \
+ -nickname NightlyTester -noexternals 2>&1 > output.log
</pre>
</div>
-<p>Take a look at the <tt>NightlyTest.pl</tt> file to see what all of the flags
-and strings do. If you start running the nightly tests, please let us know and
-we'll link your page to the global tester page. Thanks!</p>
+<p>It is also possible to specify the the location your nightly test results
+are submitted. You can do this by passing the command line option
+"-submit-server [server_address]" and "-submit-script [script_on_server]" to
+<tt>utils/NewNightlyTest.pl</tt>. For example, to submit to the llvm.org
+nightly test results page, you would invoke the nightly test script with
+"-submit-server llvm.org -submit-script /nightlytest/NightlyTestAccept.cgi".
+If these options are not specified, the nightly test script sends the results
+to the llvm.org nightly test results page.</p>
+
+<p>Take a look at the <tt>NewNightlyTest.pl</tt> file to see what all of the
+flags and strings do. If you start running the nightly tests, please let us
+know. Thanks!</p>
</div>
<a href="http://validator.w3.org/check/referer"><img
src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!" /></a>
- John T. Criswell<br>
- <a href="http://llvm.cs.uiuc.edu">The LLVM Compiler Infrastructure</a><br/>
+ John T. Criswell, Reid Spencer, and Tanya Lattner<br>
+ <a href="http://llvm.org">The LLVM Compiler Infrastructure</a><br/>
Last modified: $Date$
</address>
</body>
projects / oota-llvm.git / blobdiff