- <ul>
- <li>Features
- <p>
- This directory contains sample codes that test various features
- of the LLVM language. These pieces of sample code are run
- through various assembler, disassembler, and optimizer passes.
- </p>
-
- <li>Regression
- <p>
- This directory contains regression tests for LLVM. When a bug
- is found in LLVM, a regression test containing just enough
- code to reproduce the problem should be written and placed
- somewhere underneath this directory. In most cases, this
- will be a small piece of LLVM assembly language code, often
- distilled from an actual application or benchmark.
- </p>
-
- <li>Programs
- <p>
- The Programs directory contains programs that can be compiled
- with LLVM and executed. These programs are compiled using the
- native compiler and various LLVM backends. The output from the
- program compiled with the native compiler is assumed correct;
- the results from the other programs are compared to the native
- program output and pass if they match.
- </p><p>
- In addition for testing correctness, the Programs directory
- also performs timing tests of various LLVM optimizations.
- It also records compilation times for the compilers and the
- JIT. This information can be used to compare the
- effectiveness of LLVM's optimizations and code generation.
- </p><p>
- The Programs directory is subdivided into several smaller
- subdirectories:
- </p>
-
- <ul>
- <li>Programs/SingleSource
- <p>
- The SingleSource directory contains test programs that
- are only a single source file in size. These are
- usually small benchmark programs or small programs that
- calculate a particular value. Several such programs are
- grouped together in each directory.
- </p>
-
- <li>Programs/MultiSource
- <p>
- The MultiSource directory contains subdirectories which
- contain entire programs with multiple source files.
- Large benchmarks and whole applications go here.
- </p>
-
- <li>Programs/External
- <p>
- The External directory contains Makefiles for building
- code that is external to (i.e. not distributed with)
- LLVM. The most prominent member of this directory is
- the SPEC 2000 benchmark suite. The presence and
- location of these external programs is configured by the
- LLVM <tt>configure</tt> script.
- </p>
- </ul>
-
- <p>
-
- <li>QMTest
- <p>
- This directory contains the QMTest information files. Inside
- this directory are QMTest administration files and the Python
- code that implements the LLVM test and database classes.
- </p>
- </ul>
- </div>
-
- <!--===============================================================-->
- <div class="doc_section"><h2><a name="qmstructure">QMTest Structure</a>
- </div>
- <!--===============================================================-->
-
- <div class="doc_text">
- <p>
- The LLVM test suite is partially driven by QMTest and partially
- driven by GNU Make. Specifically, the Features and Regression tests
- are all driven by QMTest. The Programs directory is currently
- driven by a set of Makefiles.
- </p><p>
- The QMTest system needs to have several pieces of information
- available; these pieces of configuration information are known
- collectively as the "context" in QMTest parlance. Since the context
- for LLVM is relatively large, the master Makefile in llvm/test
- sets it for you.
- </p><p>
- The LLVM database class makes the subdirectories of llvm/test a
- QMTest test database. For each directory that contains tests driven by
- QMTest, it knows what type of test the source file is and how to run it.
- </p><p>
- Hence, the QMTest namespace is essentially what you see in the
- Feature and Regression directories, but there is some magic that
- the database class performs (as described below).
- </p><p>
- The QMTest namespace is currently composed of the following tests and
- test suites:
- </p>
-
- <ul>
- <li>Feature
- <p>
- These are the feature tests found in the Feature directory.
- They are broken up into the following categories:
- </p>
- <ul>
- <li>ad
- <p>
- Assembler/Disassembler tests. These tests verify that a
- piece of LLVM assembly language can be assembled into
- bytecode and then disassembled into the original
- assembly language code. It does this several times to
- ensure that assembled output can be disassembled and
- disassembler output can be assembled. It also verifies
- that the give assembly language file can be assembled
- correctly.
- </p>
-
- <li>opt
- <p>
- Optimizer tests. These tests verify that two of the
- optimizer passes completely optimize a program (i.e.
- after a single pass, they cannot optimize a program
- any further).
- </p>
-
- <li>mc
- <p>
- Machine code tests. These tests verify that the LLVM
- assembly language file can be translated into native
- assembly code.
- </p>
-
- <li>cc
- <p>
- C code tests. These tests verify that the specified
- LLVM assembly code can be converted into C source code
- using the C backend.
- </p>
- </ul>
-
- <p>
- The LLVM database class looks at every file in the Feature
- directory and creates a fake test hierarchy containing
- <tt>Feature.<testtype>.<testname></tt>. So, if you
- add an LLVM assembly language file to the Feature directory, it
- actually creates 5 new tests: assembler/disassembler, assembler,
- optimizer, machine code, and C code.
- </p>
-
- <li>Regression
- <p>
- These are the regression tests. There is one suite for each
- subdirectory of the Regression directory. If you add a new
- subdirectory there, you will need to modify, at least, the
- <tt>RegressionMap</tt> variable in <tt>QMTest/llvmdb.py</tt> so
- that QMTest knows how to run the tests in the new subdirectory.
- </p>
- </ul>
- </div>
-
- <!--===============================================================-->
- <div class="doc_section"><h2><a name="progstructure">Programs
- Structure</a></div>
- <!--===============================================================-->
-
- <div class="doc_text">
- <p>
- As mentioned previously, the Programs tree in llvm/test provides three
- types of tests: MultiSource, SingleSource, and External. Each tree is
- then subdivided into several categories, including applications,
- benchmarks, regression tests, code that is strange grammatically, etc.
- These organizations should be relatively self explanatory.
- </p><p>
- In addition to the regular Programs tests, the Programs tree also
- provides a mechanism for compiling the programs in different ways. If
- the variable TEST is defined on the gmake command line, the test system
- will include a Makefile named <tt>TEST.<value of TEST
- variable>.Makefile</tt>. This Makefile can modify build rules to
- yield different results.
- </p><p>
- For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt>
- to create the nightly test reports. To run the nightly tests, run
- <tt>gmake TEST=nightly</tt>.
- </p><p>
- There are several TEST Makefiles available in the tree. Some of them
- are designed for internal LLVM research and will not work outside of the
- LLVM research group. They may still be valuable, however, as a guide to
- writing your own TEST Makefile for any optimization or analysis passes
- that you develop with LLVM.
- </p>
- </div>
-
- <!--===============================================================-->
- <div class="doc_section"><h2><a name="run">Running the LLVM Tests</a>
- </div>
- <!--===============================================================-->
-
- <div class="doc_text">
- <p>
- First, all tests are executed within the LLVM object directory tree.
- They <i>are not</i> executed inside of the LLVM source tree. This is
- because the test suite creates temporary files during execution.
- </p><p>
- The master Makefile in llvm/test is capable of running both the
- QMTest driven tests and the Programs tests. By default, it will run
- all of the tests.
- </p><p>
- To run only the QMTest driven tests, run <tt>gmake qmtest</tt> at the
- command line in llvm/tests. To run a specific qmtest, suffix the test
- name with ".t" when running gmake.
- </p><p>
- For example, to run the Regression.LLC tests, type
- <tt>gmake Regression.LLC.t</tt> in llvm/tests.
- </p><p>
- Note that the Makefiles in llvm/test/Features and llvm/test/Regression
- are gone. You must now use QMTest from the llvm/test directory to run
- them.
- </p><p>
- To run the Programs test, cd into the llvm/test/Programs directory and
- type <tt>gmake</tt>. Alternatively, you can type <tt>gmake
- TEST=<type> test</tt> to run one of the specialized tests in
- llvm/test/Programs/TEST.<type>.Makefile. For example, you could
- run the nightly tester tests using the following commands:
- </p>
-
- <pre>
- % cd llvm/test/Programs
- % gmake TEST=nightly test
- </pre>
-
- <p>
- Regardless of which test you're running, the results are printed on
- standard output and standard error. You can redirect these results to a
- file if you choose.
- </p><p>
- Some tests are known to fail. Some are bugs that we have not fixed yet;
- others are features that we haven't added yet (or may never add). In
- QMTest, the result for such tests will be XFAIL (eXpected FAILure). In
- this way, you can tell the difference between an expected and unexpected
- failure.
- </p><p>
- The Programs tests have no such feature as of this time. If the test
- passes, only warnings and other miscellaneous output will be generated.
- If a test fails, a large <program> FAILED message will be
- displayed. This will help you separate benign warnings from actual test
- failures.
- </p>
- </div>