1 =================================
2 LLVM Testing Infrastructure Guide
3 =================================
11 TestSuiteMakefileGuide
16 This document is the reference manual for the LLVM testing
17 infrastructure. It documents the structure of the LLVM testing
18 infrastructure, the tools needed to use it, and how to add and run
24 In order to use the LLVM testing infrastructure, you will need all of
25 the software required to build LLVM, as well as
26 `Python <http://python.org>`_ 2.4 or later.
28 LLVM testing infrastructure organization
29 ========================================
31 The LLVM testing infrastructure contains two major categories of tests:
32 regression tests and whole programs. The regression tests are contained
33 inside the LLVM repository itself under ``llvm/test`` and are expected
34 to always pass -- they should be run before every commit.
36 The whole programs tests are referred to as the "LLVM test suite" (or
37 "test-suite") and are in the ``test-suite`` module in subversion. For
38 historical reasons, these tests are also referred to as the "nightly
39 tests" in places, which is less ambiguous than "test-suite" and remains
40 in use although we run them much more often than nightly.
45 The regression tests are small pieces of code that test a specific
46 feature of LLVM or trigger a specific bug in LLVM. The language they are
47 written in depends on the part of LLVM being tested. These tests are driven by
48 the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and
49 are located in the ``llvm/test`` directory.
51 Typically when a bug is found in LLVM, a regression test containing just
52 enough code to reproduce the problem should be written and placed
53 somewhere underneath this directory. For example, it can be a small
54 piece of LLVM IR distilled from an actual application or benchmark.
59 The test suite contains whole programs, which are pieces of code which
60 can be compiled and linked into a stand-alone program that can be
61 executed. These programs are generally written in high level languages
64 These programs are compiled using a user specified compiler and set of
65 flags, and then executed to capture the program output and timing
66 information. The output of these programs is compared to a reference
67 output to ensure that the program is being compiled correctly.
69 In addition to compiling and executing programs, whole program tests
70 serve as a way of benchmarking LLVM performance, both in terms of the
71 efficiency of the programs generated as well as the speed with which
72 LLVM compiles, optimizes, and generates code.
74 The test-suite is located in the ``test-suite`` Subversion module.
76 Debugging Information tests
77 ---------------------------
79 The test suite contains tests to check quality of debugging information.
80 The test are written in C based languages or in LLVM assembly language.
82 These tests are compiled and run under a debugger. The debugger output
83 is checked to validate of debugging information. See README.txt in the
84 test suite for more information . This test suite is located in the
85 ``debuginfo-tests`` Subversion module.
90 The tests are located in two separate Subversion modules. The
91 regressions tests are in the main "llvm" module under the directory
92 ``llvm/test`` (so you get these tests for free with the main LLVM tree).
93 Use ``make check-all`` to run the regression tests after building LLVM.
95 The more comprehensive test suite that includes whole programs in C and C++
96 is in the ``test-suite`` module. See :ref:`test-suite Quickstart
97 <test-suite-quickstart>` for more information on running these tests.
102 To run all of the LLVM regression tests, use the master Makefile in the
103 ``llvm/test`` directory. LLVM Makefiles require GNU Make (read the :doc:`LLVM
104 Makefile Guide <MakefileGuide>` for more details):
116 If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you
117 can run the LLVM and Clang tests simultaneously using:
123 To run the tests with Valgrind (Memcheck by default), just append
124 ``VG=1`` to the commands above, e.g.:
130 To run individual tests or subsets of tests, you can use the ``llvm-lit``
131 script which is built as part of LLVM. For example, to run the
132 ``Integer/BitPacked.ll`` test by itself you can run:
136 % llvm-lit ~/llvm/test/Integer/BitPacked.ll
138 or to run all of the ARM CodeGen tests:
142 % llvm-lit ~/llvm/test/CodeGen/ARM
144 For more information on using the :program:`lit` tool, see ``llvm-lit --help``
145 or the :doc:`lit man page <CommandGuide/lit>`.
147 Debugging Information tests
148 ---------------------------
150 To run debugging information tests simply checkout the tests inside
151 clang/test directory.
156 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
158 These tests are already set up to run as part of clang regression tests.
160 Regression test structure
161 =========================
163 The LLVM regression tests are driven by :program:`lit` and are located in the
164 ``llvm/test`` directory.
166 This directory contains a large array of small tests that exercise
167 various features of LLVM and to ensure that regressions do not occur.
168 The directory is broken into several sub-directories, each focused on a
169 particular area of LLVM.
171 Writing new regression tests
172 ----------------------------
174 The regression test structure is very simple, but does require some
175 information to be set. This information is gathered via ``configure``
176 and is written to a file, ``test/lit.site.cfg`` in the build directory.
177 The ``llvm/test`` Makefile does this work for you.
179 In order for the regression tests to work, each directory of tests must
180 have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine
181 how to run the tests. This file is just Python code and thus is very
182 flexible, but we've standardized it for the LLVM regression tests. If
183 you're adding a directory of tests, just copy ``lit.local.cfg`` from
184 another directory to get running. The standard ``lit.local.cfg`` simply
185 specifies which files to look in for tests. Any directory that contains
186 only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit
187 documentation <CommandGuide/lit>` for more information.
189 Each test file must contain lines starting with "RUN:" that tell :program:`lit`
190 how to run it. If there are no RUN lines, :program:`lit` will issue an error
191 while running a test.
193 RUN lines are specified in the comments of the test program using the
194 keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
195 to execute. Together, these lines form the "script" that :program:`lit`
196 executes to run the test case. The syntax of the RUN lines is similar to a
197 shell's syntax for pipelines including I/O redirection and variable
198 substitution. However, even though these lines may *look* like a shell
199 script, they are not. RUN lines are interpreted by :program:`lit`.
200 Consequently, the syntax differs from shell in a few ways. You can specify
201 as many RUN lines as needed.
203 :program:`lit` performs substitution on each RUN line to replace LLVM tool names
204 with the full paths to the executable built for each tool (in
205 ``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does
206 not invoke any stray LLVM tools in the user's path during testing.
208 Each RUN line is executed on its own, distinct from other lines unless
209 its last character is ``\``. This continuation character causes the RUN
210 line to be concatenated with the next one. In this way you can build up
211 long pipelines of commands without making huge line lengths. The lines
212 ending in ``\`` are concatenated until a RUN line that doesn't end in
213 ``\`` is found. This concatenated set of RUN lines then constitutes one
214 execution. :program:`lit` will substitute variables and arrange for the pipeline
215 to be executed. If any process in the pipeline fails, the entire line (and
216 test case) fails too.
218 Below is an example of legal RUN lines in a ``.ll`` file:
222 ; RUN: llvm-as < %s | llvm-dis > %t1
223 ; RUN: llvm-dis < %s.bc-13 > %t2
226 As with a Unix shell, the RUN lines permit pipelines and I/O
227 redirection to be used. However, the usage is slightly different than
228 for Bash. In general, it's useful to read the code of other tests to figure out
229 what you can use in yours. The major differences are:
231 - You can't do ``2>&1``. That will cause :program:`lit` to write to a file
232 named ``&1``. Usually this is done to get stderr to go through a pipe. You
233 can do that with ``|&`` so replace this idiom:
234 ``... 2>&1 | grep`` with ``... |& grep``
235 - You can only redirect to a file, not to another descriptor and not
236 from a here document.
238 There are some quoting rules that you must pay attention to when writing
239 your RUN lines. In general nothing needs to be quoted. :program:`lit` won't
240 strip off any quote characters so they will get passed to the invoked program.
245 ... | grep 'find this string'
247 This will fail because the ``'`` characters are passed to ``grep``. This would
248 make ``grep`` to look for ``'find`` in the files ``this`` and
249 ``string'``. To avoid this use curly braces to tell :program:`lit` that it
250 should treat everything enclosed as one value. So our example would become:
254 ... | grep {find this string}
256 In general, you should strive to keep your RUN lines as simple as possible,
257 using them only to run tools that generate the output you can then examine. The
258 recommended way to examine output to figure out if the test passes it using the
259 :doc:`FileCheck tool <CommandGuide/FileCheck>`. The usage of ``grep`` in RUN
260 lines is discouraged.
265 It is easy to write a fragile test that would fail spuriously if the tool being
266 tested outputs a full path to the input file. For example, :program:`opt` by
267 default outputs a ``ModuleID``:
269 .. code-block:: console
272 define i32 @main() nounwind {
276 $ opt -S /path/to/example.ll
277 ; ModuleID = '/path/to/example.ll'
279 define i32 @main() nounwind {
283 ``ModuleID`` can unexpetedly match against ``CHECK`` lines. For example:
287 ; RUN: opt -S %s | FileCheck
289 define i32 @main() nounwind {
294 This test will fail if placed into a ``download`` directory.
296 To make your tests robust, always use ``opt ... < %s`` in the RUN line.
297 :program:`opt` does not output a ``ModuleID`` when input comes from stdin.
299 The FileCheck utility
300 ---------------------
302 A powerful feature of the RUN lines is that it allows any arbitrary
303 commands to be executed as part of the test harness. While standard
304 (portable) unix tools like ``grep`` work fine on run lines, as you see
305 above, there are a lot of caveats due to interaction with shell syntax,
306 and we want to make sure the run lines are portable to a wide range of
307 systems. Another major problem is that ``grep`` is not very good at checking
308 to verify that the output of a tools contains a series of different
309 output in a specific order. The :program:`FileCheck` tool was designed to
310 help with these problems.
312 :program:`FileCheck` is designed to read a file to check from standard input,
313 and the set of things to verify from a file specified as a command line
314 argument. :program:`FileCheck` is described in :doc:`the FileCheck man page
315 <CommandGuide/FileCheck>`.
317 Variables and substitutions
318 ---------------------------
320 With a RUN line there are a number of substitutions that are permitted.
321 To make a substitution just write the variable's name preceded by a ``$``.
322 Additionally, for compatibility reasons with previous versions of the
323 test library, certain names can be accessed with an alternate syntax: a
324 % prefix. These alternates are deprecated and may go away in a future
327 Here are the available variable names. The alternate syntax is listed in
331 The full path to the test case's source. This is suitable for passing on
332 the command line as the input to an LLVM tool.
334 ``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
335 The number of the line where this variable is used, with an optional
336 integer offset. This can be used in tests with multiple RUN lines,
337 which reference test file's line numbers.
340 The source directory from where the ``make check`` was run.
343 The object directory that corresponds to the ``$srcdir``.
346 A partial path from the ``test`` directory that contains the
347 sub-directory that contains the test source being executed.
350 The root directory of the LLVM src tree.
353 The root directory of the LLVM object tree. This could be the same as
357 The path to the directory that contains the test case source. This is
358 for locating any supporting files that are not generated by the test,
359 but used by the test.
362 The path to a temporary file name that could be used for this test case.
363 The file name won't conflict with other test cases. You can append to it
364 if you need multiple temporaries. This is useful as the destination of
365 some redirected output.
367 ``target_triplet`` (``%target_triplet``)
368 The target triplet that corresponds to the current host machine (the one
369 running the test cases). This should probably be called "host".
372 This full link command used to link LLVM executables. This has all the
373 configured ``-I``, ``-L`` and ``-l`` options.
375 ``shlibext`` (``%shlibext``)
376 The suffix for the host platforms shared library (DLL) files. This
377 includes the period as the first character.
379 To add more variables, look at ``test/lit.cfg``.
384 To make RUN line writing easier, there are several helper scripts and programs
385 in the ``llvm/test/Scripts`` directory. This directory is in the PATH
386 when running tests, so you can just call these scripts using their name.
390 This script runs its arguments and then always returns 0. This is useful
391 in cases where the test needs to cause a tool to generate an error (e.g.
392 to check the error output). However, any program in a pipeline that
393 returns a non-zero result will cause the test to fail. This script
394 overcomes that issue and nicely documents that the test case is
395 purposefully ignoring the result code of the tool
397 This script runs its arguments and then inverts the result code from it.
398 Zero result codes become 1. Non-zero result codes become 0.
400 Sometimes it is necessary to mark a test case as "expected fail" or
401 XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
402 on a line near the top of the file. This signals that the test case
403 should succeed if the test fails. Such test cases are counted separately
404 by the testing tool. To specify an expected fail, use the XFAIL keyword
405 in the comments of the test program followed by a colon and one or more
406 failure patterns. Each failure pattern can be either ``*`` (to specify
407 fail everywhere), or a part of a target triple (indicating the test
408 should fail on that platform), or the name of a configurable feature
409 (for example, ``loadable_module``). If there is a match, the test is
410 expected to fail. If not, the test is expected to succeed. To XFAIL
411 everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
418 To make the output more useful, :program:`lit` will scan
419 the lines of the test case for ones that contain a pattern that matches
420 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
421 that is related to the test case. The number after "PR" specifies the
422 LLVM bugzilla number. When a PR number is specified, it will be used in
423 the pass/fail reporting. This is useful to quickly get some context when
426 Finally, any line that contains "END." will cause the special
427 interpretation of lines to terminate. This is generally done right after
428 the last RUN: line. This has two side effects:
430 (a) it prevents special interpretation of lines that are part of the test
431 program, not the instructions to the test case, and
433 (b) it speeds things up for really big test cases by avoiding
434 interpretation of the remainder of the file.
436 ``test-suite`` Overview
437 =======================
439 The ``test-suite`` module contains a number of programs that can be
440 compiled and executed. The ``test-suite`` includes reference outputs for
441 all of the programs, so that the output of the executed program can be
442 checked for correctness.
444 ``test-suite`` tests are divided into three types of tests: MultiSource,
445 SingleSource, and External.
447 - ``test-suite/SingleSource``
449 The SingleSource directory contains test programs that are only a
450 single source file in size. These are usually small benchmark
451 programs or small programs that calculate a particular value. Several
452 such programs are grouped together in each directory.
454 - ``test-suite/MultiSource``
456 The MultiSource directory contains subdirectories which contain
457 entire programs with multiple source files. Large benchmarks and
458 whole applications go here.
460 - ``test-suite/External``
462 The External directory contains Makefiles for building code that is
463 external to (i.e., not distributed with) LLVM. The most prominent
464 members of this directory are the SPEC 95 and SPEC 2000 benchmark
465 suites. The ``External`` directory does not contain these actual
466 tests, but only the Makefiles that know how to properly compile these
467 programs from somewhere else. When using ``LNT``, use the
468 ``--test-externals`` option to include these tests in the results.
470 .. _test-suite-quickstart:
472 ``test-suite`` Quickstart
473 -------------------------
475 The modern way of running the ``test-suite`` is focused on testing and
476 benchmarking complete compilers using the
477 `LNT <http://llvm.org/docs/lnt>`_ testing infrastructure.
479 For more information on using LNT to execute the ``test-suite``, please
480 see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_
483 ``test-suite`` Makefiles
484 ------------------------
486 Historically, the ``test-suite`` was executed using a complicated setup
487 of Makefiles. The LNT based approach above is recommended for most
488 users, but there are some testing scenarios which are not supported by
489 the LNT approach. In addition, LNT currently uses the Makefile setup
490 under the covers and so developers who are interested in how LNT works
491 under the hood may want to understand the Makefile based setup.
493 For more information on the ``test-suite`` Makefile setup, please see
494 the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`.