1 =================================
2 LLVM Testing Infrastructure Guide
3 =================================
5 Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya
14 TestSuiteMakefileGuide
19 This document is the reference manual for the LLVM testing
20 infrastructure. It documents the structure of the LLVM testing
21 infrastructure, the tools needed to use it, and how to add and run
27 In order to use the LLVM testing infrastructure, you will need all of
28 the software required to build LLVM, as well as
29 `Python <http://python.org>`_ 2.4 or later.
31 LLVM testing infrastructure organization
32 ========================================
34 The LLVM testing infrastructure contains two major categories of tests:
35 regression tests and whole programs. The regression tests are contained
36 inside the LLVM repository itself under ``llvm/test`` and are expected
37 to always pass -- they should be run before every commit.
39 The whole programs tests are referred to as the "LLVM test suite" (or
40 "test-suite") and are in the ``test-suite`` module in subversion. For
41 historical reasons, these tests are also referred to as the "nightly
42 tests" in places, which is less ambiguous than "test-suite" and remains
43 in use although we run them much more often than nightly.
48 The regression tests are small pieces of code that test a specific
49 feature of LLVM or trigger a specific bug in LLVM. They are usually
50 written in LLVM assembly language, but can be written in other languages
51 if the test targets a particular language front end (and the appropriate
52 ``--with-llvmgcc`` options were used at ``configure`` time of the
53 ``llvm`` module). These tests are driven by the 'lit' testing tool,
54 which is part of LLVM.
56 These code fragments are not complete programs. The code generated from
57 them is never executed to determine correct behavior.
59 These code fragment tests are located in the ``llvm/test`` directory.
61 Typically when a bug is found in LLVM, a regression test containing just
62 enough code to reproduce the problem should be written and placed
63 somewhere underneath this directory. In most cases, this will be a small
64 piece of LLVM assembly language code, often distilled from an actual
65 application or benchmark.
70 The test suite contains whole programs, which are pieces of code which
71 can be compiled and linked into a stand-alone program that can be
72 executed. These programs are generally written in high level languages
75 These programs are compiled using a user specified compiler and set of
76 flags, and then executed to capture the program output and timing
77 information. The output of these programs is compared to a reference
78 output to ensure that the program is being compiled correctly.
80 In addition to compiling and executing programs, whole program tests
81 serve as a way of benchmarking LLVM performance, both in terms of the
82 efficiency of the programs generated as well as the speed with which
83 LLVM compiles, optimizes, and generates code.
85 The test-suite is located in the ``test-suite`` Subversion module.
87 Debugging Information tests
88 ---------------------------
90 The test suite contains tests to check quality of debugging information.
91 The test are written in C based languages or in LLVM assembly language.
93 These tests are compiled and run under a debugger. The debugger output
94 is checked to validate of debugging information. See README.txt in the
95 test suite for more information . This test suite is located in the
96 ``debuginfo-tests`` Subversion module.
101 The tests are located in two separate Subversion modules. The
102 regressions tests are in the main "llvm" module under the directory
103 ``llvm/test`` (so you get these tests for free with the main llvm tree).
104 Use "make check-all" to run the regression tests after building LLVM.
106 The more comprehensive test suite that includes whole programs in C and C++
107 is in the ``test-suite`` module. See :ref:`test-suite Quickstart
108 <test-suite-quickstart>` for more information on running these tests.
113 To run all of the LLVM regression tests, use master Makefile in the
114 ``llvm/test`` directory:
126 If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you
127 can run the LLVM and Clang tests simultaneously using:
135 To run the tests with Valgrind (Memcheck by default), just append
136 ``VG=1`` to the commands above, e.g.:
142 To run individual tests or subsets of tests, you can use the 'llvm-lit'
143 script which is built as part of LLVM. For example, to run the
144 'Integer/BitPacked.ll' test by itself you can run:
148 % llvm-lit ~/llvm/test/Integer/BitPacked.ll
150 or to run all of the ARM CodeGen tests:
154 % llvm-lit ~/llvm/test/CodeGen/ARM
156 For more information on using the 'lit' tool, see 'llvm-lit --help' or
159 Debugging Information tests
160 ---------------------------
162 To run debugging information tests simply checkout the tests inside
163 clang/test directory.
168 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
170 These tests are already set up to run as part of clang regression tests.
172 Regression test structure
173 =========================
175 The LLVM regression tests are driven by 'lit' and are located in the
176 ``llvm/test`` directory.
178 This directory contains a large array of small tests that exercise
179 various features of LLVM and to ensure that regressions do not occur.
180 The directory is broken into several sub-directories, each focused on a
181 particular area of LLVM. A few of the important ones are:
183 - ``Analysis``: checks Analysis passes.
184 - ``Archive``: checks the Archive library.
185 - ``Assembler``: checks Assembly reader/writer functionality.
186 - ``Bitcode``: checks Bitcode reader/writer functionality.
187 - ``CodeGen``: checks code generation and each target.
188 - ``Features``: checks various features of the LLVM language.
189 - ``Linker``: tests bitcode linking.
190 - ``Transforms``: tests each of the scalar, IPO, and utility transforms
191 to ensure they make the right transformations.
192 - ``Verifier``: tests the IR verifier.
194 Writing new regression tests
195 ----------------------------
197 The regression test structure is very simple, but does require some
198 information to be set. This information is gathered via ``configure``
199 and is written to a file, ``lit.site.cfg`` in ``llvm/test``. The
200 ``llvm/test`` Makefile does this work for you.
202 In order for the regression tests to work, each directory of tests must
203 have a ``lit.local.cfg`` file. Lit looks for this file to determine how
204 to run the tests. This file is just Python code and thus is very
205 flexible, but we've standardized it for the LLVM regression tests. If
206 you're adding a directory of tests, just copy ``lit.local.cfg`` from
207 another directory to get running. The standard ``lit.local.cfg`` simply
208 specifies which files to look in for tests. Any directory that contains
209 only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit
210 documentation <CommandGuide/lit>` for more information.
212 The ``llvm-runtests`` function looks at each file that is passed to it
213 and gathers any lines together that match "RUN:". These are the "RUN"
214 lines that specify how the test is to be run. So, each test script must
215 contain RUN lines if it is to do anything. If there are no RUN lines,
216 the ``llvm-runtests`` function will issue an error and the test will
219 RUN lines are specified in the comments of the test program using the
220 keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
221 to execute. Together, these lines form the "script" that
222 ``llvm-runtests`` executes to run the test case. The syntax of the RUN
223 lines is similar to a shell's syntax for pipelines including I/O
224 redirection and variable substitution. However, even though these lines
225 may *look* like a shell script, they are not. RUN lines are interpreted
226 directly by the Tcl ``exec`` command. They are never executed by a
227 shell. Consequently the syntax differs from normal shell script syntax
228 in a few ways. You can specify as many RUN lines as needed.
230 lit performs substitution on each RUN line to replace LLVM tool names
231 with the full paths to the executable built for each tool (in
232 $(LLVM\_OBJ\_ROOT)/$(BuildMode)/bin). This ensures that lit does not
233 invoke any stray LLVM tools in the user's path during testing.
235 Each RUN line is executed on its own, distinct from other lines unless
236 its last character is ``\``. This continuation character causes the RUN
237 line to be concatenated with the next one. In this way you can build up
238 long pipelines of commands without making huge line lengths. The lines
239 ending in ``\`` are concatenated until a RUN line that doesn't end in
240 ``\`` is found. This concatenated set of RUN lines then constitutes one
241 execution. Tcl will substitute variables and arrange for the pipeline to
242 be executed. If any process in the pipeline fails, the entire line (and
243 test case) fails too.
245 Below is an example of legal RUN lines in a ``.ll`` file:
249 ; RUN: llvm-as < %s | llvm-dis > %t1
250 ; RUN: llvm-dis < %s.bc-13 > %t2
253 As with a Unix shell, the RUN: lines permit pipelines and I/O
254 redirection to be used. However, the usage is slightly different than
255 for Bash. To check what's legal, see the documentation for the `Tcl
256 exec <http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M2>`_ command and the
257 `tutorial <http://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html>`_. The
258 major differences are:
260 - You can't do ``2>&1``. That will cause Tcl to write to a file named
261 ``&1``. Usually this is done to get stderr to go through a pipe. You
262 can do that in tcl with ``|&`` so replace this idiom:
263 ``... 2>&1 | grep`` with ``... |& grep``
264 - You can only redirect to a file, not to another descriptor and not
265 from a here document.
266 - tcl supports redirecting to open files with the @ syntax but you
267 shouldn't use that here.
269 There are some quoting rules that you must pay attention to when writing
270 your RUN lines. In general nothing needs to be quoted. Tcl won't strip
271 off any quote characters so they will get passed to the invoked program.
276 ... | grep 'find this string'
278 This will fail because the ' characters are passed to grep. This would
279 instruction grep to look for ``'find`` in the files ``this`` and
280 ``string'``. To avoid this use curly braces to tell Tcl that it should
281 treat everything enclosed as one value. So our example would become:
285 ... | grep {find this string}
287 Additionally, the characters ``[`` and ``]`` are treated specially by
288 Tcl. They tell Tcl to interpret the content as a command to execute.
289 Since these characters are often used in regular expressions this can
290 have disastrous results and cause the entire test run in a directory to
291 fail. For example, a common idiom is to look for some basicblock number:
297 This, however, will cause Tcl to fail because its going to try to
298 execute a program named "2-8". Instead, what you want is this:
302 ... | grep {bb\[2-8\]}
304 Finally, if you need to pass the ``\`` character down to a program, then
305 it must be doubled. This is another Tcl special character. So, suppose
312 This will fail to match what you want (a pointer to i32). First, the
313 ``'`` do not get stripped off. Second, the ``\`` gets stripped off by
314 Tcl so what grep sees is: ``'i32*'``. That's not likely to match
315 anything. To resolve this you must use ``\\`` and the ``{}``, like this:
321 If your system includes GNU ``grep``, make sure that ``GREP_OPTIONS`` is
322 not set in your environment. Otherwise, you may get invalid results
323 (both false positives and false negatives).
325 The FileCheck utility
326 ---------------------
328 A powerful feature of the RUN: lines is that it allows any arbitrary
329 commands to be executed as part of the test harness. While standard
330 (portable) unix tools like 'grep' work fine on run lines, as you see
331 above, there are a lot of caveats due to interaction with Tcl syntax,
332 and we want to make sure the run lines are portable to a wide range of
333 systems. Another major problem is that grep is not very good at checking
334 to verify that the output of a tools contains a series of different
335 output in a specific order. The FileCheck tool was designed to help with
338 FileCheck is designed to read a file to check from standard input, and the set
339 of things to verify from a file specified as a command line argument.
340 FileCheck is described in :doc:`the FileCheck man page
341 <CommandGuide/FileCheck>`.
343 Variables and substitutions
344 ---------------------------
346 With a RUN line there are a number of substitutions that are permitted.
347 In general, any Tcl variable that is available in the ``substitute``
348 function (in ``test/lib/llvm.exp``) can be substituted into a RUN line.
349 To make a substitution just write the variable's name preceded by a $.
350 Additionally, for compatibility reasons with previous versions of the
351 test library, certain names can be accessed with an alternate syntax: a
352 % prefix. These alternates are deprecated and may go away in a future
355 Here are the available variable names. The alternate syntax is listed in
359 The full path to the test case's source. This is suitable for passing on
360 the command line as the input to an llvm tool.
362 ``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
363 The number of the line where this variable is used, with an optional
364 integer offset. This can be used in tests with multiple RUN: lines,
365 which reference test file's line numbers.
368 The source directory from where the "``make check``" was run.
371 The object directory that corresponds to the ``$srcdir``.
374 A partial path from the ``test`` directory that contains the
375 sub-directory that contains the test source being executed.
378 The root directory of the LLVM src tree.
381 The root directory of the LLVM object tree. This could be the same as
385 The path to the directory that contains the test case source. This is
386 for locating any supporting files that are not generated by the test,
387 but used by the test.
390 The path to a temporary file name that could be used for this test case.
391 The file name won't conflict with other test cases. You can append to it
392 if you need multiple temporaries. This is useful as the destination of
393 some redirected output.
395 ``target_triplet`` (``%target_triplet``)
396 The target triplet that corresponds to the current host machine (the one
397 running the test cases). This should probably be called "host".
400 This full link command used to link LLVM executables. This has all the
401 configured -I, -L and -l options.
403 ``shlibext`` (``%shlibext``)
404 The suffix for the host platforms share library (dll) files. This
405 includes the period as the first character.
407 To add more variables, two things need to be changed. First, add a line
408 in the ``test/Makefile`` that creates the ``site.exp`` file. This will
409 "set" the variable as a global in the site.exp file. Second, in the
410 ``test/lib/llvm.exp`` file, in the substitute proc, add the variable
411 name to the list of "global" declarations at the beginning of the proc.
412 That's it, the variable can then be used in test scripts.
417 To make RUN line writing easier, there are several shell scripts located
418 in the ``llvm/test/Scripts`` directory. This directory is in the PATH
419 when running tests, so you can just call these scripts using their name.
423 This script runs its arguments and then always returns 0. This is useful
424 in cases where the test needs to cause a tool to generate an error (e.g.
425 to check the error output). However, any program in a pipeline that
426 returns a non-zero result will cause the test to fail. This script
427 overcomes that issue and nicely documents that the test case is
428 purposefully ignoring the result code of the tool
430 This script runs its arguments and then inverts the result code from it.
431 Zero result codes become 1. Non-zero result codes become 0. This is
432 useful to invert the result of a grep. For example "not grep X" means
433 succeed only if you don't find X in the input.
435 Sometimes it is necessary to mark a test case as "expected fail" or
436 XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
437 on a line near the top of the file. This signals that the test case
438 should succeed if the test fails. Such test cases are counted separately
439 by the testing tool. To specify an expected fail, use the XFAIL keyword
440 in the comments of the test program followed by a colon and one or more
441 failure patterns. Each failure pattern can be either ``*`` (to specify
442 fail everywhere), or a part of a target triple (indicating the test
443 should fail on that platform), or the name of a configurable feature
444 (for example, ``loadable_module``). If there is a match, the test is
445 expected to fail. If not, the test is expected to succeed. To XFAIL
446 everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
453 To make the output more useful, the ``llvm_runtest`` function wil scan
454 the lines of the test case for ones that contain a pattern that matches
455 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
456 that is related to the test case. The number after "PR" specifies the
457 LLVM bugzilla number. When a PR number is specified, it will be used in
458 the pass/fail reporting. This is useful to quickly get some context when
461 Finally, any line that contains "END." will cause the special
462 interpretation of lines to terminate. This is generally done right after
463 the last RUN: line. This has two side effects:
465 (a) it prevents special interpretation of lines that are part of the test
466 program, not the instructions to the test case, and
468 (b) it speeds things up for really big test cases by avoiding
469 interpretation of the remainder of the file.
471 ``test-suite`` Overview
472 =======================
474 The ``test-suite`` module contains a number of programs that can be
475 compiled and executed. The ``test-suite`` includes reference outputs for
476 all of the programs, so that the output of the executed program can be
477 checked for correctness.
479 ``test-suite`` tests are divided into three types of tests: MultiSource,
480 SingleSource, and External.
482 - ``test-suite/SingleSource``
484 The SingleSource directory contains test programs that are only a
485 single source file in size. These are usually small benchmark
486 programs or small programs that calculate a particular value. Several
487 such programs are grouped together in each directory.
489 - ``test-suite/MultiSource``
491 The MultiSource directory contains subdirectories which contain
492 entire programs with multiple source files. Large benchmarks and
493 whole applications go here.
495 - ``test-suite/External``
497 The External directory contains Makefiles for building code that is
498 external to (i.e., not distributed with) LLVM. The most prominent
499 members of this directory are the SPEC 95 and SPEC 2000 benchmark
500 suites. The ``External`` directory does not contain these actual
501 tests, but only the Makefiles that know how to properly compile these
502 programs from somewhere else. When using ``LNT``, use the
503 ``--test-externals`` option to include these tests in the results.
505 .. _test-suite-quickstart:
507 ``test-suite`` Quickstart
508 -------------------------
510 The modern way of running the ``test-suite`` is focused on testing and
511 benchmarking complete compilers using the
512 `LNT <http://llvm.org/docs/lnt>`_ testing infrastructure.
514 For more information on using LNT to execute the ``test-suite``, please
515 see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_
518 ``test-suite`` Makefiles
519 ------------------------
521 Historically, the ``test-suite`` was executed using a complicated setup
522 of Makefiles. The LNT based approach above is recommended for most
523 users, but there are some testing scenarios which are not supported by
524 the LNT approach. In addition, LNT currently uses the Makefile setup
525 under the covers and so developers who are interested in how LNT works
526 under the hood may want to understand the Makefile based setup.
528 For more information on the ``test-suite`` Makefile setup, please see
529 the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`.