LLVM Testing Infrastructure Guide
=================================
-Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya
-Lattner
-
.. contents::
:local:
+.. toctree::
+ :hidden:
+
+ TestSuiteMakefileGuide
+
Overview
========
Requirements
============
-In order to use the LLVM testing infrastructure, you will need all of
-the software required to build LLVM, as well as
-`Python <http://python.org>`_ 2.4 or later.
+In order to use the LLVM testing infrastructure, you will need all of the
+software required to build LLVM, as well as `Python <http://python.org>`_ 2.7 or
+later.
LLVM testing infrastructure organization
========================================
----------------
The regression tests are small pieces of code that test a specific
-feature of LLVM or trigger a specific bug in LLVM. They are usually
-written in LLVM assembly language, but can be written in other languages
-if the test targets a particular language front end (and the appropriate
-``--with-llvmgcc`` options were used at ``configure`` time of the
-``llvm`` module). These tests are driven by the 'lit' testing tool,
-which is part of LLVM.
-
-These code fragments are not complete programs. The code generated from
-them is never executed to determine correct behavior.
-
-These code fragment tests are located in the ``llvm/test`` directory.
+feature of LLVM or trigger a specific bug in LLVM. The language they are
+written in depends on the part of LLVM being tested. These tests are driven by
+the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and
+are located in the ``llvm/test`` directory.
Typically when a bug is found in LLVM, a regression test containing just
enough code to reproduce the problem should be written and placed
-somewhere underneath this directory. In most cases, this will be a small
-piece of LLVM assembly language code, often distilled from an actual
-application or benchmark.
+somewhere underneath this directory. For example, it can be a small
+piece of LLVM IR distilled from an actual application or benchmark.
``test-suite``
--------------
The tests are located in two separate Subversion modules. The
regressions tests are in the main "llvm" module under the directory
-``llvm/test`` (so you get these tests for free with the main llvm tree).
-Use "make check-all" to run the regression tests after building LLVM.
+``llvm/test`` (so you get these tests for free with the main LLVM tree).
+Use ``make check-all`` to run the regression tests after building LLVM.
-The more comprehensive test suite that includes whole programs in C and
-C++ is in the ``test-suite`` module. See ```test-suite``
-Quickstart <#testsuitequickstart>`_ for more information on running
-these tests.
+The more comprehensive test suite that includes whole programs in C and C++
+is in the ``test-suite`` module. See :ref:`test-suite Quickstart
+<test-suite-quickstart>` for more information on running these tests.
Regression tests
----------------
-To run all of the LLVM regression tests, use master Makefile in the
-``llvm/test`` directory:
+To run all of the LLVM regression tests, use the master Makefile in the
+``llvm/test`` directory. LLVM Makefiles require GNU Make (read the :doc:`LLVM
+Makefile Guide <MakefileGuide>` for more details):
.. code-block:: bash
- % gmake -C llvm/test
+ % make -C llvm/test
-or
+or:
.. code-block:: bash
- % gmake check
+ % make check
If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you
can run the LLVM and Clang tests simultaneously using:
-or
-
.. code-block:: bash
- % gmake check-all
+ % make check-all
-To run the tests with Valgrind (Memcheck by default), just append
-``VG=1`` to the commands above, e.g.:
+To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make
+variable to pass the required options to lit. For example, you can use:
.. code-block:: bash
- % gmake check VG=1
+ % make check LIT_ARGS="-v --vg --vg-leak"
-To run individual tests or subsets of tests, you can use the 'llvm-lit'
+to enable testing with valgrind and with leak checking enabled.
+
+To run individual tests or subsets of tests, you can use the ``llvm-lit``
script which is built as part of LLVM. For example, to run the
-'Integer/BitPacked.ll' test by itself you can run:
+``Integer/BitPacked.ll`` test by itself you can run:
.. code-block:: bash
% llvm-lit ~/llvm/test/CodeGen/ARM
-For more information on using the 'lit' tool, see 'llvm-lit --help' or
-the 'lit' man page.
+For more information on using the :program:`lit` tool, see ``llvm-lit --help``
+or the :doc:`lit man page <CommandGuide/lit>`.
Debugging Information tests
---------------------------
Regression test structure
=========================
-The LLVM regression tests are driven by 'lit' and are located in the
+The LLVM regression tests are driven by :program:`lit` and are located in the
``llvm/test`` directory.
This directory contains a large array of small tests that exercise
various features of LLVM and to ensure that regressions do not occur.
The directory is broken into several sub-directories, each focused on a
-particular area of LLVM. A few of the important ones are:
-
-- ``Analysis``: checks Analysis passes.
-- ``Archive``: checks the Archive library.
-- ``Assembler``: checks Assembly reader/writer functionality.
-- ``Bitcode``: checks Bitcode reader/writer functionality.
-- ``CodeGen``: checks code generation and each target.
-- ``Features``: checks various features of the LLVM language.
-- ``Linker``: tests bitcode linking.
-- ``Transforms``: tests each of the scalar, IPO, and utility transforms
- to ensure they make the right transformations.
-- ``Verifier``: tests the IR verifier.
+particular area of LLVM.
Writing new regression tests
----------------------------
The regression test structure is very simple, but does require some
information to be set. This information is gathered via ``configure``
-and is written to a file, ``lit.site.cfg`` in ``llvm/test``. The
-``llvm/test`` Makefile does this work for you.
+and is written to a file, ``test/lit.site.cfg`` in the build directory.
+The ``llvm/test`` Makefile does this work for you.
In order for the regression tests to work, each directory of tests must
-have a ``lit.local.cfg`` file. Lit looks for this file to determine how
-to run the tests. This file is just Python code and thus is very
+have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine
+how to run the tests. This file is just Python code and thus is very
flexible, but we've standardized it for the LLVM regression tests. If
you're adding a directory of tests, just copy ``lit.local.cfg`` from
another directory to get running. The standard ``lit.local.cfg`` simply
specifies which files to look in for tests. Any directory that contains
-only directories does not need the ``lit.local.cfg`` file. Read the `Lit
-documentation <http://llvm.org/cmds/lit.html>`_ for more information.
+only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit
+documentation <CommandGuide/lit>` for more information.
-The ``llvm-runtests`` function looks at each file that is passed to it
-and gathers any lines together that match "RUN:". These are the "RUN"
-lines that specify how the test is to be run. So, each test script must
-contain RUN lines if it is to do anything. If there are no RUN lines,
-the ``llvm-runtests`` function will issue an error and the test will
-fail.
+Each test file must contain lines starting with "RUN:" that tell :program:`lit`
+how to run it. If there are no RUN lines, :program:`lit` will issue an error
+while running a test.
RUN lines are specified in the comments of the test program using the
keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
-to execute. Together, these lines form the "script" that
-``llvm-runtests`` executes to run the test case. The syntax of the RUN
-lines is similar to a shell's syntax for pipelines including I/O
-redirection and variable substitution. However, even though these lines
-may *look* like a shell script, they are not. RUN lines are interpreted
-directly by the Tcl ``exec`` command. They are never executed by a
-shell. Consequently the syntax differs from normal shell script syntax
-in a few ways. You can specify as many RUN lines as needed.
-
-lit performs substitution on each RUN line to replace LLVM tool names
+to execute. Together, these lines form the "script" that :program:`lit`
+executes to run the test case. The syntax of the RUN lines is similar to a
+shell's syntax for pipelines including I/O redirection and variable
+substitution. However, even though these lines may *look* like a shell
+script, they are not. RUN lines are interpreted by :program:`lit`.
+Consequently, the syntax differs from shell in a few ways. You can specify
+as many RUN lines as needed.
+
+:program:`lit` performs substitution on each RUN line to replace LLVM tool names
with the full paths to the executable built for each tool (in
-$(LLVM\_OBJ\_ROOT)/$(BuildMode)/bin). This ensures that lit does not
-invoke any stray LLVM tools in the user's path during testing.
+``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does
+not invoke any stray LLVM tools in the user's path during testing.
Each RUN line is executed on its own, distinct from other lines unless
its last character is ``\``. This continuation character causes the RUN
long pipelines of commands without making huge line lengths. The lines
ending in ``\`` are concatenated until a RUN line that doesn't end in
``\`` is found. This concatenated set of RUN lines then constitutes one
-execution. Tcl will substitute variables and arrange for the pipeline to
-be executed. If any process in the pipeline fails, the entire line (and
+execution. :program:`lit` will substitute variables and arrange for the pipeline
+to be executed. If any process in the pipeline fails, the entire line (and
test case) fails too.
Below is an example of legal RUN lines in a ``.ll`` file:
; RUN: llvm-dis < %s.bc-13 > %t2
; RUN: diff %t1 %t2
-As with a Unix shell, the RUN: lines permit pipelines and I/O
-redirection to be used. However, the usage is slightly different than
-for Bash. To check what's legal, see the documentation for the `Tcl
-exec <http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M2>`_ command and the
-`tutorial <http://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html>`_. The
-major differences are:
-
-- You can't do ``2>&1``. That will cause Tcl to write to a file named
- ``&1``. Usually this is done to get stderr to go through a pipe. You
- can do that in tcl with ``|&`` so replace this idiom:
- ``... 2>&1 | grep`` with ``... |& grep``
-- You can only redirect to a file, not to another descriptor and not
- from a here document.
-- tcl supports redirecting to open files with the @ syntax but you
- shouldn't use that here.
+As with a Unix shell, the RUN lines permit pipelines and I/O
+redirection to be used.
There are some quoting rules that you must pay attention to when writing
-your RUN lines. In general nothing needs to be quoted. Tcl won't strip
-off any quote characters so they will get passed to the invoked program.
-For example:
+your RUN lines. In general nothing needs to be quoted. :program:`lit` won't
+strip off any quote characters so they will get passed to the invoked program.
+To avoid this use curly braces to tell :program:`lit` that it should treat
+everything enclosed as one value.
-.. code-block:: bash
+In general, you should strive to keep your RUN lines as simple as possible,
+using them only to run tools that generate textual output you can then examine.
+The recommended way to examine output to figure out if the test passes is using
+the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN
+lines is deprecated - please do not send or commit patches that use it.]*
- ... | grep 'find this string'
+Put related tests into a single file rather than having a separate file per
+test. Check if there are files already covering your feature and consider
+adding your code there instead of creating a new file.
-This will fail because the ' characters are passed to grep. This would
-instruction grep to look for ``'find`` in the files ``this`` and
-``string'``. To avoid this use curly braces to tell Tcl that it should
-treat everything enclosed as one value. So our example would become:
+Extra files
+-----------
-.. code-block:: bash
+If your test requires extra files besides the file containing the ``RUN:``
+lines, the idiomatic place to put them is in a subdirectory ``Inputs``.
+You can then refer to the extra files as ``%S/Inputs/foo.bar``.
- ... | grep {find this string}
+For example, consider ``test/Linker/ident.ll``. The directory structure is
+as follows::
-Additionally, the characters ``[`` and ``]`` are treated specially by
-Tcl. They tell Tcl to interpret the content as a command to execute.
-Since these characters are often used in regular expressions this can
-have disastrous results and cause the entire test run in a directory to
-fail. For example, a common idiom is to look for some basicblock number:
+ test/
+ Linker/
+ ident.ll
+ Inputs/
+ ident.a.ll
+ ident.b.ll
-.. code-block:: bash
+For convenience, these are the contents:
- ... | grep bb[2-8]
+.. code-block:: llvm
-This, however, will cause Tcl to fail because its going to try to
-execute a program named "2-8". Instead, what you want is this:
+ ;;;;; ident.ll:
-.. code-block:: bash
+ ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s
- ... | grep {bb\[2-8\]}
+ ; Verify that multiple input llvm.ident metadata are linked together.
-Finally, if you need to pass the ``\`` character down to a program, then
-it must be doubled. This is another Tcl special character. So, suppose
-you had:
+ ; CHECK-DAG: !llvm.ident = !{!0, !1, !2}
+ ; CHECK-DAG: "Compiler V1"
+ ; CHECK-DAG: "Compiler V2"
+ ; CHECK-DAG: "Compiler V3"
-.. code-block:: bash
+ ;;;;; Inputs/ident.a.ll:
- ... | grep 'i32\*'
+ !llvm.ident = !{!0, !1}
+ !0 = metadata !{metadata !"Compiler V1"}
+ !1 = metadata !{metadata !"Compiler V2"}
-This will fail to match what you want (a pointer to i32). First, the
-``'`` do not get stripped off. Second, the ``\`` gets stripped off by
-Tcl so what grep sees is: ``'i32*'``. That's not likely to match
-anything. To resolve this you must use ``\\`` and the ``{}``, like this:
+ ;;;;; Inputs/ident.b.ll:
-.. code-block:: bash
+ !llvm.ident = !{!0}
+ !0 = metadata !{metadata !"Compiler V3"}
- ... | grep {i32\\*}
+For symmetry reasons, ``ident.ll`` is just a dummy file that doesn't
+actually participate in the test besides holding the ``RUN:`` lines.
-If your system includes GNU ``grep``, make sure that ``GREP_OPTIONS`` is
-not set in your environment. Otherwise, you may get invalid results
-(both false positives and false negatives).
+.. note::
-The FileCheck utility
----------------------
+ Some existing tests use ``RUN: true`` in extra files instead of just
+ putting the extra files in an ``Inputs/`` directory. This pattern is
+ deprecated.
-A powerful feature of the RUN: lines is that it allows any arbitrary
-commands to be executed as part of the test harness. While standard
-(portable) unix tools like 'grep' work fine on run lines, as you see
-above, there are a lot of caveats due to interaction with Tcl syntax,
-and we want to make sure the run lines are portable to a wide range of
-systems. Another major problem is that grep is not very good at checking
-to verify that the output of a tools contains a series of different
-output in a specific order. The FileCheck tool was designed to help with
-these problems.
+Fragile tests
+-------------
-FileCheck (whose basic command line arguments are described in `the
-FileCheck man page <http://llvm.org/cmds/FileCheck.html>`_ is designed
-to read a file to check from standard input, and the set of things to
-verify from a file specified as a command line argument. A simple
-example of using FileCheck from a RUN line looks like this:
+It is easy to write a fragile test that would fail spuriously if the tool being
+tested outputs a full path to the input file. For example, :program:`opt` by
+default outputs a ``ModuleID``:
-.. code-block:: llvm
+.. code-block:: console
- ; RUN: llvm-as < %s | llc -march=x86-64 | FileCheck %s
+ $ cat example.ll
+ define i32 @main() nounwind {
+ ret i32 0
+ }
-This syntax says to pipe the current file ("%s") into llvm-as, pipe that
-into llc, then pipe the output of llc into FileCheck. This means that
-FileCheck will be verifying its standard input (the llc output) against
-the filename argument specified (the original .ll file specified by
-"%s"). To see how this works, let's look at the rest of the .ll file
-(after the RUN line):
+ $ opt -S /path/to/example.ll
+ ; ModuleID = '/path/to/example.ll'
-.. code-block:: llvm
+ define i32 @main() nounwind {
+ ret i32 0
+ }
- define void @sub1(i32* %p, i32 %v) {
- entry:
- ; CHECK: sub1:
- ; CHECK: subl
- %0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v)
- ret void
- }
-
- define void @inc4(i64* %p) {
- entry:
- ; CHECK: inc4:
- ; CHECK: incq
- %0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1)
- ret void
- }
-
-Here you can see some "CHECK:" lines specified in comments. Now you can
-see how the file is piped into llvm-as, then llc, and the machine code
-output is what we are verifying. FileCheck checks the machine code
-output to verify that it matches what the "CHECK:" lines specify.
-
-The syntax of the CHECK: lines is very simple: they are fixed strings
-that must occur in order. FileCheck defaults to ignoring horizontal
-whitespace differences (e.g. a space is allowed to match a tab) but
-otherwise, the contents of the CHECK: line is required to match some
-thing in the test file exactly.
-
-One nice thing about FileCheck (compared to grep) is that it allows
-merging test cases together into logical groups. For example, because
-the test above is checking for the "sub1:" and "inc4:" labels, it will
-not match unless there is a "subl" in between those labels. If it
-existed somewhere else in the file, that would not count: "grep subl"
-matches if subl exists anywhere in the file.
-
-The FileCheck -check-prefix option
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The FileCheck -check-prefix option allows multiple test configurations
-to be driven from one .ll file. This is useful in many circumstances,
-for example, testing different architectural variants with llc. Here's a
-simple example:
+``ModuleID`` can unexpetedly match against ``CHECK`` lines. For example:
.. code-block:: llvm
- ; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \
- ; RUN: | FileCheck %s -check-prefix=X32
- ; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \
- ; RUN: | FileCheck %s -check-prefix=X64
+ ; RUN: opt -S %s | FileCheck
+
+ define i32 @main() nounwind {
+ ; CHECK-NOT: load
+ ret i32 0
+ }
+
+This test will fail if placed into a ``download`` directory.
+
+To make your tests robust, always use ``opt ... < %s`` in the RUN line.
+:program:`opt` does not output a ``ModuleID`` when input comes from stdin.
+
+Platform-Specific Tests
+-----------------------
+
+Whenever adding tests that require the knowledge of a specific platform,
+either related to code generated, specific output or back-end features,
+you must make sure to isolate the features, so that buildbots that
+run on different architectures (and don't even compile all back-ends),
+don't fail.
+
+The first problem is to check for target-specific output, for example sizes
+of structures, paths and architecture names, for example:
+
+* Tests containing Windows paths will fail on Linux and vice-versa.
+* Tests that check for ``x86_64`` somewhere in the text will fail anywhere else.
+* Tests where the debug information calculates the size of types and structures.
+
+Also, if the test rely on any behaviour that is coded in any back-end, it must
+go in its own directory. So, for instance, code generator tests for ARM go
+into ``test/CodeGen/ARM`` and so on. Those directories contain a special
+``lit`` configuration file that ensure all tests in that directory will
+only run if a specific back-end is compiled and available.
- define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind {
- %tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1
- ret <4 x i32> %tmp1
- ; X32: pinsrd_1:
- ; X32: pinsrd $1, 4(%esp), %xmm0
+For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is:
- ; X64: pinsrd_1:
- ; X64: pinsrd $1, %edi, %xmm0
- }
+.. code-block:: python
-In this case, we're testing that we get the expected code generation
-with both 32-bit and 64-bit code generation.
+ config.suffixes = ['.ll', '.c', '.cpp', '.test']
+ if not 'ARM' in config.root.targets:
+ config.unsupported = True
-The "CHECK-NEXT:" directive
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Other platform-specific tests are those that depend on a specific feature
+of a specific sub-architecture, for example only to Intel chips that support ``AVX2``.
-Sometimes you want to match lines and would like to verify that matches
-happen on exactly consecutive lines with no other lines in between them.
-In this case, you can use CHECK: and CHECK-NEXT: directives to specify
-this. If you specified a custom check prefix, just use "<PREFIX>-NEXT:".
-For example, something like this works as you'd expect:
+For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture
+variants:
.. code-block:: llvm
- define void @t2(<2 x double>* %r, <2 x double>* %A, double %B) {
- %tmp3 = load <2 x double>* %A, align 16
- %tmp7 = insertelement <2 x double> undef, double %B, i32 0
- %tmp9 = shufflevector <2 x double> %tmp3,
- <2 x double> %tmp7,
- <2 x i32> < i32 0, i32 2 >
- store <2 x double> %tmp9, <2 x double>* %r, align 16
- ret void
-
- ; CHECK: t2:
- ; CHECK: movl 8(%esp), %eax
- ; CHECK-NEXT: movapd (%eax), %xmm0
- ; CHECK-NEXT: movhpd 12(%esp), %xmm0
- ; CHECK-NEXT: movl 4(%esp), %eax
- ; CHECK-NEXT: movapd %xmm0, (%eax)
- ; CHECK-NEXT: ret
- }
-
-CHECK-NEXT: directives reject the input unless there is exactly one
-newline between it an the previous directive. A CHECK-NEXT cannot be the
-first directive in a file.
-
-The "CHECK-NOT:" directive
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The CHECK-NOT: directive is used to verify that a string doesn't occur
-between two matches (or the first match and the beginning of the file).
-For example, to verify that a load is removed by a transformation, a
-test like this can be used:
+ ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2
+ ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1
+ ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2
+
+And the checks are different:
.. code-block:: llvm
- define i8 @coerce_offset0(i32 %V, i32* %P) {
- store i32 %V, i32* %P
+ ; SSE2: @test1
+ ; SSE2: psubusw LCPI0_0(%rip), %xmm0
+ ; AVX1: @test1
+ ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0
+ ; AVX2: @test1
+ ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0
- %P2 = bitcast i32* %P to i8*
- %P3 = getelementptr i8* %P2, i32 2
+So, if you're testing for a behaviour that you know is platform-specific or
+depends on special features of sub-architectures, you must add the specific
+triple, test with the specific FileCheck and put it into the specific
+directory that will filter out all other architectures.
- %A = load i8* %P3
- ret i8 %A
- ; CHECK: @coerce_offset0
- ; CHECK-NOT: load
- ; CHECK: ret i8
- }
-FileCheck Pattern Matching Syntax
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Substitutions
+-------------
-The CHECK: and CHECK-NOT: directives both take a pattern to match. For
-most uses of FileCheck, fixed string matching is perfectly sufficient.
-For some things, a more flexible form of matching is desired. To support
-this, FileCheck allows you to specify regular expressions in matching
-strings, surrounded by double braces: **{{yourregex}}**. Because we want
-to use fixed string matching for a majority of what we do, FileCheck has
-been designed to support mixing and matching fixed string matching with
-regular expressions. This allows you to write things like this:
+Besides replacing LLVM tool names the following substitutions are performed in
+RUN lines:
-.. code-block:: llvm
+``%%``
+ Replaced by a single ``%``. This allows escaping other substitutions.
- ; CHECK: movhpd {{[0-9]+}}(%esp), {{%xmm[0-7]}}
+``%s``
+ File path to the test case's source. This is suitable for passing on the
+ command line as the input to an LLVM tool.
-In this case, any offset from the ESP register will be allowed, and any
-xmm register will be allowed.
+ Example: ``/home/user/llvm/test/MC/ELF/foo_test.s``
-Because regular expressions are enclosed with double braces, they are
-visually distinct, and you don't need to use escape characters within
-the double braces like you would in C. In the rare case that you want to
-match double braces explicitly from the input, you can use something
-ugly like **{{[{][{]}}** as your pattern.
+``%S``
+ Directory path to the test case's source.
-FileCheck Variables
-^^^^^^^^^^^^^^^^^^^
+ Example: ``/home/user/llvm/test/MC/ELF``
-It is often useful to match a pattern and then verify that it occurs
-again later in the file. For codegen tests, this can be useful to allow
-any register, but verify that that register is used consistently later.
-To do this, FileCheck allows named variables to be defined and
-substituted into patterns. Here is a simple example:
+``%t``
+ File path to a temporary file name that could be used for this test case.
+ The file name won't conflict with other test cases. You can append to it
+ if you need multiple temporaries. This is useful as the destination of
+ some redirected output.
-.. code-block:: llvm
+ Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp``
- ; CHECK: test5:
- ; CHECK: notw [[REGISTER:%[a-z]+]]
- ; CHECK: andw {{.*}}[[REGISTER]]
-
-The first check line matches a regex (``%[a-z]+``) and captures it into
-the variables "REGISTER". The second line verifies that whatever is in
-REGISTER occurs later in the file after an "andw". FileCheck variable
-references are always contained in ``[[ ]]`` pairs, are named, and their
-names can be formed with the regex "``[a-zA-Z][a-zA-Z0-9]*``". If a
-colon follows the name, then it is a definition of the variable, if not,
-it is a use.
-
-FileCheck variables can be defined multiple times, and uses always get
-the latest value. Note that variables are all read at the start of a
-"CHECK" line and are all defined at the end. This means that if you have
-something like "``CHECK: [[XYZ:.*]]x[[XYZ]]``" that the check line will
-read the previous value of the XYZ variable and define a new one after
-the match is performed. If you need to do something like this you can
-probably take advantage of the fact that FileCheck is not actually
-line-oriented when it matches, this allows you to define two separate
-CHECK lines that match on the same line.
-
-Variables and substitutions
----------------------------
+``%T``
+ Directory of ``%t``.
+
+ Example: ``/home/user/llvm.build/test/MC/ELF/Output``
+
+``%{pathsep}``
+
+ Expands to the path separator, i.e. ``:`` (or ``;`` on Windows).
+
+
+**LLVM-specific substitutions:**
-With a RUN line there are a number of substitutions that are permitted.
-In general, any Tcl variable that is available in the ``substitute``
-function (in ``test/lib/llvm.exp``) can be substituted into a RUN line.
-To make a substitution just write the variable's name preceded by a $.
-Additionally, for compatibility reasons with previous versions of the
-test library, certain names can be accessed with an alternate syntax: a
-% prefix. These alternates are deprecated and may go away in a future
-version.
+``%shlibext``
+ The suffix for the host platforms shared library files. This includes the
+ period as the first character.
-Here are the available variable names. The alternate syntax is listed in
-parentheses.
+ Example: ``.so`` (Linux), ``.dylib`` (OS X), ``.dll`` (Windows)
-``$test`` (``%s``)
- The full path to the test case's source. This is suitable for passing on
- the command line as the input to an llvm tool.
+``%exeext``
+ The suffix for the host platforms executable files. This includes the
+ period as the first character.
+
+ Example: ``.exe`` (Windows), empty on Linux.
``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
- The number of the line where this variable is used, with an optional
- integer offset. This can be used in tests with multiple RUN: lines,
- which reference test file's line numbers.
+ The number of the line where this substitution is used, with an optional
+ integer offset. This can be used in tests with multiple RUN lines, which
+ reference test file's line numbers.
-``$srcdir``
- The source directory from where the "``make check``" was run.
-``objdir``
- The object directory that corresponds to the ``$srcdir``.
+**Clang-specific substitutions:**
-``subdir``
- A partial path from the ``test`` directory that contains the
- sub-directory that contains the test source being executed.
+``%clang``
+ Invokes the Clang driver.
-``srcroot``
- The root directory of the LLVM src tree.
+``%clang_cpp``
+ Invokes the Clang driver for C++.
-``objroot``
- The root directory of the LLVM object tree. This could be the same as
- the srcroot.
+``%clang_cl``
+ Invokes the CL-compatible Clang driver.
-``path``
- The path to the directory that contains the test case source. This is
- for locating any supporting files that are not generated by the test,
- but used by the test.
+``%clangxx``
+ Invokes the G++-compatible Clang driver.
-``tmp``
- The path to a temporary file name that could be used for this test case.
- The file name won't conflict with other test cases. You can append to it
- if you need multiple temporaries. This is useful as the destination of
- some redirected output.
+``%clang_cc1``
+ Invokes the Clang frontend.
+
+``%itanium_abi_triple``, ``%ms_abi_triple``
+ These substitutions can be used to get the current target triple adjusted to
+ the desired ABI. For example, if the test suite is running with the
+ ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to
+ ``i686-pc-mingw32``. This allows a test to run with a specific ABI without
+ constraining it to a specific triple.
+
+To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``.
-``target_triplet`` (``%target_triplet``)
- The target triplet that corresponds to the current host machine (the one
- running the test cases). This should probably be called "host".
-``link`` (``%link``)
- This full link command used to link LLVM executables. This has all the
- configured -I, -L and -l options.
+Options
+-------
-``shlibext`` (``%shlibext``)
- The suffix for the host platforms share library (dll) files. This
- includes the period as the first character.
+The llvm lit configuration allows to customize some things with user options:
+
+``llc``, ``opt``, ...
+ Substitute the respective llvm tool name with a custom command line. This
+ allows to specify custom paths and default arguments for these tools.
+ Example:
+
+ % llvm-lit "-Dllc=llc -verify-machineinstrs"
+
+``run_long_tests``
+ Enable the execution of long running tests.
+
+``llvm_site_config``
+ Load the specified lit configuration instead of the default one.
-To add more variables, two things need to be changed. First, add a line
-in the ``test/Makefile`` that creates the ``site.exp`` file. This will
-"set" the variable as a global in the site.exp file. Second, in the
-``test/lib/llvm.exp`` file, in the substitute proc, add the variable
-name to the list of "global" declarations at the beginning of the proc.
-That's it, the variable can then be used in test scripts.
Other Features
--------------
-To make RUN line writing easier, there are several shell scripts located
-in the ``llvm/test/Scripts`` directory. This directory is in the PATH
-when running tests, so you can just call these scripts using their name.
-For example:
-
-``ignore``
- This script runs its arguments and then always returns 0. This is useful
- in cases where the test needs to cause a tool to generate an error (e.g.
- to check the error output). However, any program in a pipeline that
- returns a non-zero result will cause the test to fail. This script
- overcomes that issue and nicely documents that the test case is
- purposefully ignoring the result code of the tool
+To make RUN line writing easier, there are several helper programs. These
+helpers are in the PATH when running tests, so you can just call them using
+their name. For example:
+
``not``
- This script runs its arguments and then inverts the result code from it.
- Zero result codes become 1. Non-zero result codes become 0. This is
- useful to invert the result of a grep. For example "not grep X" means
- succeed only if you don't find X in the input.
+ This program runs its arguments and then inverts the result code from it.
+ Zero result codes become 1. Non-zero result codes become 0.
Sometimes it is necessary to mark a test case as "expected fail" or
XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
; XFAIL: darwin,sun
-To make the output more useful, the ``llvm_runtest`` function wil scan
+To make the output more useful, :program:`lit` will scan
the lines of the test case for ones that contain a pattern that matches
``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
that is related to the test case. The number after "PR" specifies the
programs from somewhere else. When using ``LNT``, use the
``--test-externals`` option to include these tests in the results.
+.. _test-suite-quickstart:
+
``test-suite`` Quickstart
-------------------------
under the hood may want to understand the Makefile based setup.
For more information on the ``test-suite`` Makefile setup, please see
-the `Test Suite Makefile Guide. <TestSuiteMakefileGuide.html>`_
+the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`.
projects / oota-llvm.git / blobdiff