X-Git-Url: http://plrg.eecs.uci.edu/git/?a=blobdiff_plain;f=docs%2FTestingGuide.rst;h=481be55b576b89e7f67ca71631fa07d20ccc618b;hb=1e556a80ffbc7c4ecb99848e14995119337c9362;hp=69d24225c9ca7b297c336b72cb13e5b55e545d4b;hpb=44da23424094f0af917697fb6eac87e923189605;p=oota-llvm.git diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst index 69d24225c9c..481be55b576 100644 --- a/docs/TestingGuide.rst +++ b/docs/TestingGuide.rst @@ -2,9 +2,6 @@ LLVM Testing Infrastructure Guide ================================= -Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya -Lattner - .. contents:: :local: @@ -24,9 +21,9 @@ tests. Requirements ============ -In order to use the LLVM testing infrastructure, you will need all of -the software required to build LLVM, as well as -`Python `_ 2.4 or later. +In order to use the LLVM testing infrastructure, you will need all of the +software required to build LLVM, as well as `Python `_ 2.5 or +later. LLVM testing infrastructure organization ======================================== @@ -46,23 +43,15 @@ Regression tests ---------------- The regression tests are small pieces of code that test a specific -feature of LLVM or trigger a specific bug in LLVM. They are usually -written in LLVM assembly language, but can be written in other languages -if the test targets a particular language front end (and the appropriate -``--with-llvmgcc`` options were used at ``configure`` time of the -``llvm`` module). These tests are driven by the 'lit' testing tool, -which is part of LLVM. - -These code fragments are not complete programs. The code generated from -them is never executed to determine correct behavior. - -These code fragment tests are located in the ``llvm/test`` directory. +feature of LLVM or trigger a specific bug in LLVM. The language they are +written in depends on the part of LLVM being tested. These tests are driven by +the :doc:`Lit ` testing tool (which is part of LLVM), and +are located in the ``llvm/test`` directory. Typically when a bug is found in LLVM, a regression test containing just enough code to reproduce the problem should be written and placed -somewhere underneath this directory. In most cases, this will be a small -piece of LLVM assembly language code, often distilled from an actual -application or benchmark. +somewhere underneath this directory. For example, it can be a small +piece of LLVM IR distilled from an actual application or benchmark. ``test-suite`` -------------- @@ -100,8 +89,8 @@ Quick start The tests are located in two separate Subversion modules. The regressions tests are in the main "llvm" module under the directory -``llvm/test`` (so you get these tests for free with the main llvm tree). -Use "make check-all" to run the regression tests after building LLVM. +``llvm/test`` (so you get these tests for free with the main LLVM tree). +Use ``make check-all`` to run the regression tests after building LLVM. The more comprehensive test suite that includes whole programs in C and C++ is in the ``test-suite`` module. See :ref:`test-suite Quickstart @@ -110,38 +99,39 @@ is in the ``test-suite`` module. See :ref:`test-suite Quickstart Regression tests ---------------- -To run all of the LLVM regression tests, use master Makefile in the -``llvm/test`` directory: +To run all of the LLVM regression tests, use the master Makefile in the +``llvm/test`` directory. LLVM Makefiles require GNU Make (read the :doc:`LLVM +Makefile Guide ` for more details): .. code-block:: bash - % gmake -C llvm/test + % make -C llvm/test -or +or: .. code-block:: bash - % gmake check + % make check If you have `Clang `_ checked out and built, you can run the LLVM and Clang tests simultaneously using: -or - .. code-block:: bash - % gmake check-all + % make check-all -To run the tests with Valgrind (Memcheck by default), just append -``VG=1`` to the commands above, e.g.: +To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make +variable to pass the required options to lit. For example, you can use: .. code-block:: bash - % gmake check VG=1 + % make check LIT_ARGS="-v --vg --vg-leak" -To run individual tests or subsets of tests, you can use the 'llvm-lit' +to enable testing with valgrind and with leak checking enabled. + +To run individual tests or subsets of tests, you can use the ``llvm-lit`` script which is built as part of LLVM. For example, to run the -'Integer/BitPacked.ll' test by itself you can run: +``Integer/BitPacked.ll`` test by itself you can run: .. code-block:: bash @@ -153,8 +143,8 @@ or to run all of the ARM CodeGen tests: % llvm-lit ~/llvm/test/CodeGen/ARM -For more information on using the 'lit' tool, see 'llvm-lit --help' or -the 'lit' man page. +For more information on using the :program:`lit` tool, see ``llvm-lit --help`` +or the :doc:`lit man page `. Debugging Information tests --------------------------- @@ -172,36 +162,25 @@ These tests are already set up to run as part of clang regression tests. Regression test structure ========================= -The LLVM regression tests are driven by 'lit' and are located in the +The LLVM regression tests are driven by :program:`lit` and are located in the ``llvm/test`` directory. This directory contains a large array of small tests that exercise various features of LLVM and to ensure that regressions do not occur. The directory is broken into several sub-directories, each focused on a -particular area of LLVM. A few of the important ones are: - -- ``Analysis``: checks Analysis passes. -- ``Archive``: checks the Archive library. -- ``Assembler``: checks Assembly reader/writer functionality. -- ``Bitcode``: checks Bitcode reader/writer functionality. -- ``CodeGen``: checks code generation and each target. -- ``Features``: checks various features of the LLVM language. -- ``Linker``: tests bitcode linking. -- ``Transforms``: tests each of the scalar, IPO, and utility transforms - to ensure they make the right transformations. -- ``Verifier``: tests the IR verifier. +particular area of LLVM. Writing new regression tests ---------------------------- The regression test structure is very simple, but does require some information to be set. This information is gathered via ``configure`` -and is written to a file, ``lit.site.cfg`` in ``llvm/test``. The -``llvm/test`` Makefile does this work for you. +and is written to a file, ``test/lit.site.cfg`` in the build directory. +The ``llvm/test`` Makefile does this work for you. In order for the regression tests to work, each directory of tests must -have a ``lit.local.cfg`` file. Lit looks for this file to determine how -to run the tests. This file is just Python code and thus is very +have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine +how to run the tests. This file is just Python code and thus is very flexible, but we've standardized it for the LLVM regression tests. If you're adding a directory of tests, just copy ``lit.local.cfg`` from another directory to get running. The standard ``lit.local.cfg`` simply @@ -209,28 +188,24 @@ specifies which files to look in for tests. Any directory that contains only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit documentation ` for more information. -The ``llvm-runtests`` function looks at each file that is passed to it -and gathers any lines together that match "RUN:". These are the "RUN" -lines that specify how the test is to be run. So, each test script must -contain RUN lines if it is to do anything. If there are no RUN lines, -the ``llvm-runtests`` function will issue an error and the test will -fail. +Each test file must contain lines starting with "RUN:" that tell :program:`lit` +how to run it. If there are no RUN lines, :program:`lit` will issue an error +while running a test. RUN lines are specified in the comments of the test program using the keyword ``RUN`` followed by a colon, and lastly the command (pipeline) -to execute. Together, these lines form the "script" that -``llvm-runtests`` executes to run the test case. The syntax of the RUN -lines is similar to a shell's syntax for pipelines including I/O -redirection and variable substitution. However, even though these lines -may *look* like a shell script, they are not. RUN lines are interpreted -directly by the Tcl ``exec`` command. They are never executed by a -shell. Consequently the syntax differs from normal shell script syntax -in a few ways. You can specify as many RUN lines as needed. - -lit performs substitution on each RUN line to replace LLVM tool names +to execute. Together, these lines form the "script" that :program:`lit` +executes to run the test case. The syntax of the RUN lines is similar to a +shell's syntax for pipelines including I/O redirection and variable +substitution. However, even though these lines may *look* like a shell +script, they are not. RUN lines are interpreted by :program:`lit`. +Consequently, the syntax differs from shell in a few ways. You can specify +as many RUN lines as needed. + +:program:`lit` performs substitution on each RUN line to replace LLVM tool names with the full paths to the executable built for each tool (in -$(LLVM\_OBJ\_ROOT)/$(BuildMode)/bin). This ensures that lit does not -invoke any stray LLVM tools in the user's path during testing. +``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does +not invoke any stray LLVM tools in the user's path during testing. Each RUN line is executed on its own, distinct from other lines unless its last character is ``\``. This continuation character causes the RUN @@ -238,8 +213,8 @@ line to be concatenated with the next one. In this way you can build up long pipelines of commands without making huge line lengths. The lines ending in ``\`` are concatenated until a RUN line that doesn't end in ``\`` is found. This concatenated set of RUN lines then constitutes one -execution. Tcl will substitute variables and arrange for the pipeline to -be executed. If any process in the pipeline fails, the entire line (and +execution. :program:`lit` will substitute variables and arrange for the pipeline +to be executed. If any process in the pipeline fails, the entire line (and test case) fails too. Below is an example of legal RUN lines in a ``.ll`` file: @@ -250,377 +225,212 @@ Below is an example of legal RUN lines in a ``.ll`` file: ; RUN: llvm-dis < %s.bc-13 > %t2 ; RUN: diff %t1 %t2 -As with a Unix shell, the RUN: lines permit pipelines and I/O -redirection to be used. However, the usage is slightly different than -for Bash. To check what's legal, see the documentation for the `Tcl -exec `_ command and the -`tutorial `_. The -major differences are: - -- You can't do ``2>&1``. That will cause Tcl to write to a file named - ``&1``. Usually this is done to get stderr to go through a pipe. You - can do that in tcl with ``|&`` so replace this idiom: - ``... 2>&1 | grep`` with ``... |& grep`` -- You can only redirect to a file, not to another descriptor and not - from a here document. -- tcl supports redirecting to open files with the @ syntax but you - shouldn't use that here. +As with a Unix shell, the RUN lines permit pipelines and I/O +redirection to be used. There are some quoting rules that you must pay attention to when writing -your RUN lines. In general nothing needs to be quoted. Tcl won't strip -off any quote characters so they will get passed to the invoked program. -For example: - -.. code-block:: bash +your RUN lines. In general nothing needs to be quoted. :program:`lit` won't +strip off any quote characters so they will get passed to the invoked program. +To avoid this use curly braces to tell :program:`lit` that it should treat +everything enclosed as one value. - ... | grep 'find this string' +In general, you should strive to keep your RUN lines as simple as possible, +using them only to run tools that generate textual output you can then examine. +The recommended way to examine output to figure out if the test passes is using +the :doc:`FileCheck tool `. *[The usage of grep in RUN +lines is deprecated - please do not send or commit patches that use it.]* -This will fail because the ' characters are passed to grep. This would -instruction grep to look for ``'find`` in the files ``this`` and -``string'``. To avoid this use curly braces to tell Tcl that it should -treat everything enclosed as one value. So our example would become: +Fragile tests +------------- -.. code-block:: bash +It is easy to write a fragile test that would fail spuriously if the tool being +tested outputs a full path to the input file. For example, :program:`opt` by +default outputs a ``ModuleID``: - ... | grep {find this string} +.. code-block:: console -Additionally, the characters ``[`` and ``]`` are treated specially by -Tcl. They tell Tcl to interpret the content as a command to execute. -Since these characters are often used in regular expressions this can -have disastrous results and cause the entire test run in a directory to -fail. For example, a common idiom is to look for some basicblock number: + $ cat example.ll + define i32 @main() nounwind { + ret i32 0 + } -.. code-block:: bash + $ opt -S /path/to/example.ll + ; ModuleID = '/path/to/example.ll' - ... | grep bb[2-8] + define i32 @main() nounwind { + ret i32 0 + } -This, however, will cause Tcl to fail because its going to try to -execute a program named "2-8". Instead, what you want is this: +``ModuleID`` can unexpetedly match against ``CHECK`` lines. For example: -.. code-block:: bash +.. code-block:: llvm - ... | grep {bb\[2-8\]} + ; RUN: opt -S %s | FileCheck -Finally, if you need to pass the ``\`` character down to a program, then -it must be doubled. This is another Tcl special character. So, suppose -you had: + define i32 @main() nounwind { + ; CHECK-NOT: load + ret i32 0 + } -.. code-block:: bash +This test will fail if placed into a ``download`` directory. - ... | grep 'i32\*' +To make your tests robust, always use ``opt ... < %s`` in the RUN line. +:program:`opt` does not output a ``ModuleID`` when input comes from stdin. -This will fail to match what you want (a pointer to i32). First, the -``'`` do not get stripped off. Second, the ``\`` gets stripped off by -Tcl so what grep sees is: ``'i32*'``. That's not likely to match -anything. To resolve this you must use ``\\`` and the ``{}``, like this: +Platform-Specific Tests +----------------------- -.. code-block:: bash +Whenever adding tests that require the knowledge of a specific platform, +either related to code generated, specific output or back-end features, +you must make sure to isolate the features, so that buildbots that +run on different architectures (and don't even compile all back-ends), +don't fail. - ... | grep {i32\\*} +The first problem is to check for target-specific output, for example sizes +of structures, paths and architecture names, for example: -If your system includes GNU ``grep``, make sure that ``GREP_OPTIONS`` is -not set in your environment. Otherwise, you may get invalid results -(both false positives and false negatives). +* Tests containing Windows paths will fail on Linux and vice-versa. +* Tests that check for ``x86_64`` somewhere in the text will fail anywhere else. +* Tests where the debug information calculates the size of types and structures. -The FileCheck utility ---------------------- +Also, if the test rely on any behaviour that is coded in any back-end, it must +go in its own directory. So, for instance, code generator tests for ARM go +into ``test/CodeGen/ARM`` and so on. Those directories contain a special +``lit`` configuration file that ensure all tests in that directory will +only run if a specific back-end is compiled and available. -A powerful feature of the RUN: lines is that it allows any arbitrary -commands to be executed as part of the test harness. While standard -(portable) unix tools like 'grep' work fine on run lines, as you see -above, there are a lot of caveats due to interaction with Tcl syntax, -and we want to make sure the run lines are portable to a wide range of -systems. Another major problem is that grep is not very good at checking -to verify that the output of a tools contains a series of different -output in a specific order. The FileCheck tool was designed to help with -these problems. +For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is: -FileCheck (whose basic command line arguments are described in `the -FileCheck man page `_ is designed -to read a file to check from standard input, and the set of things to -verify from a file specified as a command line argument. A simple -example of using FileCheck from a RUN line looks like this: +.. code-block:: python -.. code-block:: llvm + config.suffixes = ['.ll', '.c', '.cpp', '.test'] + if not 'ARM' in config.root.targets: + config.unsupported = True - ; RUN: llvm-as < %s | llc -march=x86-64 | FileCheck %s +Other platform-specific tests are those that depend on a specific feature +of a specific sub-architecture, for example only to Intel chips that support ``AVX2``. -This syntax says to pipe the current file ("%s") into llvm-as, pipe that -into llc, then pipe the output of llc into FileCheck. This means that -FileCheck will be verifying its standard input (the llc output) against -the filename argument specified (the original .ll file specified by -"%s"). To see how this works, let's look at the rest of the .ll file -(after the RUN line): +For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture +variants: .. code-block:: llvm - define void @sub1(i32* %p, i32 %v) { - entry: - ; CHECK: sub1: - ; CHECK: subl - %0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v) - ret void - } - - define void @inc4(i64* %p) { - entry: - ; CHECK: inc4: - ; CHECK: incq - %0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1) - ret void - } - -Here you can see some "CHECK:" lines specified in comments. Now you can -see how the file is piped into llvm-as, then llc, and the machine code -output is what we are verifying. FileCheck checks the machine code -output to verify that it matches what the "CHECK:" lines specify. - -The syntax of the CHECK: lines is very simple: they are fixed strings -that must occur in order. FileCheck defaults to ignoring horizontal -whitespace differences (e.g. a space is allowed to match a tab) but -otherwise, the contents of the CHECK: line is required to match some -thing in the test file exactly. - -One nice thing about FileCheck (compared to grep) is that it allows -merging test cases together into logical groups. For example, because -the test above is checking for the "sub1:" and "inc4:" labels, it will -not match unless there is a "subl" in between those labels. If it -existed somewhere else in the file, that would not count: "grep subl" -matches if subl exists anywhere in the file. - -The FileCheck -check-prefix option -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The FileCheck -check-prefix option allows multiple test configurations -to be driven from one .ll file. This is useful in many circumstances, -for example, testing different architectural variants with llc. Here's a -simple example: - -.. code-block:: llvm + ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2 + ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1 + ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2 - ; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \ - ; RUN: | FileCheck %s -check-prefix=X32 - ; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \ - ; RUN: | FileCheck %s -check-prefix=X64 +And the checks are different: - define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind { - %tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1 - ret <4 x i32> %tmp1 - ; X32: pinsrd_1: - ; X32: pinsrd $1, 4(%esp), %xmm0 +.. code-block:: llvm - ; X64: pinsrd_1: - ; X64: pinsrd $1, %edi, %xmm0 - } + ; SSE2: @test1 + ; SSE2: psubusw LCPI0_0(%rip), %xmm0 + ; AVX1: @test1 + ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 + ; AVX2: @test1 + ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 -In this case, we're testing that we get the expected code generation -with both 32-bit and 64-bit code generation. +So, if you're testing for a behaviour that you know is platform-specific or +depends on special features of sub-architectures, you must add the specific +triple, test with the specific FileCheck and put it into the specific +directory that will filter out all other architectures. -The "CHECK-NEXT:" directive -^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Sometimes you want to match lines and would like to verify that matches -happen on exactly consecutive lines with no other lines in between them. -In this case, you can use CHECK: and CHECK-NEXT: directives to specify -this. If you specified a custom check prefix, just use "-NEXT:". -For example, something like this works as you'd expect: +Substitutions +------------- -.. code-block:: llvm +Besides replacing LLVM tool names the following substitutions are performed in +RUN lines: - define void @t2(<2 x double>* %r, <2 x double>* %A, double %B) { - %tmp3 = load <2 x double>* %A, align 16 - %tmp7 = insertelement <2 x double> undef, double %B, i32 0 - %tmp9 = shufflevector <2 x double> %tmp3, - <2 x double> %tmp7, - <2 x i32> < i32 0, i32 2 > - store <2 x double> %tmp9, <2 x double>* %r, align 16 - ret void - - ; CHECK: t2: - ; CHECK: movl 8(%esp), %eax - ; CHECK-NEXT: movapd (%eax), %xmm0 - ; CHECK-NEXT: movhpd 12(%esp), %xmm0 - ; CHECK-NEXT: movl 4(%esp), %eax - ; CHECK-NEXT: movapd %xmm0, (%eax) - ; CHECK-NEXT: ret - } - -CHECK-NEXT: directives reject the input unless there is exactly one -newline between it an the previous directive. A CHECK-NEXT cannot be the -first directive in a file. - -The "CHECK-NOT:" directive -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The CHECK-NOT: directive is used to verify that a string doesn't occur -between two matches (or the first match and the beginning of the file). -For example, to verify that a load is removed by a transformation, a -test like this can be used: +``%%`` + Replaced by a single ``%``. This allows escaping other substitutions. -.. code-block:: llvm +``%s`` + File path to the test case's source. This is suitable for passing on the + command line as the input to an LLVM tool. - define i8 @coerce_offset0(i32 %V, i32* %P) { - store i32 %V, i32* %P + Example: ``/home/user/llvm/test/MC/ELF/foo_test.s`` - %P2 = bitcast i32* %P to i8* - %P3 = getelementptr i8* %P2, i32 2 +``%S`` + Directory path to the test case's source. - %A = load i8* %P3 - ret i8 %A - ; CHECK: @coerce_offset0 - ; CHECK-NOT: load - ; CHECK: ret i8 - } + Example: ``/home/user/llvm/test/MC/ELF`` -FileCheck Pattern Matching Syntax -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +``%t`` + File path to a temporary file name that could be used for this test case. + The file name won't conflict with other test cases. You can append to it + if you need multiple temporaries. This is useful as the destination of + some redirected output. -The CHECK: and CHECK-NOT: directives both take a pattern to match. For -most uses of FileCheck, fixed string matching is perfectly sufficient. -For some things, a more flexible form of matching is desired. To support -this, FileCheck allows you to specify regular expressions in matching -strings, surrounded by double braces: ``{{yourregex}}``. Because we want -to use fixed string matching for a majority of what we do, FileCheck has -been designed to support mixing and matching fixed string matching with -regular expressions. This allows you to write things like this: + Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp`` -.. code-block:: llvm +``%T`` + Directory of ``%t``. - ; CHECK: movhpd {{[0-9]+}}(%esp), {{%xmm[0-7]}} + Example: ``/home/user/llvm.build/test/MC/ELF/Output`` -In this case, any offset from the ESP register will be allowed, and any -xmm register will be allowed. +``%{pathsep}`` -Because regular expressions are enclosed with double braces, they are -visually distinct, and you don't need to use escape characters within -the double braces like you would in C. In the rare case that you want to -match double braces explicitly from the input, you can use something -ugly like ``{{[{][{]}}`` as your pattern. + Expands to the path separator, i.e. ``:`` (or ``;`` on Windows). -FileCheck Variables -^^^^^^^^^^^^^^^^^^^ -It is often useful to match a pattern and then verify that it occurs -again later in the file. For codegen tests, this can be useful to allow -any register, but verify that that register is used consistently later. -To do this, FileCheck allows named variables to be defined and -substituted into patterns. Here is a simple example: +**LLVM-specific substitutions:** -.. code-block:: llvm +``%shlibext`` + The suffix for the host platforms shared library files. This includes the + period as the first character. - ; CHECK: test5: - ; CHECK: notw [[REGISTER:%[a-z]+]] - ; CHECK: andw {{.*}}[[REGISTER]] - -The first check line matches a regex (``%[a-z]+``) and captures it into -the variables "REGISTER". The second line verifies that whatever is in -REGISTER occurs later in the file after an "andw". FileCheck variable -references are always contained in ``[[ ]]`` pairs, are named, and their -names can be formed with the regex "``[a-zA-Z][a-zA-Z0-9]*``". If a -colon follows the name, then it is a definition of the variable, if not, -it is a use. - -FileCheck variables can be defined multiple times, and uses always get -the latest value. Note that variables are all read at the start of a -"CHECK" line and are all defined at the end. This means that if you have -something like "``CHECK: [[XYZ:.*]]x[[XYZ]]``" that the check line will -read the previous value of the XYZ variable and define a new one after -the match is performed. If you need to do something like this you can -probably take advantage of the fact that FileCheck is not actually -line-oriented when it matches, this allows you to define two separate -CHECK lines that match on the same line. - -Variables and substitutions ---------------------------- - -With a RUN line there are a number of substitutions that are permitted. -In general, any Tcl variable that is available in the ``substitute`` -function (in ``test/lib/llvm.exp``) can be substituted into a RUN line. -To make a substitution just write the variable's name preceded by a $. -Additionally, for compatibility reasons with previous versions of the -test library, certain names can be accessed with an alternate syntax: a -% prefix. These alternates are deprecated and may go away in a future -version. + Example: ``.so`` (Linux), ``.dylib`` (OS X), ``.dll`` (Windows) -Here are the available variable names. The alternate syntax is listed in -parentheses. +``%exeext`` + The suffix for the host platforms executable files. This includes the + period as the first character. -``$test`` (``%s``) - The full path to the test case's source. This is suitable for passing on - the command line as the input to an llvm tool. + Example: ``.exe`` (Windows), empty on Linux. ``%(line)``, ``%(line+)``, ``%(line-)`` - The number of the line where this variable is used, with an optional - integer offset. This can be used in tests with multiple RUN: lines, - which reference test file's line numbers. + The number of the line where this substitution is used, with an optional + integer offset. This can be used in tests with multiple RUN lines, which + reference test file's line numbers. -``$srcdir`` - The source directory from where the "``make check``" was run. -``objdir`` - The object directory that corresponds to the ``$srcdir``. +**Clang-specific substitutions:** -``subdir`` - A partial path from the ``test`` directory that contains the - sub-directory that contains the test source being executed. +``%clang`` + Invokes the Clang driver. -``srcroot`` - The root directory of the LLVM src tree. +``%clang_cpp`` + Invokes the Clang driver for C++. -``objroot`` - The root directory of the LLVM object tree. This could be the same as - the srcroot. +``%clang_cl`` + Invokes the CL-compatible Clang driver. -``path`` - The path to the directory that contains the test case source. This is - for locating any supporting files that are not generated by the test, - but used by the test. +``%clangxx`` + Invokes the G++-compatible Clang driver. -``tmp`` - The path to a temporary file name that could be used for this test case. - The file name won't conflict with other test cases. You can append to it - if you need multiple temporaries. This is useful as the destination of - some redirected output. - -``target_triplet`` (``%target_triplet``) - The target triplet that corresponds to the current host machine (the one - running the test cases). This should probably be called "host". +``%clang_cc1`` + Invokes the Clang frontend. -``link`` (``%link``) - This full link command used to link LLVM executables. This has all the - configured -I, -L and -l options. +``%itanium_abi_triple``, ``%ms_abi_triple`` + These substitutions can be used to get the current target triple adjusted to + the desired ABI. For example, if the test suite is running with the + ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to + ``i686-pc-mingw32``. This allows a test to run with a specific ABI without + constraining it to a specific triple. -``shlibext`` (``%shlibext``) - The suffix for the host platforms share library (dll) files. This - includes the period as the first character. +To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``. -To add more variables, two things need to be changed. First, add a line -in the ``test/Makefile`` that creates the ``site.exp`` file. This will -"set" the variable as a global in the site.exp file. Second, in the -``test/lib/llvm.exp`` file, in the substitute proc, add the variable -name to the list of "global" declarations at the beginning of the proc. -That's it, the variable can then be used in test scripts. Other Features -------------- -To make RUN line writing easier, there are several shell scripts located -in the ``llvm/test/Scripts`` directory. This directory is in the PATH -when running tests, so you can just call these scripts using their name. -For example: - -``ignore`` - This script runs its arguments and then always returns 0. This is useful - in cases where the test needs to cause a tool to generate an error (e.g. - to check the error output). However, any program in a pipeline that - returns a non-zero result will cause the test to fail. This script - overcomes that issue and nicely documents that the test case is - purposefully ignoring the result code of the tool +To make RUN line writing easier, there are several helper programs. These +helpers are in the PATH when running tests, so you can just call them using +their name. For example: + ``not`` - This script runs its arguments and then inverts the result code from it. - Zero result codes become 1. Non-zero result codes become 0. This is - useful to invert the result of a grep. For example "not grep X" means - succeed only if you don't find X in the input. + This program runs its arguments and then inverts the result code from it. + Zero result codes become 1. Non-zero result codes become 0. Sometimes it is necessary to mark a test case as "expected fail" or XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:`` @@ -640,7 +450,7 @@ line: ; XFAIL: darwin,sun -To make the output more useful, the ``llvm_runtest`` function wil scan +To make the output more useful, :program:`lit` will scan the lines of the test case for ones that contain a pattern that matches ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number that is related to the test case. The number after "PR" specifies the