X-Git-Url: http://plrg.eecs.uci.edu/git/?a=blobdiff_plain;f=docs%2FTestingGuide.rst;h=134ddd88c87d5a63fc8d4c6810cb3d51287fbf59;hb=fbbab8b9598762b23d1cc870d4a7a1cba4158792;hp=3cfbb219e6509c8d488b789fd0f2cbddc5827a1f;hpb=f0126ea0a1a574b6b45c93a1a56a24d0ce30de87;p=oota-llvm.git diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst index 3cfbb219e65..134ddd88c87 100644 --- a/docs/TestingGuide.rst +++ b/docs/TestingGuide.rst @@ -21,9 +21,9 @@ tests. Requirements ============ -In order to use the LLVM testing infrastructure, you will need all of -the software required to build LLVM, as well as -`Python `_ 2.4 or later. +In order to use the LLVM testing infrastructure, you will need all of the +software required to build LLVM, as well as `Python `_ 2.7 or +later. LLVM testing infrastructure organization ======================================== @@ -120,12 +120,14 @@ can run the LLVM and Clang tests simultaneously using: % make check-all -To run the tests with Valgrind (Memcheck by default), just append -``VG=1`` to the commands above, e.g.: +To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make +variable to pass the required options to lit. For example, you can use: .. code-block:: bash - % make check VG=1 + % make check LIT_ARGS="-v --vg --vg-leak" + +to enable testing with valgrind and with leak checking enabled. To run individual tests or subsets of tests, you can use the ``llvm-lit`` script which is built as part of LLVM. For example, to run the @@ -238,6 +240,62 @@ The recommended way to examine output to figure out if the test passes is using the :doc:`FileCheck tool `. *[The usage of grep in RUN lines is deprecated - please do not send or commit patches that use it.]* +Put related tests into a single file rather than having a separate file per +test. Check if there are files already covering your feature and consider +adding your code there instead of creating a new file. + +Extra files +----------- + +If your test requires extra files besides the file containing the ``RUN:`` +lines, the idiomatic place to put them is in a subdirectory ``Inputs``. +You can then refer to the extra files as ``%S/Inputs/foo.bar``. + +For example, consider ``test/Linker/ident.ll``. The directory structure is +as follows:: + + test/ + Linker/ + ident.ll + Inputs/ + ident.a.ll + ident.b.ll + +For convenience, these are the contents: + +.. code-block:: llvm + + ;;;;; ident.ll: + + ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s + + ; Verify that multiple input llvm.ident metadata are linked together. + + ; CHECK-DAG: !llvm.ident = !{!0, !1, !2} + ; CHECK-DAG: "Compiler V1" + ; CHECK-DAG: "Compiler V2" + ; CHECK-DAG: "Compiler V3" + + ;;;;; Inputs/ident.a.ll: + + !llvm.ident = !{!0, !1} + !0 = metadata !{metadata !"Compiler V1"} + !1 = metadata !{metadata !"Compiler V2"} + + ;;;;; Inputs/ident.b.ll: + + !llvm.ident = !{!0} + !0 = metadata !{metadata !"Compiler V3"} + +For symmetry reasons, ``ident.ll`` is just a dummy file that doesn't +actually participate in the test besides holding the ``RUN:`` lines. + +.. note:: + + Some existing tests use ``RUN: true`` in extra files instead of just + putting the extra files in an ``Inputs/`` directory. This pattern is + deprecated. + Fragile tests ------------- @@ -302,8 +360,7 @@ For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is: .. code-block:: python config.suffixes = ['.ll', '.c', '.cpp', '.test'] - targets = set(config.root.targets_to_build.split()) - if not 'ARM' in targets: + if not 'ARM' in config.root.targets: config.unsupported = True Other platform-specific tests are those that depend on a specific feature @@ -335,87 +392,119 @@ triple, test with the specific FileCheck and put it into the specific directory that will filter out all other architectures. -Variables and substitutions ---------------------------- +Substitutions +------------- + +Besides replacing LLVM tool names the following substitutions are performed in +RUN lines: + +``%%`` + Replaced by a single ``%``. This allows escaping other substitutions. + +``%s`` + File path to the test case's source. This is suitable for passing on the + command line as the input to an LLVM tool. -With a RUN line there are a number of substitutions that are permitted. -To make a substitution just write the variable's name preceded by a ``$``. -Additionally, for compatibility reasons with previous versions of the -test library, certain names can be accessed with an alternate syntax: a -% prefix. These alternates are deprecated and may go away in a future -version. + Example: ``/home/user/llvm/test/MC/ELF/foo_test.s`` -Here are the available variable names. The alternate syntax is listed in -parentheses. +``%S`` + Directory path to the test case's source. -``$test`` (``%s``) - The full path to the test case's source. This is suitable for passing on - the command line as the input to an LLVM tool. + Example: ``/home/user/llvm/test/MC/ELF`` + +``%t`` + File path to a temporary file name that could be used for this test case. + The file name won't conflict with other test cases. You can append to it + if you need multiple temporaries. This is useful as the destination of + some redirected output. + + Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp`` + +``%T`` + Directory of ``%t``. + + Example: ``/home/user/llvm.build/test/MC/ELF/Output`` + +``%{pathsep}`` + + Expands to the path separator, i.e. ``:`` (or ``;`` on Windows). + + +**LLVM-specific substitutions:** + +``%shlibext`` + The suffix for the host platforms shared library files. This includes the + period as the first character. + + Example: ``.so`` (Linux), ``.dylib`` (OS X), ``.dll`` (Windows) + +``%exeext`` + The suffix for the host platforms executable files. This includes the + period as the first character. + + Example: ``.exe`` (Windows), empty on Linux. ``%(line)``, ``%(line+)``, ``%(line-)`` - The number of the line where this variable is used, with an optional - integer offset. This can be used in tests with multiple RUN lines, - which reference test file's line numbers. + The number of the line where this substitution is used, with an optional + integer offset. This can be used in tests with multiple RUN lines, which + reference test file's line numbers. -``$srcdir`` - The source directory from where the ``make check`` was run. -``objdir`` - The object directory that corresponds to the ``$srcdir``. +**Clang-specific substitutions:** -``subdir`` - A partial path from the ``test`` directory that contains the - sub-directory that contains the test source being executed. +``%clang`` + Invokes the Clang driver. -``srcroot`` - The root directory of the LLVM src tree. +``%clang_cpp`` + Invokes the Clang driver for C++. -``objroot`` - The root directory of the LLVM object tree. This could be the same as - the srcroot. +``%clang_cl`` + Invokes the CL-compatible Clang driver. -``path`` - The path to the directory that contains the test case source. This is - for locating any supporting files that are not generated by the test, - but used by the test. +``%clangxx`` + Invokes the G++-compatible Clang driver. -``tmp`` - The path to a temporary file name that could be used for this test case. - The file name won't conflict with other test cases. You can append to it - if you need multiple temporaries. This is useful as the destination of - some redirected output. +``%clang_cc1`` + Invokes the Clang frontend. + +``%itanium_abi_triple``, ``%ms_abi_triple`` + These substitutions can be used to get the current target triple adjusted to + the desired ABI. For example, if the test suite is running with the + ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to + ``i686-pc-mingw32``. This allows a test to run with a specific ABI without + constraining it to a specific triple. -``target_triplet`` (``%target_triplet``) - The target triplet that corresponds to the current host machine (the one - running the test cases). This should probably be called "host". +To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``. -``link`` (``%link``) - This full link command used to link LLVM executables. This has all the - configured ``-I``, ``-L`` and ``-l`` options. -``shlibext`` (``%shlibext``) - The suffix for the host platforms shared library (DLL) files. This - includes the period as the first character. +Options +------- + +The llvm lit configuration allows to customize some things with user options: + +``llc``, ``opt``, ... + Substitute the respective llvm tool name with a custom command line. This + allows to specify custom paths and default arguments for these tools. + Example: + + % llvm-lit "-Dllc=llc -verify-machineinstrs" + +``run_long_tests`` + Enable the execution of long running tests. + +``llvm_site_config`` + Load the specified lit configuration instead of the default one. -To add more variables, look at ``test/lit.cfg``. Other Features -------------- -To make RUN line writing easier, there are several helper scripts and programs -in the ``llvm/test/Scripts`` directory. This directory is in the PATH -when running tests, so you can just call these scripts using their name. -For example: - -``ignore`` - This script runs its arguments and then always returns 0. This is useful - in cases where the test needs to cause a tool to generate an error (e.g. - to check the error output). However, any program in a pipeline that - returns a non-zero result will cause the test to fail. This script - overcomes that issue and nicely documents that the test case is - purposefully ignoring the result code of the tool +To make RUN line writing easier, there are several helper programs. These +helpers are in the PATH when running tests, so you can just call them using +their name. For example: + ``not`` - This script runs its arguments and then inverts the result code from it. + This program runs its arguments and then inverts the result code from it. Zero result codes become 1. Non-zero result codes become 0. Sometimes it is necessary to mark a test case as "expected fail" or