X-Git-Url: http://plrg.eecs.uci.edu/git/?a=blobdiff_plain;f=docs%2FTestingGuide.rst;h=3463156495a9c85588789a942aaf68c0b3c7f302;hb=ec07ff56b988d537fbcd4b16ff65e31c688bf64e;hp=c9a35cd746348cbd42616a8914a2dc1ffa89c231;hpb=88429cfb39df6f62e521968ba0b2780f38a261d9;p=oota-llvm.git diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst index c9a35cd7463..3463156495a 100644 --- a/docs/TestingGuide.rst +++ b/docs/TestingGuide.rst @@ -22,7 +22,7 @@ Requirements ============ In order to use the LLVM testing infrastructure, you will need all of the -software required to build LLVM, as well as `Python `_ 2.5 or +software required to build LLVM, as well as `Python `_ 2.7 or later. LLVM testing infrastructure organization @@ -240,6 +240,58 @@ The recommended way to examine output to figure out if the test passes is using the :doc:`FileCheck tool `. *[The usage of grep in RUN lines is deprecated - please do not send or commit patches that use it.]* +Extra files +----------- + +If your test requires extra files besides the file containing the ``RUN:`` +lines, the idiomatic place to put them is in a subdirectory ``Inputs``. +You can then refer to the extra files as ``%S/Inputs/foo.bar``. + +For example, consider ``test/Linker/ident.ll``. The directory structure is +as follows:: + + test/ + Linker/ + ident.ll + Inputs/ + ident.a.ll + ident.b.ll + +For convenience, these are the contents: + +.. code-block:: llvm + + ;;;;; ident.ll: + + ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s + + ; Verify that multiple input llvm.ident metadata are linked together. + + ; CHECK-DAG: !llvm.ident = !{!0, !1, !2} + ; CHECK-DAG: "Compiler V1" + ; CHECK-DAG: "Compiler V2" + ; CHECK-DAG: "Compiler V3" + + ;;;;; Inputs/ident.a.ll: + + !llvm.ident = !{!0, !1} + !0 = metadata !{metadata !"Compiler V1"} + !1 = metadata !{metadata !"Compiler V2"} + + ;;;;; Inputs/ident.b.ll: + + !llvm.ident = !{!0} + !0 = metadata !{metadata !"Compiler V3"} + +For symmetry reasons, ``ident.ll`` is just a dummy file that doesn't +actually participate in the test besides holding the ``RUN:`` lines. + +.. note:: + + Some existing tests use ``RUN: true`` in extra files instead of just + putting the extra files in an ``Inputs/`` directory. This pattern is + deprecated. + Fragile tests ------------- @@ -304,8 +356,7 @@ For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is: .. code-block:: python config.suffixes = ['.ll', '.c', '.cpp', '.test'] - targets = set(config.root.targets_to_build.split()) - if not 'ARM' in targets: + if not 'ARM' in config.root.targets: config.unsupported = True Other platform-specific tests are those that depend on a specific feature @@ -337,87 +388,100 @@ triple, test with the specific FileCheck and put it into the specific directory that will filter out all other architectures. -Variables and substitutions ---------------------------- +Substitutions +------------- -With a RUN line there are a number of substitutions that are permitted. -To make a substitution just write the variable's name preceded by a ``$``. -Additionally, for compatibility reasons with previous versions of the -test library, certain names can be accessed with an alternate syntax: a -% prefix. These alternates are deprecated and may go away in a future -version. +Besides replacing LLVM tool names the following substitutions are performed in +RUN lines: -Here are the available variable names. The alternate syntax is listed in -parentheses. +``%%`` + Replaced by a single ``%``. This allows escaping other substitutions. -``$test`` (``%s``) - The full path to the test case's source. This is suitable for passing on - the command line as the input to an LLVM tool. +``%s`` + File path to the test case's source. This is suitable for passing on the + command line as the input to an LLVM tool. -``%(line)``, ``%(line+)``, ``%(line-)`` - The number of the line where this variable is used, with an optional - integer offset. This can be used in tests with multiple RUN lines, - which reference test file's line numbers. + Example: ``/home/user/llvm/test/MC/ELF/foo_test.s`` -``$srcdir`` - The source directory from where the ``make check`` was run. +``%S`` + Directory path to the test case's source. -``objdir`` - The object directory that corresponds to the ``$srcdir``. + Example: ``/home/user/llvm/test/MC/ELF`` -``subdir`` - A partial path from the ``test`` directory that contains the - sub-directory that contains the test source being executed. +``%t`` + File path to a temporary file name that could be used for this test case. + The file name won't conflict with other test cases. You can append to it + if you need multiple temporaries. This is useful as the destination of + some redirected output. -``srcroot`` - The root directory of the LLVM src tree. + Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp`` -``objroot`` - The root directory of the LLVM object tree. This could be the same as - the srcroot. +``%T`` + Directory of ``%t``. -``path`` - The path to the directory that contains the test case source. This is - for locating any supporting files that are not generated by the test, - but used by the test. + Example: ``/home/user/llvm.build/test/MC/ELF/Output`` -``tmp`` - The path to a temporary file name that could be used for this test case. - The file name won't conflict with other test cases. You can append to it - if you need multiple temporaries. This is useful as the destination of - some redirected output. +``%{pathsep}`` + + Expands to the path separator, i.e. ``:`` (or ``;`` on Windows). -``target_triplet`` (``%target_triplet``) - The target triplet that corresponds to the current host machine (the one - running the test cases). This should probably be called "host". -``link`` (``%link``) - This full link command used to link LLVM executables. This has all the - configured ``-I``, ``-L`` and ``-l`` options. +**LLVM-specific substitutions:** -``shlibext`` (``%shlibext``) - The suffix for the host platforms shared library (DLL) files. This - includes the period as the first character. +``%shlibext`` + The suffix for the host platforms shared library files. This includes the + period as the first character. + + Example: ``.so`` (Linux), ``.dylib`` (OS X), ``.dll`` (Windows) + +``%exeext`` + The suffix for the host platforms executable files. This includes the + period as the first character. + + Example: ``.exe`` (Windows), empty on Linux. + +``%(line)``, ``%(line+)``, ``%(line-)`` + The number of the line where this substitution is used, with an optional + integer offset. This can be used in tests with multiple RUN lines, which + reference test file's line numbers. + + +**Clang-specific substitutions:** + +``%clang`` + Invokes the Clang driver. + +``%clang_cpp`` + Invokes the Clang driver for C++. + +``%clang_cl`` + Invokes the CL-compatible Clang driver. + +``%clangxx`` + Invokes the G++-compatible Clang driver. + +``%clang_cc1`` + Invokes the Clang frontend. + +``%itanium_abi_triple``, ``%ms_abi_triple`` + These substitutions can be used to get the current target triple adjusted to + the desired ABI. For example, if the test suite is running with the + ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to + ``i686-pc-mingw32``. This allows a test to run with a specific ABI without + constraining it to a specific triple. + +To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``. -To add more variables, look at ``test/lit.cfg``. Other Features -------------- -To make RUN line writing easier, there are several helper scripts and programs -in the ``llvm/test/Scripts`` directory. This directory is in the PATH -when running tests, so you can just call these scripts using their name. -For example: - -``ignore`` - This script runs its arguments and then always returns 0. This is useful - in cases where the test needs to cause a tool to generate an error (e.g. - to check the error output). However, any program in a pipeline that - returns a non-zero result will cause the test to fail. This script - overcomes that issue and nicely documents that the test case is - purposefully ignoring the result code of the tool +To make RUN line writing easier, there are several helper programs. These +helpers are in the PATH when running tests, so you can just call them using +their name. For example: + ``not`` - This script runs its arguments and then inverts the result code from it. + This program runs its arguments and then inverts the result code from it. Zero result codes become 1. Non-zero result codes become 0. Sometimes it is necessary to mark a test case as "expected fail" or