<li><a href="#dgstructure">DejaGNU structure</a>
<ul>
<li><a href="#dgcustom">Writing new DejaGNU tests</a></li>
+ <li><a href="#FileCheck">The FileCheck utility</a></li>
<li><a href="#dgvars">Variables and substitutions</a></li>
<li><a href="#dgfeatures">Other features</a></li>
</ul>
<li><a href="#testsuitestructure">Test suite structure</a></li>
<li><a href="#testsuiterun">Running the test suite</a>
<ul>
- <li><a href="#testsuiteexternal">Configuring external tests</a></li>
+ <li><a href="#testsuiteexternal">Configuring External Tests</a></li>
<li><a href="#testsuitetests">Running different tests</a></li>
<li><a href="#testsuiteoutput">Generating test output</a></li>
<li><a href="#testsuitecustom">Writing custom tests for llvm-test</a></li>
<dd>Expect is required by DejaGNU.</dd>
<dt><a href="http://www.tcl.tk/software/tcltk/">tcl</a></dt>
<dd>Tcl is required by DejaGNU. </dd>
-
-<ul>
-<li><tt>./configure --with-f2c=$DIR</tt><br>
-This will specify a new <tt>$DIR</tt> for the above-described search
-process. This will only work if the binary, header, and library are in their
-respective subdirectories of <tt>$DIR</tt>.</li>
-
-<li><tt>./configure --with-f2c-bin=/binary/path --with-f2c-inc=/include/path
---with-f2c-lib=/lib/path</tt><br>
-This allows you to specify the F2C components separately. Note: if you choose
-this route, you MUST specify all three components, and you need to only specify
-<em>directories</em> where the files are located; do NOT include the
-filenames themselves on the <tt>configure</tt> line.</li>
-</ul></dd>
</dl>
-<p>Darwin (Mac OS X) developers can simplify the installation of Expect and tcl
-by using fink. <tt>fink install expect</tt> will install both. Alternatively,
-Darwinports users can use <tt>sudo port install expect</tt> to install Expect
-and tcl.</p>
-
</div>
<!--=========================================================================-->
<div class="doc_text">
-<p>Code fragments are small pieces of code that test a specific feature of LLVM
-or trigger a specific bug in LLVM. They are usually written in LLVM assembly
-language, but can be written in other languages if the test targets a particular
-language front end. These tests are driven by the DejaGNU testing framework,
-which is hidden behind a few simple makefiles.</p>
+<p>Code fragments are small pieces of code that test a specific
+feature of LLVM or trigger a specific bug in LLVM. They are usually
+written in LLVM assembly language, but can be written in other
+languages if the test targets a particular language front end (and the
+appropriate <tt>--with-llvmgcc</tt> options were used
+at <tt>configure</tt> time of the <tt>llvm</tt> module). These tests
+are driven by the DejaGNU testing framework, which is hidden behind a
+few simple makefiles.</p>
-<p>These code fragments are not complete programs. The code generated from them is
-never executed to determine correct behavior.</p>
+<p>These code fragments are not complete programs. The code generated
+from them is never executed to determine correct behavior.</p>
<p>These code fragment tests are located in the <tt>llvm/test</tt>
directory.</p>
% cd ..
% ./configure --with-llvmgccdir=$LLVM_GCC_DIR
</pre>
-<p>where <tt>$LLVM_GCC_DIR</tt> is the directory where you <em>installed</em>
-llvm-gcc, not it's src or obj dir.</p>
</div>
+<p>where <tt>$LLVM_GCC_DIR</tt> is the directory where
+you <em>installed</em> llvm-gcc, not it's src or obj
+dir. The <tt>--with-llvmgccdir</tt> option assumes that
+the <tt>llvm-gcc-4.2</tt> module was configured with
+<tt>--program-prefix=llvm-</tt>, and therefore that the C and C++
+compiler drivers are called <tt>llvm-gcc</tt> and <tt>llvm-g++</tt>
+respectively. If this is not the case,
+use <tt>--with-llvmgcc</tt>/<tt>--with-llvmgxx</tt> to specify each
+executable's location.</p>
+
<p>Then, run the entire test suite by running make in the <tt>test-suite</tt>
directory:</p>
</pre>
</div>
+<p>If your system includes GNU <tt>grep</tt>, make sure
+that <tt>GREP_OPTIONS</tt> is not set in your environment. Otherwise,
+you may get invalid results (both false positives and false
+negatives).</p>
+
</div>
<!-- _______________________________________________________________________ -->
-<div class="doc_subsection"><a name="dgvars">Variables and substitutions</a></div>
+<div class="doc_subsection"><a name="FileCheck">The FileCheck utility</a></div>
+<!-- _______________________________________________________________________ -->
+
+<div class="doc_text">
+
+<p>A powerful feature of the RUN: lines is that it allows any arbitrary commands
+ to be executed as part of the test harness. While standard (portable) unix
+ tools like 'grep' work fine on run lines, as you see above, there are a lot
+ of caveats due to interaction with Tcl syntax, and we want to make sure the
+ run lines are portable to a wide range of systems. Another major problem is
+ that grep is not very good at checking to verify that the output of a tools
+ contains a series of different output in a specific order. The FileCheck
+ tool was designed to help with these problems.</p>
+
+<p>FileCheck (whose basic command line arguments are described in <a
+ href="http://llvm.org/cmds/FileCheck.html">the FileCheck man page</a> is
+ designed to read a file to check from standard input, and the set of things
+ to verify from a file specified as a command line argument. A simple example
+ of using FileCheck from a RUN line looks like this:</p>
+
+<div class="doc_code">
+<pre>
+; RUN: llvm-as < %s | llc -march=x86-64 | <b>FileCheck %s</b>
+</pre>
+</div>
+
+<p>This syntax says to pipe the current file ("%s") into llvm-as, pipe that into
+llc, then pipe the output of llc into FileCheck. This means that FileCheck will
+be verifying its standard input (the llc output) against the filename argument
+specified (the original .ll file specified by "%s"). To see how this works,
+lets look at the rest of the .ll file (after the RUN line):</p>
+
+<div class="doc_code">
+<pre>
+define void @sub1(i32* %p, i32 %v) {
+entry:
+; <b>CHECK: sub1:</b>
+; <b>CHECK: subl</b>
+ %0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v)
+ ret void
+}
+
+define void @inc4(i64* %p) {
+entry:
+; <b>CHECK: inc4:</b>
+; <b>CHECK: incq</b>
+ %0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1)
+ ret void
+}
+</pre>
+</div>
+
+<p>Here you can see some "CHECK:" lines specified in comments. Now you can see
+how the file is piped into llvm-as, then llc, and the machine code output is
+what we are verifying. FileCheck checks the machine code output to verify that
+it matches what the "CHECK:" lines specify.</p>
+
+<p>The syntax of the CHECK: lines is very simple: they are fixed strings that
+must occur in order. FileCheck defaults to ignoring horizontal whitespace
+differences (e.g. a space is allowed to match a tab) but otherwise, the contents
+of the CHECK: line is required to match some thing in the test file exactly.</p>
+
+<p>One nice thing about FileCheck (compared to grep) is that it allows merging
+test cases together into logical groups. For example, because the test above
+is checking for the "sub1:" and "inc4:" labels, it will not match unless there
+is a "subl" in between those labels. If it existed somewhere else in the file,
+that would not count: "grep subl" matches if subl exists anywhere in the
+file.</p>
+
+</div>
+
+<!-- _______________________________________________________________________ -->
+<div class="doc_subsubsection"><a
+name="FileCheck-check-prefix">The FileCheck -check-prefix option</a></div>
+
+<div class="doc_text">
+
+<p>The FileCheck -check-prefix option allows multiple test configurations to be
+driven from one .ll file. This is useful in many circumstances, for example,
+testing different architectural variants with llc. Here's a simple example:</p>
+
+<div class="doc_code">
+<pre>
+; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \
+; RUN: | <b>FileCheck %s -check-prefix=X32</b>
+; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \
+; RUN: | <b>FileCheck %s -check-prefix=X64</b>
+
+define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind {
+ %tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1
+ ret <4 x i32> %tmp1
+; <b>X32:</b> pinsrd_1:
+; <b>X32:</b> pinsrd $1, 4(%esp), %xmm0
+
+; <b>X64:</b> pinsrd_1:
+; <b>X64:</b> pinsrd $1, %edi, %xmm0
+}
+</pre>
+</div>
+
+<p>In this case, we're testing that we get the expected code generation with
+both 32-bit and 64-bit code generation.</p>
+
+</div>
+
+<!-- _______________________________________________________________________ -->
+<div class="doc_subsubsection"><a
+name="FileCheck-CHECK-NEXT">The "CHECK-NEXT:" directive</a></div>
+
+<div class="doc_text">
+
+<p>Sometimes you want to match lines and would like to verify that matches
+happen on exactly consequtive lines with no other lines in between them. In
+this case, you can use CHECK: and CHECK-NEXT: directives to specify this. If
+you specified a custom check prefix, just use "<PREFIX>-NEXT:". For
+example, something like this works as you'd expect:</p>
+
+<div class="doc_code">
+<pre>
+define void @t2(<2 x double>* %r, <2 x double>* %A, double %B) {
+ %tmp3 = load <2 x double>* %A, align 16
+ %tmp7 = insertelement <2 x double> undef, double %B, i32 0
+ %tmp9 = shufflevector <2 x double> %tmp3,
+ <2 x double> %tmp7,
+ <2 x i32> < i32 0, i32 2 >
+ store <2 x double> %tmp9, <2 x double>* %r, align 16
+ ret void
+
+; <b>CHECK:</b> t2:
+; <b>CHECK:</b> movl 8(%esp), %eax
+; <b>CHECK-NEXT:</b> movapd (%eax), %xmm0
+; <b>CHECK-NEXT:</b> movhpd 12(%esp), %xmm0
+; <b>CHECK-NEXT:</b> movl 4(%esp), %eax
+; <b>CHECK-NEXT:</b> movapd %xmm0, (%eax)
+; <b>CHECK-NEXT:</b> ret
+}
+</pre>
+</div>
+
+<p>CHECK-NEXT: directives reject the input unless there is exactly one newline
+between it an the previous directive. A CHECK-NEXT cannot be the first
+directive in a file.</p>
+
+</div>
+
+<!-- _______________________________________________________________________ -->
+<div class="doc_subsubsection"><a
+name="FileCheck-CHECK-NOT">The "CHECK-NOT:" directive</a></div>
+
+<div class="doc_text">
+
+<p>The CHECK-NOT: directive is used to verify that a string doesn't occur
+between two matches (or the first match and the beginning of the file). For
+example, to verify that a load is removed by a transformation, a test like this
+can be used:</p>
+
+<div class="doc_code">
+<pre>
+define i8 @coerce_offset0(i32 %V, i32* %P) {
+ store i32 %V, i32* %P
+
+ %P2 = bitcast i32* %P to i8*
+ %P3 = getelementptr i8* %P2, i32 2
+
+ %A = load i8* %P3
+ ret i8 %A
+; <b>CHECK:</b> @coerce_offset0
+; <b>CHECK-NOT:</b> load
+; <b>CHECK:</b> ret i8
+}
+</pre>
+</div>
+
+</div>
+
+<!-- _______________________________________________________________________ -->
+<div class="doc_subsubsection"><a
+name="FileCheck-Matching">FileCheck Pattern Matching Syntax</a></div>
+
+<div class="doc_text">
+
+<p>The CHECK: and CHECK-NOT: directives both take a pattern to match. For most
+uses of FileCheck, fixed string matching is perfectly sufficient. For some
+things, a more flexible form of matching is desired. To support this, FileCheck
+allows you to specify regular expressions in matching strings, surrounded by
+double braces: <b>{{yourregex}}</b>. Because we want to use fixed string
+matching for a majority of what we do, FileCheck has been designed to support
+mixing and matching fixed string matching with regular expressions. This allows
+you to write things like this:</p>
+
+<div class="doc_code">
+<pre>
+; CHECK: movhpd <b>{{[0-9]+}}</b>(%esp), <b>{{%xmm[0-7]}}</b>
+</pre>
+</div>
+
+<p>In this case, any offset from the ESP register will be allowed, and any xmm
+register will be allowed.</p>
+
+<p>Because regular expressions are enclosed with double braces, they are
+visually distinct, and you don't need to use escape characters within the double
+braces like you would in C. In the rare case that you want to match double
+braces explicitly from the input, you can use something ugly like
+<b>{{[{][{]}}</b> as your pattern.</p>
+
+</div>
+
+<!-- _______________________________________________________________________ -->
+<div class="doc_subsubsection"><a
+name="FileCheck-Variables">FileCheck Variables</a></div>
+
+<div class="doc_text">
+
+<p>It is often useful to match a pattern and then verify that it occurs again
+later in the file. For codegen tests, this can be useful to allow any register,
+but verify that that register is used consistently later. To do this, FileCheck
+allows named variables to be defined and substituted into patterns. Here is a
+simple example:</p>
+
+<div class="doc_code">
+<pre>
+; CHECK: test5:
+; CHECK: notw <b>[[REGISTER:%[a-z]+]]</b>
+; CHECK: andw {{.*}}<b>[[REGISTER]]</b>
+</pre>
+</div>
+
+<p>The first check line matches a regex (<tt>%[a-z]+</tt>) and captures it into
+the variables "REGISTER". The second line verifies that whatever is in REGISTER
+occurs later in the file after an "andw". FileCheck variable references are
+always contained in <tt>[[ ]]</tt> pairs, are named, and their names can be
+formed with the regex "<tt>[a-zA-Z][a-zA-Z0-9]*</tt>". If a colon follows the
+name, then it is a definition of the variable, if not, it is a use.</p>
+
+<p>FileCheck variables can be defined multiple times, and uses always get the
+latest value. Note that variables are all read at the start of a "CHECK" line
+and are all defined at the end. This means that if you have something like
+"<tt>CHECK: [[XYZ:.*]]x[[XYZ]]</tt>" that the check line will read the previous
+value of the XYZ variable and define a new one after the match is performed. If
+you need to do something like this you can probably take advantage of the fact
+that FileCheck is not actually line-oriented when it matches, this allows you to
+define two separate CHECK lines that match on the same line.
+</p>
+
+
+
+</div>
+
+<!-- _______________________________________________________________________ -->
+<div class="doc_subsection"><a name="dgvars">Variables and
+substitutions</a></div>
<!-- _______________________________________________________________________ -->
<div class="doc_text">
<p>With a RUN line there are a number of substitutions that are permitted. In
<dd>The target triplet that corresponds to the current host machine (the one
running the test cases). This should probably be called "host".<dd>
- <dt><b>prcontext</b> (%prcontext)</dt>
- <dd>Path to the prcontext tcl script that prints some context around a
- line that matches a pattern. This isn't strictly necessary as the test suite
- is run with its PATH altered to include the test/Scripts directory where
- the prcontext script is located. Note that this script is similar to
- <tt>grep -C</tt> but you should use the <tt>prcontext</tt> script because
- not all platforms support <tt>grep -C</tt>.</dd>
-
<dt><b>llvmgcc</b> (%llvmgcc)</dt>
<dd>The full path to the <tt>llvm-gcc</tt> executable as specified in the
configured LLVM environment</dd>
<p>The External directory contains Makefiles for building code that is external
to (i.e., not distributed with) LLVM. The most prominent members of this
directory are the SPEC 95 and SPEC 2000 benchmark suites. The <tt>External</tt>
-directory does not contain these actual tests,but only the Makefiles that know
+directory does not contain these actual tests, but only the Makefiles that know
how to properly compile these programs from somewhere else. The presence and
location of these external programs is configured by the llvm-test
<tt>configure</tt> script.</p></li>
% svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
</pre>
</div>
-
<p>This will get the test suite into <tt>llvm/projects/test-suite</tt>.</p>
+ <p>This will get the test suite into <tt>llvm/projects/test-suite</tt>.</p>
</li>
- <li><p>Configure llvm from the top level of each build tree (LLVM object directory tree)
- in which you want to run the test suite, just as you do before building LLVM.</p>
- <p>When running configure, you must either: (1) have <tt>llvm-gcc</tt>
- in your path, or (2) specify the directory where <tt>llvm-gcc</tt> is
- installed using <tt>--with-llvmgccdir=$LLVM_GCC_DIR</tt>.</p>
- <p>This step tells the configure machinery that the test suite
- is now available so it can be configured for your build tree:</p>
+ <li><p>Configure and build <tt>llvm</tt>.</p></li>
+ <li><p>Configure and build <tt>llvm-gcc</tt>.</p></li>
+ <li><p>Install <tt>llvm-gcc</tt> somewhere.</p></li>
+ <li><p><em>Re-configure</em> <tt>llvm</tt> from the top level of
+ each build tree (LLVM object directory tree) in which you want
+ to run the test suite, just as you do before building LLVM.</p>
+ <p>During the <em>re-configuration</em>, you must either: (1)
+ have <tt>llvm-gcc</tt> you just built in your path, or (2)
+ specify the directory where your just-built <tt>llvm-gcc</tt> is
+ installed using <tt>--with-llvmgccdir=$LLVM_GCC_DIR</tt>.</p>
+ <p>You must also tell the configure machinery that the test suite
+ is available so it can be configured for your build tree:</p>
<div class="doc_code">
<pre>
% cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure [--with-llvmgccdir=$LLVM_GCC_DIR]
have the suite checked out and configured, you don't need to do it again (unless
the test code or configure script changes).</p>
+</div>
+
<!-- _______________________________________________________________________ -->
<div class="doc_subsection">
-<a name="testsuiteexternal">Configuring external tests</a></div>
+<a name="testsuiteexternal">Configuring External Tests</a></div>
<!-- _______________________________________________________________________ -->
<div class="doc_text">
-<p>Note, when configuring the <tt>test-suite</tt> module, you might want to
-specify the following configuration option:</p>
+<p>In order to run the External tests in the <tt>test-suite</tt>
+ module, you must specify <i>--with-externals</i>. This
+ must be done during the <em>re-configuration</em> step (see above),
+ and the <tt>llvm</tt> re-configuration must recognize the
+ previously-built <tt>llvm-gcc</tt>. If any of these is missing or
+ neglected, the External tests won't work.</p>
<dl>
<dt><i>--with-externals</i></dt>
<dt><i>--with-externals=<<tt>directory</tt>></i></dt>
<a name="testsuitetests">Running different tests</a></div>
<!-- _______________________________________________________________________ -->
<div class="doc_text">
-<p>In addition to the regular "whole program" tests, the <tt>test-suite</tt>
+<p>In addition to the regular "whole program" tests, the <tt>test-suite</tt>
module also provides a mechanism for compiling the programs in different ways.
If the variable TEST is defined on the <tt>gmake</tt> command line, the test system will
include a Makefile named <tt>TEST.<value of TEST variable>.Makefile</tt>.
projects / oota-llvm.git / blobdiff