From 70360e67624b5897490de29b51b1b616afcb8f5e Mon Sep 17 00:00:00 2001 From: Daniel Dunbar Date: Fri, 16 Aug 2013 23:30:23 +0000 Subject: [PATCH] [lit] Rewrite TODO list, and elaborate on some things. - If anyone is interested in lit's feature set, I'd appreciate any comments on the elaborated items. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188590 91177308-0d34-0410-b5e6-96231b3b80d8 --- utils/lit/TODO | 141 ++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 129 insertions(+), 12 deletions(-) diff --git a/utils/lit/TODO b/utils/lit/TODO index e209fada355..419056a78f4 100644 --- a/utils/lit/TODO +++ b/utils/lit/TODO @@ -1,22 +1,139 @@ - - Move temp directory name into local test config. +================ + lit TODO Items +================ - - Add --show-unsupported, don't show by default? +Infrastructure +============== - - Optionally use multiprocessing. +1. Change to always load suites, then resolve command line arguments? - - Support valgrind in all configs, and LLVM style valgrind. + Currently we expect each input argument to be a path on disk; we do a + recursive search to find the test suite for each item, but then we only do a + local search based at the input path to find tests. Additionally, for any path + that matches a file on disk we explicitly construct a test instance (bypassing + the formats on discovery implementation). - - Support a timeout / ulimit. + This has a couple problems: - - Create an explicit test suite object (instead of using the top-level - TestingConfig object). + * The test format doesn't have control over the test instances that result + from file paths. - - Allow 'lit' driver to cooperate with test suites to add options (or at least - sanitize accepted params). + * It isn't possible to specify virtual tests as inputs. For example, it is not + possible to specify an individual subtest to run with the googletest format. - - Consider move to identifying all tests by path-to-test-suite and then path to + * The test format doesn't have full control over the discovery of tests in + subdirectories. + + Instead, we should move to a model whereby first all of the input specifiers + are resolved to test suites, and then the resolution of the input specifier is + delegated to each test suite. This could take a couple forms: + + * We could resolve to test suites, then fully load each test suite, then have + a fixed process to map input specifiers to tests in the test suite + (presumably based on path-in-suite derivations). This has the benefit of + being consistent across all test formats, but the downside of requiring + loading the entire test suite. + + * We could delegate all of the resolution of specifiers to the test + suite. This would allow formats that anticipate large test suites to manage + their own resolution for better performance. We could provide a default + resolution strategy that was similar to what we do now (start at subpaths + for directories, but allow the test format control over what happens for + individual tests). + +2. Consider move to identifying all tests by path-to-test-suite and then path to subtest, and don't use test suite names. - - Consider move to change workflow to always load suites, then resolve command - line arguments. + Currently the test suite name is presented as part of test names, but it has + no other useful function, and it is something that has to be skipped over to + cut-and-paste a name to subsequently use to rerun a test. If we just + represented each test suite by the path to its suite, then it would allow more + easy cut-and-paste of the test output lines. This has the downside that the + lines might get rather long. + +3. Allow 'lit' driver to cooperate with test formats and suites to add options + (or at least sanitize accepted params). + + We have started to use the --params method more and more extensively, and it is + cumbersome and error prone. Additionally, there are currently various options + ``lit`` honors that should more correctly be specified as belonging to the + ShTest test format. + + It would be really nice if we could allow test formats and test suites to add + their own options to be parsed. The difficulty here, of course, is that we + don't know what test formats or test suites are in use until we have parsed the + input specifiers. For test formats we could ostensibly require all the possible + formats to be registered in order to have options, but for test suites we would + certainly have to load the suite before we can query it for what options it + understands. + + That leaves us with the following options: + + * Currently we could almost get away with parsing the input specifiers without + having done option parsing first (the exception is ``--config-prefix``) but + that isn't a very extensible design. + + * We could make a distinction in the command line syntax for test format and + test suite options. For example, we could require something like:: + + lit -j 1 -sv input-specifier -- --some-format-option + + which would be relatively easy to implement with optparser (I think). + + * We could allow fully interspersed arguments by first extracting the options + lit knows about and parsing them, then dispatching the remainder to the + formats. This seems the most convenient for users, who are unlikely to care + about (or even be aware of) the distinction between the generic lit + infrastructure and format or suite specific options. + +4. Eliminate duplicate execution models for ShTest tests. + + Currently, the ShTest format uses tests written with shell-script like syntax, + and executes them in one of two ways. The first way is by converting them into + a bash script and literally executing externally them using bash. The second + way is through the use of an internal shell parser and shell execution code + (built on the subprocess module). The external execution mode is used on most + Unix systems that have bash, the internal execution mode is used on Windows. + + Having two ways to do the same thing is error prone and leads to unnecessary + complexity in the testing environment. Additionally, because the mode that + converts scripts to bash doesn't try and validate the syntax, it is possible + to write tests that use bash shell features unsupported by the internal + shell. Such tests won't work on Windows but this may not be obvious to the + developer writing the test. + + Another limitation is that when executing the scripts externally, the ShTest + format has no idea which commands fail, or what output comes from which + commands, so this limits how convenient the output of ShTest failures can be + and limits other features (for example, knowing what temporary files were + written). + + We should eliminate having two ways of executing the same tests to reduce + platform differences and make it easier to develop new features in the ShTest + module. This is currently blocked on: + + * The external execution mode is faster in some situations, because it avoids + being bottlenecked on the GIL. We could fix this by moving to a good + multiprocessing model. + + * Some tests in LLVM/Clang are explicitly disabled with the internal shell + (because they use features specific to bash). We would need to rewrite these + tests, or add additional features to the internal shell handling to allow + them to pass. + + +Miscellaneous +============= + +* Move temp directory name into local test config. + +* Add --show-unsupported, don't show by default? + +* Optionally use multiprocessing. + +* Support valgrind in all configs, and LLVM style valgrind. + +* Support a timeout / ulimit. +* Create an explicit test suite object (instead of using the top-level + TestingConfig object). -- 2.34.1