Navigation
- index
- modules |
- next |
- previous |
- 中文 |
- 中英文对照 |
- Python»
- 2.7.8 Documentation »
- The Python Standard Library »
- 25. Development Tools »
doctest模块搜索看上去像交互式Python会话的文本，然后执行这些会话，以验证它们像它显示的那样工作。有几种使用doctest的常用方式：
这是一个小而完整的例子模块：
"""
This is the "example" module.
The example module supplies one function, factorial(). For example,
>>> factorial(5)
120
"""
def factorial(n):
"""Return the factorial of n, an exact integer >= 0.
If the result is small enough to fit in an int, return an int.
Else return a long.
>>> [factorial(n) for n in range(6)]
[1, 1, 2, 6, 24, 120]
>>> [factorial(long(n)) for n in range(6)]
[1, 1, 2, 6, 24, 120]
>>> factorial(30)
265252859812191058636308480000000L
>>> factorial(30L)
265252859812191058636308480000000L
>>> factorial(-1)
Traceback (most recent call last):
...
ValueError: n must be >= 0
Factorials of floats are OK, but the float must be an exact integer:
>>> factorial(30.1)
Traceback (most recent call last):
...
ValueError: n must be exact integer
>>> factorial(30.0)
265252859812191058636308480000000L
It must also not be ridiculously large:
>>> factorial(1e100)
Traceback (most recent call last):
...
OverflowError: n too large
"""
import math
if not n >= 0:
raise ValueError("n must be >= 0")
if math.floor(n) != n:
raise ValueError("n must be exact integer")
if n+1 == n: # catch a value like 1e300
raise OverflowError("n too large")
result = 1
factor = 2
while factor <= n:
result *= factor
factor += 1
return result
if __name__ == "__main__":
import doctest
doctest.testmod()
如果直接从命令行运行example.py，doctest发挥它魔法般的作用：
$ python example.py
$
没有任何输出！这是正常的，它意味着所有的例子都工作。给脚本传递-v参数，doctest就会打印详细的日志，包括它尝试了什么，以及结尾处的摘要。
$ python example.py -v
Trying:
factorial(5)
Expecting:
120
ok
Trying:
[factorial(n) for n in range(6)]
Expecting:
[1, 1, 2, 6, 24, 120]
ok
Trying:
[factorial(long(n)) for n in range(6)]
Expecting:
[1, 1, 2, 6, 24, 120]
ok
等等，最后的是：
Trying:
factorial(1e100)
Expecting:
Traceback (most recent call last):
...
OverflowError: n too large
ok
2 items passed all tests:
1 tests in __main__
8 tests in __main__.factorial
9 tests in 2 items.
9 passed and 0 failed.
Test passed.
$
这就是所有你需要知道的，开始大量使用doctest吧！投入吧。下面的章节提供了完整的细节。注意，在标准Python测试集和库中包括了大量的doctest的例子。特别有用的例子可以在标准测试文件Lib/test/test_doctest.py中找到。
开始使用doctest的最简单的方式（如果你继续，这不是必需的方式），就是在每个模块M的结尾加上：
if __name__ == "__main__":
import doctest
doctest.testmod()
doctest就会检查模块M中的docstring。
把模块当脚本运行就会导致docstring中的例子得到执行和验证。
python M.py
这不会显示任何东西，除非例子失败，失败的例子及其原因会被打印到标准输出，输出的最后一行是***Test Failed*** N failures.，N是失败的例子的数目。
运行时加上-v开关：
python M.py -v
会打印出详细的报告，以及一个汇总摘要。
给testmod()传递verbose=True 可以强制冗余模式，传递verbose=False可以禁止它。不管哪种情况testmod()不会检查sys.argv，传-v也就不会有作用。
Python 2.6之后，有一个从命令行运行testmod()的快捷方式。可以让Python解释器直接从标准库中运行doctest模块，并在命令行中传递要测试的模块的名字：
python -m doctest -v example.py
这会将example.py以独立模块的方式导入，并在其上运行testmod()。注意，如果这个文件是包的一部分且会从包中导入其它的子模块，这个方法就不会正确工作。
doctest另一个简单的应用就是测试文本文件中的交互式例子。这由testfile()函数完成：
import doctest
doctest.testfile("example.txt")
这个简短的脚本执行并验证example.txt文件中的任何交互式Python的例子。文档的内容被认为是个单一的巨大的docstring；文件不需要包含Python程序!！例如，example.txt可以包含：
The ``example`` module
======================
Using ``factorial``
-------------------
This is an example text file in reStructuredText format. First import
``factorial`` from the ``example`` module:
>>> from example import factorial
Now use it:
>>> factorial(6)
120
运行doctest.testfile("example.txt")会找出文档中的错误：
File "./example.txt", line 14, in example.txt
Failed example:
factorial(6)
Expected:
120
Got:
720
和testmod()一样，testfile()不会显示任何东西，除非有例子失败。如果有失败，会打印出失败的例子及其原因，格式和testmod()一样。
默认情况下，testfile()会在调用模块的目录下查找文件。参考基本API章节，可以找到参数，使得它在其它位置查找文件。
和testmod()一样，testfile()的冗余程度可以由-v命令行开关来控制，或者是verbose参数。
Python 2.6之后，也有一个从命令行运行testfile()的快捷方式。可以让Python解释器直接从标准库中运行doctest模块，并在命令行中传递文件名。
python -m doctest -v example.txt
因为文件名不以.py结尾，doctest推测要用testfile()，而不是testmod()。
testfile()的更多信息请参阅Basic API章节。
这一章详细讨论了doctest是如何工作的：它查看哪些docstring，如何找到交互式的例子，它使用什么样的执行上下文，它如何处理异常，有哪些可选的开关可以控制它的行为。为了写doctest的例子，你需要知道这些信息；在这些例子上真正的运行doctest的信息，请参见后续章节。
模块的docstring，所有的函数、类和方法的docstring都会被搜索。导入模块的对象不会被搜索。
除此以外，如果M.__test__存在且“为真”，它必须是个字典，每个记录是名字字符串到函数对象、类对象或字符串的映射。M.__test__中的函数对象和类对象的docstring被搜索，字符串被认为是docstring。在输出中，M.__test__的K显示为：
<name of M>.__test__.K
任何找到的类会被类似的递归搜索，以测试它们包含的方法和嵌套类的docstring。
改变于版本2.4： “私有名字”概念被废弃，并且不再被文档记录。
大多数情况下，交互式控制台会话的拷贝会工作的很好；doctest并不会模仿任何特定的Python解释器。
>>> # comments are ignored
>>> x = 12
>>> x
12
>>> if x == 13:
... print "yes"
... else:
... print "no"
... print "NO"
... print "NO!!!"
...
no
NO
NO!!!
>>>
任何期待的输出必须紧跟在包含代码的'>>> ' 或者 '...’ 行的后面， 输出结束于下一个'>>> '或者空白行。
有效输出：
期待的输出不能包含空白行，因为它会被认为是输出的结束。如果输出中确实有空白行，用<BLANKLINE>代替每个空白行。
添加于版本2.4：添加了<BLANKLINE>；在以前的版本中无法表示输出中有空白行。
所有的制表符会被扩展成空白符，使用8列制表位。被测试的代码生成的输出中的制表符不会被修改。由于所有的制表符被扩展，这意味着如果代码输出包含制表符，doctest只有在NORMALIZE_WHITESPACE选项或指令有效的情况下才能通过测试。另一种方法，重写测试，让其捕获输出并和期待的值做比较。在源码中这样处理制表符可以通过试错而达成，被证明是较少出错的处理方式。写个自定义的DocTestParser类使用不同的算法来处理制表符也是可行的。
改变于版本2.4：新加了把制表符扩展成空白符；由于其困惑性，以前的版本会尝试保留制表符。
只捕获标准输出的输出，而没有标准错误的输出（异常回溯由不同的方法来捕获）。
如果在交互式会话中使用反斜线续行，或者由于其它的原因使用了反斜线，那么要使用原始的docstring，它可以如输入那样保留反斜线。
>>> def f(x):
... r'''Backslashes in a raw docstring: m\n'''
>>> print f.__doc__
Backslashes in a raw docstring: m\n
否则，反斜线被解释成字符串的一部分。例如， 上面的例子被解释成一个换行符。 也可以在doctest中使用双反斜线，从而不使用原始字符串。
>>> def f(x):
... '''Backslashes in a raw docstring: m\\n'''
>>> print f.__doc__
Backslashes in a raw docstring: m\n
起始栏的位置无关紧要：
>>> assert "Easy!"
>>> import math
>>> math.floor(1.9)
1.0
只要'>>> '开始的例子和期待的输出前有相同的空白即可。
默认情况下，当doctest找到docstring并测试时，它使用M的globals的浅拷贝，这样，运行的测试就不会改变模块的globals，M中的测试也不会留下一些对象，从而影响别的测试。这意味着例子可以随意使用定义于M中顶级的名字，以及在运行docstring之前定义的名字。例子无法看到定义于其它docstring中的名字。
通过给testmod()或者testfile()传递globs=your_dict，从而使用自己的字典作为执行上下文。
没问题，如果例子产生的输出只有回溯的话：只要把回溯贴出来就可以了。[1]由于回溯包含的细节会经常改变（比如文件路径和行号），doctest做了很多工作来保持弹性。
简单的例子：
>>> [1, 2, 3].remove(42)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: list.remove(x): x not in list
这个doctest会成功，如果ValueError被抛出，且细节为list.remove(x): x not in list。
对于异常，期待的输出必须以回溯头开始，它可以是下面两行中的一个，和例子的第一行有相同的缩进：
Traceback (most recent call last):
Traceback (innermost last):
回溯头后面是可选的回溯栈，它会被doctest忽略。典型地，回溯栈被省略，或者从交互式会话中拷贝而来。
回溯栈后面就是最有意思的部分了：包含异常类型和细节的行。一般来说，它是回溯的最后一行，但是如果有多行细节，它也可能是多行：
>>> raise ValueError('multi\n line\ndetail')
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: multi
line
detail
最后三行（ValueError打头）会和异常类型和细节进行比较，其余被忽略。
改变于版本2.4：以前的版本不能处理多行异常细节。
最佳实践就是省略掉回溯栈，除非它能给例子添加明显的文档价值。所以对于上面的例子，这样更好：
>>> raise ValueError('multi\n line\ndetail')
Traceback (most recent call last):
...
ValueError: multi
line
detail
注意回溯被特别对待。尤其是在重写的例子中，...和doctest的ELLIPSIS选项无关。例子中的省略号可以被忽视，也可以换成3个（或者300个）逗号或者数字，或者是“巨蟒剧团”短剧的脚本（带缩进的）。
有些细节需要了解，但不必记住：
doctest不会判断期待的输出是来自于异常回溯还是常规的打印。所以，期待ValueError: 42 is prime的例子会通过测试， 无论是由于ValueError被抛出，还是例子只是打印出回溯文本。实际上常规的输出极少以回溯头开始，所以这不会产生什么大问题。
回溯栈（如果有的话）的每一行必须要在例子的第一行的基础上进一步缩进，或者以非字母数字字符打头。回溯头后的第一行，和回溯头有相同的缩进，以字母数字打头，会被认为是异常细节的开始。这为真正的回溯做了正确的事情。
如果指明了doctest的IGNORE_EXCEPTION_DETAIL选项，最左边冒号后的所有东西和异常名中的模块信息都会被忽略。
交互式shell对于某些SyntaxError略去了回溯头。但是doctest使用回溯头来区分异常和非异常。所以在极少数情况下需要测试略去回溯头的SyntaxError的时候，需要在测试例子中手动加上回溯头。
对于一些SyntaxError，Python使用^来显示语法错误的字符位置：
>>> 1 1
File "<stdin>", line 1
1 1
^
SyntaxError: invalid syntax
由于这些显示错误位置的行早于异常类型和细节，doctest不检查它们。例如，下面的测试会通过，哪怕^指向了错误的位置：
>>> 1 1
File "<stdin>", line 1
1 1
^
SyntaxError: invalid syntax
有一些选项标志控制着doctest的各种行为。这些标志的符号名以模块常量的方式提供，可以做与运算，并传给各种函数。这些名字也可以用于doctest指令。
第一组选项定义了测试的语义，doctest靠它们来决定实际的输出是否匹配例子的期待的输出：
By default, if an expected output block contains just 1, an actual output block containing just 1 or just True is considered to be a match, and similarly for 0 versus False. When DONT_ACCEPT_TRUE_FOR_1 is specified, neither substitution is allowed. The default behavior caters to that Python changed the return type of many functions from integer to boolean; doctests expecting “little integer” output still work in these cases. This option will probably go away, but not for several years.
By default, if an expected output block contains a line containing only the string <BLANKLINE>, then that line will match a blank line in the actual output. Because a genuinely blank line delimits the expected output, this is the only way to communicate that a blank line is expected. When DONT_ACCEPT_BLANKLINE is specified, this substitution is not allowed.
When specified, all sequences of whitespace (blanks and newlines) are treated as equal. Any sequence of whitespace within the expected output will match any sequence of whitespace within the actual output. By default, whitespace must match exactly. NORMALIZE_WHITESPACE is especially useful when a line of expected output is very long, and you want to wrap it across multiple lines in your source.
When specified, an ellipsis marker (...) in the expected output can match any substring in the actual output. This includes substrings that span line boundaries, and empty substrings, so it’s best to keep usage of this simple. Complicated uses can lead to the same kinds of “oops, it matched too much!” surprises that .* is prone to in regular expressions.
When specified, an example that expects an exception passes if an exception of the expected type is raised, even if the exception detail does not match. For example, an example expecting ValueError: 42 will pass if the actual exception raised is ValueError: 3*14, but will fail, e.g., if TypeError is raised.
It will also ignore the module name used in Python 3 doctest reports. Hence both of these variations will work with the flag specified, regardless of whether the test is run under Python 2.7 or Python 3.2 (or later versions):
>>> raise CustomError('message')
Traceback (most recent call last):
CustomError: message
>>> raise CustomError('message')
Traceback (most recent call last):
my_module.CustomError: message
Note that ELLIPSIS can also be used to ignore the details of the exception message, but such a test may still fail based on whether or not the module details are printed as part of the exception name. Using IGNORE_EXCEPTION_DETAIL and the details from Python 2.3 is also the only clear way to write a doctest that doesn’t care about the exception detail yet continues to pass under Python 2.3 or earlier (those releases do not support doctest directives and ignore them as irrelevant comments). For example:
>>> (1, 2)[3] = 'moo'
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: object doesn't support item assignment
passes under Python 2.3 and later Python versions with the flag specified, even though the detail changed in Python 2.4 to say “does not” instead of “doesn’t”.
Changed in version 2.7: IGNORE_EXCEPTION_DETAIL now also ignores any information relating to the module containing the exception under test
When specified, do not run the example at all. This can be useful in contexts where doctest examples serve as both documentation and test cases, and an example should be included for documentation purposes, but should not be checked. E.g., the example’s output might be random; or the example might depend on resources which would be unavailable to the test driver.
The SKIP flag can also be used for temporarily “commenting out” examples.
New in version 2.5.
A bitmask or’ing together all the comparison flags above.
The second group of options controls how test failures are reported:
When specified, failures that involve multi-line expected and actual outputs are displayed using a unified diff.
When specified, failures that involve multi-line expected and actual outputs will be displayed using a context diff.
When specified, differences are computed by difflib.Differ, using the same algorithm as the popular ndiff.py utility. This is the only method that marks differences within lines as well as across lines. For example, if a line of expected output contains digit 1 where actual output contains letter l, a line is inserted with a caret marking the mismatching column positions.
When specified, display the first failing example in each doctest, but suppress output for all remaining examples. This will prevent doctest from reporting correct examples that break because of earlier failures; but it might also hide incorrect examples that fail independently of the first failure. When REPORT_ONLY_FIRST_FAILURE is specified, the remaining examples are still run, and still count towards the total number of failures reported; only the output is suppressed.
A bitmask or’ing together all the reporting flags above.
New in version 2.4: The constants DONT_ACCEPT_BLANKLINE, NORMALIZE_WHITESPACE, ELLIPSIS, IGNORE_EXCEPTION_DETAIL, REPORT_UDIFF, REPORT_CDIFF, REPORT_NDIFF, REPORT_ONLY_FIRST_FAILURE, COMPARISON_FLAGS and REPORTING_FLAGS were added.
There’s also a way to register new option flag names, although this isn’t useful unless you intend to extend doctest internals via subclassing:
Create a new option flag with a given name, and return the new flag’s integer value. register_optionflag() can be used when subclassing OutputChecker or DocTestRunner to create new options that are supported by your subclasses. register_optionflag() should always be called using the following idiom:
MY_FLAG = register_optionflag('MY_FLAG')
New in version 2.4.
Doctest directives may be used to modify the option flags for an individual example. Doctest directives are special Python comments following an example’s source code:
directive ::= "#" "doctest:" directive_options directive_options ::= directive_option ("," directive_option)\* directive_option ::= on_or_off directive_option_name on_or_off ::= "+" \| "-" directive_option_name ::= "DONT_ACCEPT_BLANKLINE" \| "NORMALIZE_WHITESPACE" \| ...
Whitespace is not allowed between the + or - and the directive option name. The directive option name can be any of the option flag names explained above.
An example’s doctest directives modify doctest’s behavior for that single example. Use + to enable the named behavior, or - to disable it.
For example, this test passes:
>>> print range(20) # doctest: +NORMALIZE_WHITESPACE
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Without the directive it would fail, both because the actual output doesn’t have two blanks before the single-digit list elements, and because the actual output is on a single line. This test also passes, and also requires a directive to do so:
>>> print range(20) # doctest: +ELLIPSIS
[0, 1, ..., 18, 19]
Multiple directives can be used on a single physical line, separated by commas:
>>> print range(20) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
[0, 1, ..., 18, 19]
If multiple directive comments are used for a single example, then they are combined:
>>> print range(20) # doctest: +ELLIPSIS
... # doctest: +NORMALIZE_WHITESPACE
[0, 1, ..., 18, 19]
As the previous example shows, you can add ... lines to your example containing only directives. This can be useful when an example is too long for a directive to comfortably fit on the same line:
>>> print range(5) + range(10,20) + range(30,40) + range(50,60)
... # doctest: +ELLIPSIS
[0, ..., 4, 10, ..., 19, 30, ..., 39, 50, ..., 59]
Note that since all options are disabled by default, and directives apply only to the example they appear in, enabling options (via + in a directive) is usually the only meaningful choice. However, option flags can also be passed to functions that run doctests, establishing different defaults. In such cases, disabling an option via - in a directive can be useful.
New in version 2.4: Support for doctest directives was added.
doctest is serious about requiring exact matches in expected output. If even a single character doesn’t match, the test fails. This will probably surprise you a few times, as you learn exactly what Python does and doesn’t guarantee about output. For example, when printing a dict, Python doesn’t guarantee that the key-value pairs will be printed in any particular order, so a test like
>>> foo()
{"Hermione": "hippogryph", "Harry": "broomstick"}
is vulnerable! One workaround is to do
>>> foo() == {"Hermione": "hippogryph", "Harry": "broomstick"}
True
instead. Another is to do
>>> d = foo().items()
>>> d.sort()
>>> d
[('Harry', 'broomstick'), ('Hermione', 'hippogryph')]
There are others, but you get the idea.
Another bad idea is to print things that embed an object address, like
>>> id(1.0) # certain to fail some of the time
7948648
>>> class C: pass
>>> C() # the default repr() for instances embeds an address
<__main__.C instance at 0x00AC18F0>
The ELLIPSIS directive gives a nice approach for the last example:
>>> C() #doctest: +ELLIPSIS
<__main__.C instance at 0x...>
Floating-point numbers are also subject to small output variations across platforms, because Python defers to the platform C library for float formatting, and C libraries vary widely in quality here.
>>> 1./7 # risky
0.14285714285714285
>>> print 1./7 # safer
0.142857142857
>>> print round(1./7, 6) # much safer
0.142857
Numbers of the form I/2.**J are safe across all platforms, and I often contrive doctest examples to produce numbers of that form:
>>> 3./4 # utterly safe
0.75
Simple fractions are also easier for people to understand, and that makes for better documentation.
The functions testmod() and testfile() provide a simple interface to doctest that should be sufficient for most basic uses. For a less formal introduction to these two functions, see sections Simple Usage: Checking Examples in Docstrings and Simple Usage: Checking Examples in a Text File.
All arguments except filename are optional, and should be specified in keyword form.
Test examples in the file named filename. Return (failure_count, test_count).
Optional argument module_relative specifies how the filename should be interpreted:
Optional argument name gives the name of the test; by default, or if None, os.path.basename(filename) is used.
Optional argument package is a Python package or the name of a Python package whose directory should be used as the base directory for a module-relative filename. If no package is specified, then the calling module’s directory is used as the base directory for module-relative filenames. It is an error to specify package if module_relative is False.
Optional argument globs gives a dict to be used as the globals when executing examples. A new shallow copy of this dict is created for the doctest, so its examples start with a clean slate. By default, or if None, a new empty dict is used.
Optional argument extraglobs gives a dict merged into the globals used to execute examples. This works like dict.update(): if globs and extraglobs have a common key, the associated value in extraglobs appears in the combined dict. By default, or if None, no extra globals are used. This is an advanced feature that allows parameterization of doctests. For example, a doctest can be written for a base class, using a generic name for the class, then reused to test any number of subclasses by passing an extraglobs dict mapping the generic name to the subclass to be tested.
Optional argument verbose prints lots of stuff if true, and prints only failures if false; by default, or if None, it’s true if and only if '-v' is in sys.argv.
Optional argument report prints a summary at the end when true, else prints nothing at the end. In verbose mode, the summary is detailed, else the summary is very brief (in fact, empty if all tests passed).
Optional argument optionflags or’s together option flags. See section Option Flags.
Optional argument raise_on_error defaults to false. If true, an exception is raised upon the first failure or unexpected exception in an example. This allows failures to be post-mortem debugged. Default behavior is to continue running examples.
Optional argument parser specifies a DocTestParser (or subclass) that should be used to extract tests from the files. It defaults to a normal parser (i.e., DocTestParser()).
Optional argument encoding specifies an encoding that should be used to convert the file to unicode.
New in version 2.4.
Changed in version 2.5: The parameter encoding was added.
All arguments are optional, and all except for m should be specified in keyword form.
Test examples in docstrings in functions and classes reachable from module m (or module __main__ if m is not supplied or is None), starting with m.__doc__.
Also test examples reachable from dict m.__test__, if it exists and is not None. m.__test__ maps names (strings) to functions, classes and strings; function and class docstrings are searched for examples; strings are searched directly, as if they were docstrings.
Only docstrings attached to objects belonging to module m are searched.
Return (failure_count, test_count).
Optional argument name gives the name of the module; by default, or if None, m.__name__ is used.
Optional argument exclude_empty defaults to false. If true, objects for which no doctests are found are excluded from consideration. The default is a backward compatibility hack, so that code still using doctest.master.summarize() in conjunction with testmod() continues to get output for objects with no tests. The exclude_empty argument to the newer DocTestFinder constructor defaults to true.
Optional arguments extraglobs, verbose, report, optionflags, raise_on_error, and globs are the same as for function testfile() above, except that globs defaults to m.__dict__.
Changed in version 2.3: The parameter optionflags was added.
Changed in version 2.4: The parameters extraglobs, raise_on_error and exclude_empty were added.
Changed in version 2.5: The optional argument isprivate, deprecated in 2.4, was removed.
There’s also a function to run the doctests associated with a single object. This function is provided for backward compatibility. There are no plans to deprecate it, but it’s rarely useful:
Test examples associated with object f; for example, f may be a module, function, or class object.
A shallow copy of dictionary argument globs is used for the execution context.
Optional argument name is used in failure messages, and defaults to "NoName".
If optional argument verbose is true, output is generated even if there are no failures. By default, output is generated only in case of an example failure.
Optional argument compileflags gives the set of flags that should be used by the Python compiler when running the examples. By default, or if None, flags are deduced corresponding to the set of future features found in globs.
Optional argument optionflags works as for function testfile() above.
As your collection of doctest’ed modules grows, you’ll want a way to run all their doctests systematically. Prior to Python 2.4, doctest had a barely documented Tester class that supplied a rudimentary way to combine doctests from multiple modules. Tester was feeble, and in practice most serious Python testing frameworks build on the unittest module, which supplies many flexible ways to combine tests from multiple sources. So, in Python 2.4, doctest‘s Tester class is deprecated, and doctest provides two functions that can be used to create unittest test suites from modules and text files containing doctests. To integrate with unittest test discovery, include a load_tests() function in your test module:
import unittest
import doctest
import my_module_with_doctests
def load_tests(loader, tests, ignore):
tests.addTests(doctest.DocTestSuite(my_module_with_doctests))
return tests
There are two main functions for creating unittest.TestSuite instances from text files and modules with doctests:
Convert doctest tests from one or more text files to a unittest.TestSuite.
The returned unittest.TestSuite is to be run by the unittest framework and runs the interactive examples in each file. If an example in any file fails, then the synthesized unit test fails, and a failureException exception is raised showing the name of the file containing the test and a (sometimes approximate) line number.
Pass one or more paths (as strings) to text files to be examined.
Options may be provided as keyword arguments:
Optional argument module_relative specifies how the filenames in paths should be interpreted:
Optional argument package is a Python package or the name of a Python package whose directory should be used as the base directory for module-relative filenames in paths. If no package is specified, then the calling module’s directory is used as the base directory for module-relative filenames. It is an error to specify package if module_relative is False.
Optional argument setUp specifies a set-up function for the test suite. This is called before running the tests in each file. The setUp function will be passed a DocTest object. The setUp function can access the test globals as the globs attribute of the test passed.
Optional argument tearDown specifies a tear-down function for the test suite. This is called after running the tests in each file. The tearDown function will be passed a DocTest object. The setUp function can access the test globals as the globs attribute of the test passed.
Optional argument globs is a dictionary containing the initial global variables for the tests. A new copy of this dictionary is created for each test. By default, globs is a new empty dictionary.
Optional argument optionflags specifies the default doctest options for the tests, created by or-ing together individual option flags. See section Option Flags. See function set_unittest_reportflags() below for a better way to set reporting options.
Optional argument parser specifies a DocTestParser (or subclass) that should be used to extract tests from the files. It defaults to a normal parser (i.e., DocTestParser()).
Optional argument encoding specifies an encoding that should be used to convert the file to unicode.
New in version 2.4.
Changed in version 2.5: The global __file__ was added to the globals provided to doctests loaded from a text file using DocFileSuite().
Changed in version 2.5: The parameter encoding was added.
Note
Unlike testmod() and DocTestFinder, this function raises a ValueError if module contains no docstrings. You can prevent this error by passing a DocTestFinder instance as the test_finder argument with its exclude_empty keyword argument set to False:
>>> finder = doctest.DocTestFinder(exclude_empty=False)
>>> suite = doctest.DocTestSuite(test_finder=finder)
Convert doctest tests for a module to a unittest.TestSuite.
The returned unittest.TestSuite is to be run by the unittest framework and runs each doctest in the module. If any of the doctests fail, then the synthesized unit test fails, and a failureException exception is raised showing the name of the file containing the test and a (sometimes approximate) line number.
Optional argument module provides the module to be tested. It can be a module object or a (possibly dotted) module name. If not specified, the module calling this function is used.
Optional argument globs is a dictionary containing the initial global variables for the tests. A new copy of this dictionary is created for each test. By default, globs is a new empty dictionary.
Optional argument extraglobs specifies an extra set of global variables, which is merged into globs. By default, no extra globals are used.
Optional argument test_finder is the DocTestFinder object (or a drop-in replacement) that is used to extract doctests from the module.
Optional arguments setUp, tearDown, and optionflags are the same as for function DocFileSuite() above.
New in version 2.3.
Changed in version 2.4: The parameters globs, extraglobs, test_finder, setUp, tearDown, and optionflags were added; this function now uses the same search technique as testmod().
Under the covers, DocTestSuite() creates a unittest.TestSuite out of doctest.DocTestCase instances, and DocTestCase is a subclass of unittest.TestCase. DocTestCase isn’t documented here (it’s an internal detail), but studying its code can answer questions about the exact details of unittest integration.
Similarly, DocFileSuite() creates a unittest.TestSuite out of doctest.DocFileCase instances, and DocFileCase is a subclass of DocTestCase.
So both ways of creating a unittest.TestSuite run instances of DocTestCase. This is important for a subtle reason: when you run doctest functions yourself, you can control the doctest options in use directly, by passing option flags to doctest functions. However, if you’re writing a unittest framework, unittest ultimately controls when and how tests get run. The framework author typically wants to control doctest reporting options (perhaps, e.g., specified by command line options), but there’s no way to pass options through unittest to doctest test runners.
For this reason, doctest also supports a notion of doctest reporting flags specific to unittest support, via this function:
Set the doctest reporting flags to use.
Argument flags or’s together option flags. See section Option Flags. Only “reporting flags” can be used.
This is a module-global setting, and affects all future doctests run by module unittest: the runTest() method of DocTestCase looks at the option flags specified for the test case when the DocTestCase instance was constructed. If no reporting flags were specified (which is the typical and expected case), doctest‘s unittest reporting flags are or’ed into the option flags, and the option flags so augmented are passed to the DocTestRunner instance created to run the doctest. If any reporting flags were specified when the DocTestCase instance was constructed, doctest‘s unittest reporting flags are ignored.
The value of the unittest reporting flags in effect before the function was called is returned by the function.
New in version 2.4.
The basic API is a simple wrapper that’s intended to make doctest easy to use. It is fairly flexible, and should meet most users’ needs; however, if you require more fine-grained control over testing, or wish to extend doctest’s capabilities, then you should use the advanced API.
The advanced API revolves around two container classes, which are used to store the interactive examples extracted from doctest cases:
Additional processing classes are defined to find, parse, and run, and check doctest examples:
The relationships among these processing classes are summarized in the following diagram:
list of:
+------+ +---------+
|module| --DocTestFinder-> | DocTest | --DocTestRunner-> results
+------+ | ^ +---------+ | ^ (printed)
| | | Example | | |
v | | ... | v |
DocTestParser | Example | OutputChecker
+---------+
A collection of doctest examples that should be run in a single namespace. The constructor arguments are used to initialize the attributes of the same names.
New in version 2.4.
DocTest defines the following attributes. They are initialized by the constructor, and should not be modified directly.
A list of Example objects encoding the individual interactive Python examples that should be run by this test.
The namespace (aka globals) that the examples should be run in. This is a dictionary mapping names to values. Any changes to the namespace made by the examples (such as binding new variables) will be reflected in globs after the test is run.
A string name identifying the DocTest. Typically, this is the name of the object or file that the test was extracted from.
The name of the file that this DocTest was extracted from; or None if the filename is unknown, or if the DocTest was not extracted from a file.
The line number within filename where this DocTest begins, or None if the line number is unavailable. This line number is zero-based with respect to the beginning of the file.
The string that the test was extracted from, or ‘None’ if the string is unavailable, or if the test was not extracted from a string.
A single interactive example, consisting of a Python statement and its expected output. The constructor arguments are used to initialize the attributes of the same names.
New in version 2.4.
Example defines the following attributes. They are initialized by the constructor, and should not be modified directly.
A string containing the example’s source code. This source code consists of a single Python statement, and always ends with a newline; the constructor adds a newline when necessary.
The expected output from running the example’s source code (either from stdout, or a traceback in case of exception). want ends with a newline unless no output is expected, in which case it’s an empty string. The constructor adds a newline when necessary.
The exception message generated by the example, if the example is expected to generate an exception; or None if it is not expected to generate an exception. This exception message is compared against the return value of traceback.format_exception_only(). exc_msg ends with a newline unless it’s None. The constructor adds a newline if needed.
The line number within the string containing this example where the example begins. This line number is zero-based with respect to the beginning of the containing string.
The example’s indentation in the containing string, i.e., the number of space characters that precede the example’s first prompt.
A dictionary mapping from option flags to True or False, which is used to override default options for this example. Any option flags not contained in this dictionary are left at their default value (as specified by the DocTestRunner‘s optionflags). By default, no options are set.
A processing class used to extract the DocTests that are relevant to a given object, from its docstring and the docstrings of its contained objects. DocTests can currently be extracted from the following object types: modules, functions, classes, methods, staticmethods, classmethods, and properties.
The optional argument verbose can be used to display the objects searched by the finder. It defaults to False (no output).
The optional argument parser specifies the DocTestParser object (or a drop-in replacement) that is used to extract doctests from docstrings.
If the optional argument recurse is false, then DocTestFinder.find() will only examine the given object, and not any contained objects.
If the optional argument exclude_empty is false, then DocTestFinder.find() will include tests for objects with empty docstrings.
New in version 2.4.
DocTestFinder defines the following method:
Return a list of the DocTests that are defined by obj‘s docstring, or by any of its contained objects’ docstrings.
The optional argument name specifies the object’s name; this name will be used to construct names for the returned DocTests. If name is not specified, then obj.__name__ is used.
The optional parameter module is the module that contains the given object. If the module is not specified or is None, then the test finder will attempt to automatically determine the correct module. The object’s module is used:
If module is False, no attempt to find the module will be made. This is obscure, of use mostly in testing doctest itself: if module is False, or is None but cannot be found automatically, then all objects are considered to belong to the (non-existent) module, so all contained objects will (recursively) be searched for doctests.
The globals for each DocTest is formed by combining globs and extraglobs (bindings in extraglobs override bindings in globs). A new shallow copy of the globals dictionary is created for each DocTest. If globs is not specified, then it defaults to the module’s __dict__, if specified, or {} otherwise. If extraglobs is not specified, then it defaults to {}.
A processing class used to extract interactive examples from a string, and use them to create a DocTest object.
New in version 2.4.
DocTestParser defines the following methods:
Extract all doctest examples from the given string, and collect them into a DocTest object.
globs, name, filename, and lineno are attributes for the new DocTest object. See the documentation for DocTest for more information.
A processing class used to execute and verify the interactive examples in a DocTest.
The comparison between expected outputs and actual outputs is done by an OutputChecker. This comparison may be customized with a number of option flags; see section Option Flags for more information. If the option flags are insufficient, then the comparison may also be customized by passing a subclass of OutputChecker to the constructor.
The test runner’s display output can be controlled in two ways. First, an output function can be passed to TestRunner.run(); this function will be called with strings that should be displayed. It defaults to sys.stdout.write. If capturing the output is not sufficient, then the display output can be also customized by subclassing DocTestRunner, and overriding the methods report_start(), report_success(), report_unexpected_exception(), and report_failure().
The optional keyword argument checker specifies the OutputChecker object (or drop-in replacement) that should be used to compare the expected outputs to the actual outputs of doctest examples.
The optional keyword argument verbose controls the DocTestRunner‘s verbosity. If verbose is True, then information is printed about each example, as it is run. If verbose is False, then only failures are printed. If verbose is unspecified, or None, then verbose output is used iff the command-line switch -v is used.
The optional keyword argument optionflags can be used to control how the test runner compares expected output to actual output, and how it displays failures. For more information, see section Option Flags.
New in version 2.4.
DocTestParser defines the following methods:
Report that the test runner is about to process the given example. This method is provided to allow subclasses of DocTestRunner to customize their output; it should not be called directly.
example is the example about to be processed. test is the test containing example. out is the output function that was passed to DocTestRunner.run().
Report that the given example ran successfully. This method is provided to allow subclasses of DocTestRunner to customize their output; it should not be called directly.
example is the example about to be processed. got is the actual output from the example. test is the test containing example. out is the output function that was passed to DocTestRunner.run().
Report that the given example failed. This method is provided to allow subclasses of DocTestRunner to customize their output; it should not be called directly.
example is the example about to be processed. got is the actual output from the example. test is the test containing example. out is the output function that was passed to DocTestRunner.run().
Report that the given example raised an unexpected exception. This method is provided to allow subclasses of DocTestRunner to customize their output; it should not be called directly.
example is the example about to be processed. exc_info is a tuple containing information about the unexpected exception (as returned by sys.exc_info()). test is the test containing example. out is the output function that was passed to DocTestRunner.run().
Run the examples in test (a DocTest object), and display the results using the writer function out.
The examples are run in the namespace test.globs. If clear_globs is true (the default), then this namespace will be cleared after the test runs, to help with garbage collection. If you would like to examine the namespace after the test completes, then use clear_globs=False.
compileflags gives the set of flags that should be used by the Python compiler when running the examples. If not specified, then it will default to the set of future-import flags that apply to globs.
The output of each example is checked using the DocTestRunner‘s output checker, and the results are formatted by the DocTestRunner.report_*() methods.
Print a summary of all the test cases that have been run by this DocTestRunner, and return a named tuple TestResults(failed, attempted).
The optional verbose argument controls how detailed the summary is. If the verbosity is not specified, then the DocTestRunner‘s verbosity is used.
Changed in version 2.6: Use a named tuple.
A class used to check the whether the actual output from a doctest example matches the expected output. OutputChecker defines two methods: check_output(), which compares a given pair of outputs, and returns true if they match; and output_difference(), which returns a string describing the differences between two outputs.
New in version 2.4.
OutputChecker defines the following methods:
Return True iff the actual output from an example (got) matches the expected output (want). These strings are always considered to match if they are identical; but depending on what option flags the test runner is using, several non-exact match types are also possible. See section Option Flags for more information about option flags.
Return a string describing the differences between the expected output for a given example (example) and the actual output (got). optionflags is the set of option flags used to compare want and got.
Doctest provides several mechanisms for debugging doctest examples:
Several functions convert doctests to executable Python programs, which can be run under the Python debugger, pdb.
The DebugRunner class is a subclass of DocTestRunner that raises an exception for the first failing example, containing information about that example. This information can be used to perform post-mortem debugging on the example.
The unittest cases generated by DocTestSuite() support the debug() method defined by unittest.TestCase.
You can add a call to pdb.set_trace() in a doctest example, and you’ll drop into the Python debugger when that line is executed. Then you can inspect current values of variables, and so on. For example, suppose a.py contains just this module docstring:
"""
>>> def f(x):
... g(x*2)
>>> def g(x):
... print x+3
... import pdb; pdb.set_trace()
>>> f(3)
9
"""
Then an interactive Python session may look like this:
>>> import a, doctest
>>> doctest.testmod(a)
--Return--
> <doctest a[1]>(3)g()->None
-> import pdb; pdb.set_trace()
(Pdb) list
1 def g(x):
2 print x+3
3 -> import pdb; pdb.set_trace()
[EOF]
(Pdb) print x
6
(Pdb) step
--Return--
> <doctest a[0]>(2)f()->None
-> g(x*2)
(Pdb) list
1 def f(x):
2 -> g(x*2)
[EOF]
(Pdb) print x
3
(Pdb) step
--Return--
> <doctest a[2]>(1)?()->None
-> f(3)
(Pdb) cont
(0, 3)
>>>
Changed in version 2.4: The ability to use pdb.set_trace() usefully inside doctests was added.
Functions that convert doctests to Python code, and possibly run the synthesized code under the debugger:
Convert text with examples to a script.
Argument s is a string containing doctest examples. The string is converted to a Python script, where doctest examples in s are converted to regular code, and everything else is converted to Python comments. The generated script is returned as a string. For example,
import doctest
print doctest.script_from_examples(r"""
Set x and y to 1 and 2.
>>> x, y = 1, 2
Print their sum:
>>> print x+y
3
""")
displays:
# Set x and y to 1 and 2.
x, y = 1, 2
#
# Print their sum:
print x+y
# Expected:
## 3
This function is used internally by other functions (see below), but can also be useful when you want to transform an interactive Python session into a Python script.
New in version 2.4.
Convert the doctest for an object to a script.
Argument module is a module object, or dotted name of a module, containing the object whose doctests are of interest. Argument name is the name (within the module) of the object with the doctests of interest. The result is a string, containing the object’s docstring converted to a Python script, as described for script_from_examples() above. For example, if module a.py contains a top-level function f(), then
import a, doctest
print doctest.testsource(a, "a.f")
prints a script version of function f()‘s docstring, with doctests converted to code, and the rest placed in comments.
New in version 2.3.
Debug the doctests for an object.
The module and name arguments are the same as for function testsource() above. The synthesized Python script for the named object’s docstring is written to a temporary file, and then that file is run under the control of the Python debugger, pdb.
A shallow copy of module.__dict__ is used for both local and global execution context.
Optional argument pm controls whether post-mortem debugging is used. If pm has a true value, the script file is run directly, and the debugger gets involved only if the script terminates via raising an unhandled exception. If it does, then post-mortem debugging is invoked, via pdb.post_mortem(), passing the traceback object from the unhandled exception. If pm is not specified, or is false, the script is run under the debugger from the start, via passing an appropriate execfile() call to pdb.run().
New in version 2.3.
Changed in version 2.4: The pm argument was added.
Debug the doctests in a string.
This is like function debug() above, except that a string containing doctest examples is specified directly, via the src argument.
Optional argument pm has the same meaning as in function debug() above.
Optional argument globs gives a dictionary to use as both local and global execution context. If not specified, or None, an empty dictionary is used. If specified, a shallow copy of the dictionary is used.
New in version 2.4.
The DebugRunner class, and the special exceptions it may raise, are of most interest to testing framework authors, and will only be sketched here. See the source code, and especially DebugRunner‘s docstring (which is a doctest!) for more details:
A subclass of DocTestRunner that raises an exception as soon as a failure is encountered. If an unexpected exception occurs, an UnexpectedException exception is raised, containing the test, the example, and the original exception. If the output doesn’t match, then a DocTestFailure exception is raised, containing the test, the example, and the actual output.
For information about the constructor parameters and methods, see the documentation for DocTestRunner in section Advanced API.
There are two exceptions that may be raised by DebugRunner instances:
An exception raised by DocTestRunner to signal that a doctest example’s actual output did not match its expected output. The constructor arguments are used to initialize the attributes of the same names.
DocTestFailure defines the following attributes:
The example’s actual output.
An exception raised by DocTestRunner to signal that a doctest example raised an unexpected exception. The constructor arguments are used to initialize the attributes of the same names.
UnexpectedException defines the following attributes:
A tuple containing information about the unexpected exception, as returned by sys.exc_info().
As mentioned in the introduction, doctest has grown to have three primary uses:
These uses have different requirements, and it is important to distinguish them. In particular, filling your docstrings with obscure test cases makes for bad documentation.
When writing a docstring, choose docstring examples with care. There’s an art to this that needs to be learned—it may not be natural at first. Examples should add genuine value to the documentation. A good example can often be worth many words. If done with care, the examples will be invaluable for your users, and will pay back the time it takes to collect them many times over as the years go by and things change. I’m still amazed at how often one of my doctest examples stops working after a “harmless” change.
Doctest also makes an excellent tool for regression testing, especially if you don’t skimp on explanatory text. By interleaving prose and examples, it becomes much easier to keep track of what’s actually being tested, and why. When a test fails, good prose can make it much easier to figure out what the problem is, and how it should be fixed. It’s true that you could write extensive comments in code-based testing, but few programmers do. Many have found that using doctest approaches instead leads to much clearer tests. Perhaps this is simply because doctest makes writing prose a little easier than writing code, while writing comments in code is a little harder. I think it goes deeper than just that: the natural attitude when writing a doctest-based test is that you want to explain the fine points of your software, and illustrate them with examples. This in turn naturally leads to test files that start with the simplest features, and logically progress to complications and edge cases. A coherent narrative is the result, instead of a collection of isolated functions that test isolated bits of functionality seemingly at random. It’s a different attitude, and produces different results, blurring the distinction between testing and explaining.
Regression testing is best confined to dedicated objects or files. There are several options for organizing tests:
Footnotes
| [1] | Examples containing both expected output and an exception are not supported. Trying to guess where one ends and the other begins is too error-prone, and that also makes for a confusing test. |